cannot mount a custom cassandra.yaml

[ltagliamonte]

Hello guys,
I'm hitting the same problem described here:
moby/moby#6011
so i cannot mount a custom cassandra.yaml file.
Is there any change that the community has already found a way to customise it?

[ltagliamonte]

@tianon @yosifkit what do you thing to add and additional env variable and perform something like the following code:

#example of the env variable:
ADDITIONAL_CONF='/etc/cassandra/cassandra.yaml@batch_size_warn_threshold_in_kb:@1024 /etc/cassandra/cassandra.yaml@read_request_timeout_in_ms:@10000'

if [[ ! -z ${ADDITIONAL_CONF} ]]; then
    for custom_conf in ${ADDITIONAL_CONF}
    do
        file=$(echo ${custom_conf} | cut -d'@' -f1)
        conf=$(echo ${custom_conf} | cut -d'@' -f2)
        val=$(echo ${custom_conf} | cut -d'@' -f3)
        echo "customising ${conf} ${val} in ${file}"
        sed -i "s|^${conf}.*|${conf} ${val}|" ${file}
    done
fi

[yosifkit]

Not sure I would be a fan of the ADDITIONAL_CONF; too bad cassandra doesn't just have a conf.d directory like mysql. 😢

I have two possible workarounds to avoid the inode issue:

• copy all of /etc/cassandra/ to your host and mount the whole directory to the container (not so great but gives at lot of choices on configuring any of those files)
• make /etc/cassandra.yaml a symlink in the container to something like /cassandra-config/cassandra.yaml and bind your host folder there. (probably requires you to create an image FROM cassandra to then RUN ln -sf)

[ltagliamonte]

Hi @yosifkit thank you for the feedback,

• I cannot go for your first solution because i'm using an as a service platform so I cannot put files to mount on the FS.
• About the second workaround; the goal of my proposal is to avoid to maintain another a docker image, and continue to use the official image from the repo.
  Do you see something dangerous or wrong in the ADDITIONAL_CONF proposal?

[avramirez]

I tried @yosifkit 1st option and on bootup it still uses all the configs from default /var/lib/cassandra.

I copied all configs from default /etc/cassandra then modified it, then I mounted it on docker container to /etc/cassandra

the configs took effect but I'm getting this at the end

Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any seeds java.lang.RuntimeException: Unable to gossip with any seeds at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1346) at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:526) at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:774) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:691) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:578) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) ERROR 09:16:34 Exception encountered during startup java.lang.RuntimeException: Unable to gossip with any seeds at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1346) ~[apache-cassandra-3.2.1.jar:3.2.1] at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:526) ~[apache-cassandra-3.2.1.jar:3.2.1] at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:774) ~[apache-cassandra-3.2.1.jar:3.2.1] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:691) ~[apache-cassandra-3.2.1.jar:3.2.1] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:578) ~[apache-cassandra-3.2.1.jar:3.2.1] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349) [apache-cassandra-3.2.1.jar:3.2.1] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.2.1.jar:3.2.1] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.2.1.jar:3.2.1]

[dyangelo-grullon]

What is the status on this?

[jacktasia]

FWIW One option is to use -Dcassandra.config=/path/to/custom/cassandra.yml inside your JVM_EXTRA_OPTS env when calling docker.

For example here's loading a cassandra.yaml from a volume

docker run --net bridge -e JVM_EXTRA_OPTS=-Dcassandra.config=/tmp/custom_cassandra/config.yaml -p 7199:7199 -p 7000:7000 -p 7001:7001 -p 9160:9160 -p 9042:9042 -v /tmp/custom_cassandra:/tmp/custom_cassandra --restart=always --name my-custom-config-cassandra cassandra:3.11

then inside the container I can see

INFO  [main] 2017-11-09 21:24:18,035 YamlConfigurationLoader.java:89 - Configuration location: file:/tmp/custom_cassandra/config.yaml

Hope that helps!

[yosifkit]

Looks like this issue is solved. #58 (comment) is a solid workaround.

[domeniconappo]

Just a comment (and let me know if I miss something important here):

to me it seems that cassandra.yaml is created dinamically (with all RPC bind addresses based on docker networks and so on) and some env variables listed here: https://docs.docker.com/samples/library/cassandra/#environment-variables

How is it possible then to use a custom static cassandra configuration in docker and docker compose/swarm environments?

e.g.

# For security reasons, you should not expose this port to the internet.  Firewall it if needed.
rpc_address: 0.0.0.0

# Set rpc_address OR rpc_interface, not both. Interfaces must correspond
# to a single address, IP aliasing is not supported.
# rpc_interface: eth1

# If you choose to specify the interface by name and the interface has an ipv4 and an ipv6 address
# you can specify which should be chosen using rpc_interface_prefer_ipv6. If false the first ipv4
# address will be used. If true the first ipv6 address will be used. Defaults to false preferring
# ipv4. If there is only one address it will be selected regardless of ipv4/ipv6.
# rpc_interface_prefer_ipv6: false

# port for Thrift to listen for clients on
rpc_port: 9160

# RPC address to broadcast to drivers and other Cassandra nodes. This cannot
# be set to 0.0.0.0. If left blank, this will be set to the value of
# rpc_address. If rpc_address is set to 0.0.0.0, broadcast_rpc_address must
# be set.
# ......this seems problematic to me.........
broadcast_rpc_address: 172.18.0.4

Would be nice maybe to add the Authentication related variables to the list of ENV variables that are used by the image to produce cassandra.yaml

#   If using PasswordAuthenticator, CassandraRoleManager must also be used (see below)
authenticator: AllowAllAuthenticator

see #106

[domeniconappo]

cassandra.yaml is actually 'sed' by docker-entrypoint.sh for listed ENV variables.

https://github.com/docker-library/cassandra/blob/01786683a1b060f813bce5539b1743695f9cc043/3.11/docker-entrypoint.sh

So it should work to mount your own cassandra.yaml and let entrypoint substitute relevant variables.

Sorry for confusion!

[tianon]

FYI, this was finally fixed properly over in #160.