Configuration from docker config or secret?

[endyjasmi]

I am trying to run CouchDB cluster with this image in production and everything works fine. My question is, is there anyway for us to mount config file from docker swarm config or secret. Currently doing so will cause the image to complain about chmod againts read only file system.

Expected Behavior

CouchDB should be able to read the docker swarm secret mounted at /opt/couchdb/etc/local.d/config.ini.

Current Behavior

Complain about changing permission for read only file system on boot.

Possible Solution

Maybe remove the chmod command at docker-entrypoint.sh?

Steps to Reproduce (for bugs)

1. Prepare a sample config.ini
2. Create a docker-compose.yml file with the following content;

version: "3.5"

networks:
  network:
    driver: overlay

secrets:
  config:
    file: ./config.ini

services:
  server-0:
    environment:
      COUCHDB_PASSWORD: -pbkdf2-847043acc65626c8eb98da6d78682fbc493a1787,f7b1a3e4b624f4f0bbfe87e96841eda0,10 // <- password
      COUCHDB_SECRET: 0123456789abcdef0123456789abcdef
      COUCHDB_USER: couchdb
      NODENAME: couchdb-0.docker.com
    image: couchdb:2.1.1
    networks:
      network:
        aliases:
          - couchdb-0.docker.com
    ports:
      - "5984:5984"
      - "5986:5986"
    secrets:
      - source: config
        target: /opt/couchdb/etc/local.d/config.ini
        uid: "5984"
        gid: "5984"
        mode: 0444
    volumes:
      - "volume-0:/opt/couchdb/data"
  server-1:
    environment:
      COUCHDB_PASSWORD: -pbkdf2-847043acc65626c8eb98da6d78682fbc493a1787,f7b1a3e4b624f4f0bbfe87e96841eda0,10
      COUCHDB_SECRET: 0123456789abcdef0123456789abcdef
      COUCHDB_USER: couchdb
      NODENAME: couchdb-1.docker.com
    image: couchdb:2.1.1
    networks:
      network:
        aliases:
          - couchdb-1.docker.com
    ports:
      - "15984:5984"
      - "15986:5986"
    secrets:
      - source: config
        target: /opt/couchdb/etc/local.d/config.ini
        uid: "5984"
        gid: "5984"
        mode: 0444
    volumes:
      - "volume-1:/opt/couchdb/data"
  server-2:
    environment:
      COUCHDB_PASSWORD: -pbkdf2-847043acc65626c8eb98da6d78682fbc493a1787,f7b1a3e4b624f4f0bbfe87e96841eda0,10
      COUCHDB_SECRET: 0123456789abcdef0123456789abcdef
      COUCHDB_USER: couchdb
      NODENAME: couchdb-2.docker.com
    image: couchdb:2.1.1
    networks:
      network:
        aliases:
          - couchdb-2.docker.com
    ports:
      - "25984:5984"
      - "25986:5986"
    secrets:
      - source: config
        target: /opt/couchdb/etc/local.d/config.ini
        uid: "5984"
        gid: "5984"
        mode: 0444
    volumes:
      - "volume-2:/opt/couchdb/data"

volumes:
  volume-0:
  volume-1:
  volume-2:

3. Run docker stack deploy --compose-file docker-compose.yml couchdb againts a docker swarm.
4. You can view the log at docker service logs -f couchdb_server-0

Context

Basically I am trying to deploy CouchDB cluster to the production using docker swarm.

[endyjasmi]

Docker config and secret by default seems to be read only system system. By chaging ownership and permission at the following docker-entrypoint.sh seems to cause trouble for the image to run.

couchdb-docker/2.1.1/docker-entrypoint.sh

Lines 29 to 35 in 1f85f08

	chown -R couchdb:couchdb /opt/couchdb
	chmod -R 0770 /opt/couchdb/data
	chmod 664 /opt/couchdb/etc/*.ini
	chmod 664 /opt/couchdb/etc/default.d/*.ini
	chmod 775 /opt/couchdb/etc/*.d

Maybe we can change it to the following command where the script will ignore the error?

 chown -R couchdb:couchdb /opt/couchdb
  
 chmod -R 0770 /opt/couchdb/data
  
 chmod 664 /opt/couchdb/etc/*.ini || true
 chmod 664 /opt/couchdb/etc/default.d/*.ini
 chmod 775 /opt/couchdb/etc/*.d

It seems that couchdb will overwrite .ini's inside default.d and local.d hence the config and secret cannot be mounted into those places. It should be save for us to mount it to /opt/couchdb/etc/local.ini.

[wohali]

CouchDB has to be able to write to its ini files. If your setup doesn't allow this then it will fail any PUT /_node/_local/_config operations, and it will not be able to overwrite any ini file defined admins where the passwords are not hashed.

Is there no way to allow docker swarm to mount configuration files in a writeable form?

[endyjasmi]

@wohali Thanks for the reply. Based on https://docs.docker.com/compose/compose-file/#long-syntax and https://docs.docker.com/compose/compose-file/#long-syntax-2 . It seems to me that both config and secret is mounted at read only filesystem.

My current workaround is based on https://github.com/apache/couchdb-docker#build-your-own where I build my own image for persistent. The only caveat I encounter is whenever I update and re-docker stack deploy. All the container will be destroyed in favor of the new image. When this happens, the nodes within couchdb cluster will disconnect from each other and doesn't seems to recover themself automatically. Any suggestion for this problem?

[wohali]

If you are preserving your etc/ and data/ directories between restarts, then this shouldn't happen. Please confirm.

For your other problem as reported in this ticket, please open a ticket against CouchDB itself. I'm not going to change the Dockerfile until CouchDB has a way to separate read-only and read-write configuration files.

[wohali]

@endyjasmi One option is that, instead of mounting the entire etc/ directory you can just mount individual .ini files. We've just updated the README file to make this clearer.

Remember that CouchDB will always write to the very last file in your config file chain, which is often the alphabetically-sorted last .ini file under etc/local.d/.

So, you can mount your shared secret files ahead of these, read only, then mount something like etc/local.d/999-writeable-ini-file.ini from a different writeable share. Does this work for you?

[endyjasmi]

@wohali After I -setcookie monster to all the instance, it will auto recover the cluster after the configuration update.

On the config and secret side of thing, I am currently mounting the .ini on file level rather than on etc/ directory level.

As for mounting the shared config as read only then mount a etc/local.d/999-writeable-ini-file.ini, I think this will works. Before that though, we will need to update this line to skip chmod on the mounted config or secret .ini.

couchdb-docker/2.1.1/docker-entrypoint.sh

Line 33 in 1f85f08

chmod 664 /opt/couchdb/etc/*.ini

Personally though, I thought we can maybe fix /opt/couchdb/etc/local.ini as readonly and all writable .ini should contain within /opt/couchdb/etc/local.d/* directory. Then we can skip chmod on /opt/couchdb/etc/local.ini.

[wohali]

What I'm going to do is just have the chmod/chown script do a -f, which means if it fails it won't complain and won't error out. That way we both get what we want.

[wohali]

Closed with 45b9dd1 in dev - this will be added to the latest released image when the next version of CouchDB is released (shortly).

[sepbot]

Not sure if things have changed since the last post, but I could not get it to work when mounting a Docker Swarm config to /opt/couchdb/etc/local.d/couch.ini. The container exits immediately with status of 1 and no logs produced. Using 3.1.0.

This workaround resolved it for me:

command: '/bin/bash -c "cp -f /couch-v1 /opt/couchdb/etc/local.d/couch.ini && /opt/couchdb/bin/couchdb"'
configs:
  - couch-v1

[SSARCandy]

Not sure if things have changed since the last post, but I could not get it to work when mounting a Docker Swarm config to /opt/couchdb/etc/local.d/couch.ini. The container exits immediately with status of 1 and no logs produced. Using 3.1.0.

This workaround resolved it for me:

command: '/bin/bash -c "cp -f /couch-v1 /opt/couchdb/etc/local.d/couch.ini && /opt/couchdb/bin/couchdb"'
configs:
  - couch-v1

This workaround will not set variables properly, i.e. NODENAME.
It is because overriding command will cause the docker-entrypoint.sh not entering the if:

couchdb-docker/3.1.1/docker-entrypoint.sh

Line 27 in 03af73d

if [ "$1" = '/opt/couchdb/bin/couchdb' ]; then

Instead of overriding command, we can overwrite entrypoint:

services:
  couchdb:
    # ...skip
    entrypoint: /bin/bash -c "cp -f /couchdb_conf /opt/couchdb/etc/local.d/couch.ini && tini -- /docker-entrypoint.sh /opt/couchdb/bin/couchdb"
    configs:
      - couchdb_conf

configs:
  couchdb_conf:
    file: ./config.ini