Docker compose not using external named volume on Mac

[sivagollapalli]

# docker-compose.yml
version: '3'
services:
  db:
    image: postgres
    volumes:
      - data:/var/lib/postgres/data
  web:
    build: .
    command: bundle exec rails s -p 3000 -b '0.0.0.0'
    volumes:
      - .:/myapp
    ports:
      - "3000:3000"
    depends_on:
      - db

volumes:
  data:
    external: true

I have created a volume with options as docker volume create --name=data. Then I run docker-compose up which creates a volume apart from data as shown below.

DRIVER              VOLUME NAME
local               2293833c2781bc0c57baad7add8af771086b8c693b2346030cfb59e04b8fcc22
local               data

docker version is 17.03.1-ce, build c6d412e and docker compose version is 1.11.2, build dfed245.
OSX version is 10.12.5

Could you let me know what went wrong?

[shin-]

Can you try upgrading Compose to the latest (1.14.0) version and check if the problem persists?

[sivagollapalli]

@shin- Yes it is still persists in 1.14 version also

[shin-]

I'm unable to reproduce this locally. Are you sure that volume is created by Compose? They're usually given human-readable names, like project_data. You might want to create an issue on the docker/for-mac repo as well.

[sivagollapalli]

@shin- Yes the volume is created by compose only. Moreover it is not giving human-readable names like you specified. As you said I would raise this issue on docker/for-mac also. Thanks for help.

[sivagollapalli]

@shin-

Here are the detail log details when I do docker-compose up

Creating demo_db_1 ...
Creating demo_db_1 ... done
Creating demo_web_1 ...
Creating demo_web_1 ... done
Attaching to demo_db_1, demo_web_1
db_1   | The files belonging to this database system will be owned by user "postgres".
db_1   | This user must also own the server process.
db_1   |
db_1   | The database cluster will be initialized with locale "en_US.utf8".
db_1   | The default database encoding has accordingly been set to "UTF8".
db_1   | The default text search configuration will be set to "english".
db_1   |
db_1   | Data page checksums are disabled.
db_1   |
db_1   | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1   | creating subdirectories ... ok
db_1   | selecting default max_connections ... 100
db_1   | selecting default shared_buffers ... 128MB
db_1   | selecting dynamic shared memory implementation ... posix
db_1   | creating configuration files ... ok
db_1   | running bootstrap script ... ok
db_1   | performing post-bootstrap initialization ... ok
db_1   | syncing data to disk ... ok
db_1   |
db_1   | WARNING: enabling "trust" authentication for local connections
db_1   | You can change this by editing pg_hba.conf or using the option -A, or
db_1   | --auth-local and --auth-host, the next time you run initdb.
db_1   |
db_1   | Success. You can now start the database server using:
db_1   |
db_1   |     pg_ctl -D /var/lib/postgresql/data -l logfile start
db_1   |
db_1   | ****************************************************
db_1   | WARNING: No password has been set for the database.
db_1   |          This will allow anyone with access to the
db_1   |          Postgres port to access your database. In
db_1   |          Docker's default configuration, this is
db_1   |          effectively any other container on the same
db_1   |          system.
db_1   |
db_1   |          Use "-e POSTGRES_PASSWORD=password" to set
db_1   |          it in "docker run".
db_1   | ****************************************************
db_1   | waiting for server to start....LOG:  could not bind IPv6 socket: Cannot assign requested address
db_1   | HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
db_1   | LOG:  database system was shut down at 2017-07-17 11:59:25 UTC
db_1   | LOG:  MultiXact member wraparound protections are now enabled
db_1   | LOG:  database system is ready to accept connections
db_1   | LOG:  autovacuum launcher started
db_1   |  done
db_1   | server started
db_1   | ALTER ROLE
db_1   |
db_1   |
db_1   | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
db_1   |
db_1   | LOG:  received fast shutdown request
db_1   | LOG:  aborting any active transactions
db_1   | LOG:  autovacuum launcher shutting down
db_1   | LOG:  shutting down
db_1   | waiting for server to shut down....LOG:  database system is shut down
db_1   |  done
db_1   | server stopped
db_1   |
db_1   | PostgreSQL init process complete; ready for start up.
db_1   |
db_1   | LOG:  database system was shut down at 2017-07-17 11:59:27 UTC
db_1   | LOG:  MultiXact member wraparound protections are now enabled
db_1   | LOG:  database system is ready to accept connections
db_1   | LOG:  autovacuum launcher started
web_1  | => Booting Puma
web_1  | => Rails 5.1.0 application starting in development on http://0.0.0.0:3000
web_1  | => Run `rails server -h` for more startup options
web_1  | Puma starting in single mode...
web_1  | * Version 3.9.1 (ruby 2.3.3-p222), codename: Private Caller
web_1  | * Min threads: 5, max threads: 5
web_1  | * Environment: development
web_1  | * Listening on tcp://0.0.0.0:3000
web_1  | Use Ctrl-C to stop

Since I provided external data volume I hope it should not display Creating demo_db_1 in logs. Hope this would help in debugging. Let me know if you need any info from my side. Thanks.

[shin-]

demo_db_1 is the DB's container, it should absolutely be created. Can you try running docker-compose --verbose up and share those logs as well?

[sivagollapalli]

@shin- Added as a gist over here

[shin-]

As you can see in those logs, data is recognized as external and no other volume is created.

compose.cli.verbose_proxy.proxy_callable: docker inspect_volume <- ('data')
compose.cli.verbose_proxy.proxy_callable: docker inspect_volume -> {u'Driver': u'local',
 u'Labels': {},
 u'Mountpoint': u'/var/lib/docker/volumes/data/_data',
 u'Name': u'data',
 u'Options': {},
 u'Scope': u'local'}
compose.volume.initialize: Volume data declared as external. No new volume will be created.

There is no reasonable indication that this volume you're seeing is being created or used by Compose.

[sivagollapalli]

@shin- I am following this docker compose example. I am trying to make my data persist. Upon docker-compose down I am losing my data but upon docker-compose stop I am not losing any data. What would be the reason for my data lose? Thoughts, please?

[shin-]

Right, I see the issue now. The Dockerfile for the official postgres image already defines an anonymous volume on that container path, which conflicts with the one you're declaring yourself. The way I would go about it is use the $PGDATA environment variable to define an alternative location for the data, and link your data volume there. For example:

version: '3'
services:
  db:
    image: postgres
    environment:
      PGDATA: /var/postgres_data
    volumes:
      - data:/var/postgres_data
  web:
    build: .
    command: bundle exec rails s -p 3000 -b '0.0.0.0'
    volumes:
      - .:/myapp
    ports:
      - "3000:3000"
    depends_on:
      - db

volumes:
  data:
    external: true

The other alternative would be to build your own postgres image, but that's probably unnecessarily complicated at this stage.

[sivagollapalli]

@shin- Still it creates anonymous volume on docker-compose up. On Linux, it is not creating anonymous volume. If it is Postgres Dockerfile issue then I think it should create an anonymous container in Linux as well right? Thoughts, please?

FYI, in my docker-compose.yml I have used /Users/sivagollapalli/work/postgres instead of /var/postgres_data . I hope it doesn't make any difference.

[shin-]

I'm testing this on Linux - I can verify that the anonymous volume is created there as well. It doesn't matter as long as you're not storing any data in it (setting PGDATA to a different directory ensures that).

Still, if you're that worried, just build your own postgres image without anonymous volumes and use that.

FYI, in my docker-compose.yml I have used /Users/sivagollapalli/work/postgres instead of /var/postgres_data . I hope it doesn't make any difference.

Doesn't matter - it's the path inside the container, and it will be created regardless.

[MikeMitterer]

I have the same problem - added my inspect-output in docker/for-mac#1856

[MikeMitterer]

@shin- It doesn't help if you are testing on Linux - I'm sure it works on Linux. On Mac Docker runs in it's own VM. I think this config is responsible for the problem.