Since docker 1.3.0 / fig 1.0.0 volumes are not mounted when fig up'ing existing containers #622

Closed
morvans opened this Issue Nov 6, 2014 · 53 comments

Projects

None yet
@morvans
morvans commented Nov 6, 2014

Since docker 1.3.0 / fig 1.0.0 (I upgraded both at the same time), I cannot fig up my projects without fig rm'ing before.

If I do fig up on already existing (stopper) containers, volumes specified in fig.yml aren't remounted (default docker mounts instead, so empty dirs).

fig rm && fig up recreates everything correctly but for locally built containers this is pretty time consuming.

@mindo
mindo commented Nov 6, 2014

This also happens with docker 1.3.1 and fig 1.0.1.
Even more weird is that if a container is ran using fig:

fig run mycontainer bash

The volumes are mounted just fine.

@dnephin
Member
dnephin commented Nov 6, 2014

Are you using volumes or volumes_from? A sample fig.yml may help in debugging.

@mindo
mindo commented Nov 6, 2014

My fig.yml:


haproxy:
    build: haproxy
    ports:
        - "80:80"
        - "443:443"
    links:
        - ngx

ngx:
    build: ngx-php
    volumes:
        - var/logs/:/var/log
        - ngx-php/etc/:/etc/nginx
        - myapp/:/var/www
    links:
        - percona
        - memcached

memcached:
    build: memcached

percona:
    build: percona
    ports: 
        - "3306:3306"
    volumes:
        - var/mysql/:/var/lib/mysql

All folders defined as volumes are on the same level as fig.yml.

@dnephin
Member
dnephin commented Nov 8, 2014

This seems correct to me. Some things you could try to debug it:

  • docker inspect the containers when they're up to see what it's doing
  • docker will silently ignore a volume if the path doesn't exist, it's possible it's not doing these relative paths correctly. Maybe try adding a ./ prefix
  • paths in the fig.yml are relative to the current working directory, so if you do something like fig -f somepath/fig.yml it might not work (because the paths will be wrong
@nireak
nireak commented Nov 9, 2014

Perhaps it's a silly question.... but are you using a Virtual Machine to run docker and fig ? If so, do the paths referenced in your "volumes" directives exists inside the VM?

@mindo
mindo commented Nov 11, 2014
cd /to/folder/with/fig.yml
  • fig up
    "Volumes": {
        "/etc/nginx": "/var/lib/docker/vfs/dir/6b04e7ee535de76bfd359ad1c08fbe35db3e435c2301c464a7390ba67cfefe63",
        "/var/log": "/var/lib/docker/vfs/dir/17b86c34458c046e15709731c3e3c760e2c766ce113c5b2fd0f7de9aafdcc17b",
        "/var/www": "/var/lib/docker/vfs/dir/11a480315b57b48c6b7f5abaada5741a5656e5febdd3365e40e62f04e94f3be2"
    },

  • fig run ngx bash
    "Volumes": {
        "/etc/nginx": "/mnt/HG/docker/ngx-php/etc",
        "/var/log": "/mnt/HG/docker/var/logs",
        "/var/www": "/mnt/HG/docker/myapp"
    },

I am running this directly on my notebook (ubuntu 14.10, no virtual machine).
I also tried:

ngx:
    build: ngx-php
    volumes:
        - ./var/logs/:/var/log
        - ./ngx-php/etc/:/etc/nginx
        - ./myapp/:/var/www

and

ngx:
    build: ngx-php
    volumes:
        - /mnt/HG/docker/var/logs/:/var/log
        - /mnt/HG/docker/ngx-php/etc/:/etc/nginx
        - /mnt/HG/docker/myapp/:/var/www

and neither worked.

@dnephin
Member
dnephin commented Nov 11, 2014

@mindo this looks like a similar issue to #613. fig up will copy volumes from the previous container when it recreates them. I think if you do: fig stop && fig rm --force -v ngx it should remove those volumes and the next time you create it, it should mount the correct host volumes.

@mindo
mindo commented Nov 11, 2014

@dnephin yes, doing

fig stop && fig rm && fig build

solves it but only until the next reboot/service docker restart.

Test run:

fig rm
fig build
fig up

Volumes are mounted just fine.

fig stop
fig up

Volumes still work.

fig stop
service docker stop
service docker start
fig up

Volumes fail to mount! But running:

fig run ngx

The Volumes work without problems.

@schickling

Same here. Any progress on this?

@morvans
morvans commented Dec 5, 2014

FWIW, the issue seems fixed for me with docker 1.3.2 / fig 1.0.1

@schickling

I'm using docker 1.3.2 / fig 1.0.1. Still doesn't work for me.

@morvans
morvans commented Dec 8, 2014

@schickling : yeah, spoke too soon, it still doesn't work for me either eventually.

@bercab
bercab commented Dec 12, 2014

Hello, I have the same issue. Can't bind-mount volumes with fig run nor fig up.

  • fig versions tested: 0.3.2, 1.0.0, 1.0.1 and master
  • docker version: 1.4.0
  • docker-py versions tested: 0.5.3, 0.6.0 (with fig master)

Running with fig --verbose, I see volume is correctly parsed when passing to client.start():

 binds={u'/test': {u'bind': u'/test', u'ro': False}}

But docker inspect shows volume is not bind-mounted:

    "Volumes": {
        "/test": "/var/lib/docker/vfs/dir/fa2cbf2c4b3a560b9e99dd07b68968b99260a717e4ab4f6371dc88e7aa805414"
    },

I think is not a problem of fig, but a problem of docker-py with newer versions of docker.
According to docs, binds dict seems correct: http://docker-py.readthedocs.org/en/latest/volumes/
Maybe docker-py is breaking its API?

Looking at docs link above, seems that client.create_volume() spects a list of volumes (ej: ['test', ]), but fig is passing a dict:

 volumes={u'/test': {}}

Also seems other projects are running the same issue:
saltstack/salt#14089

@s2ugimot

I've faced the exact same issue using

  • Ubuntu 14.04
  • docker 1.4.0, installed from ppa (lxc-docker-1.4.0)
  • fig 1.0.1

Reverting docker version back to lxc-docker-1.3.3 solved the problem for now.

@dnephin
Member
dnephin commented Dec 14, 2014

@s2ugimot for docker 1.4.0 see #723

@anthonylouisburns

has anyone fixed this

@dnephin
Member
dnephin commented Dec 15, 2014

I believe #711 will resolve the original issue

@anthonylouisburns

Awesome - I'm new to github - how do I know what version #711 is in or whether it is in at all

@ihrwein
ihrwein commented Dec 16, 2014

I have the same problem:

  • Ubuntu 14.04
  • docker 1.4.0, installed from ppa (lxc-docker-1.4.0)
  • fig 1.0.1
@loa
loa commented Dec 16, 2014

Same issue

$ fig --version
fig 1.0.1
$ docker version
Client version: 1.3.3
Client API version: 1.15
Go version (client): go1.4
Git commit (client): d344625
OS/Arch (client): darwin/amd64
Server version: 1.4.0
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 4595d4f
@jaycyb
jaycyb commented Dec 19, 2014

+1 also reported here #443

docker 1.3.1 + fig 1.0.1 works fine
docker 1.4.0 + fig 1.0.1 does not

@gquemener

From my experience, docker 1.4.1 + fig 1.0.1 does not work also: #443 (comment)

@cpuguy83

@jaycyb @loa Bug is in Docker 1.4.0, please use Docker 1.4.1 if you want to use 1.4.x.

@gquemener

This does not solve the issue for me :-/

@cpuguy83

@gquemener See my note on the other issue, you're still using 1.4.0 on the server.

@loa
loa commented Dec 22, 2014

@cpuguy83 works great! Thanks!

@dreamcat4

Same experience as @gquemener here: docker 1.4.1 + fig 1.0.1. adding host volumes does not work unless i do a 'fig rm' first. Not so-sure what happens on restarts. #723

@dmp1ce
dmp1ce commented Jan 7, 2015

I also experienced the same thing. For one of my containers the 'fig up' would not mount host volumes. However it would mount if I used 'fig run --rm mycontainer bash'. Running 'fig rm mycontainer' fixed it.

@vh
vh commented Jan 29, 2015

Same issue: docker 1.4.1 + docker-compose 1.1.0-rc1. Fixed it to run 'docker rm container' before 'docker-compose up'

@jgillich
jgillich commented Feb 6, 2015

+1, I just wasted a lot of time on this.

Docker version 1.3.2, build 39fa2fa/1.3.2
fig 1.0.1
CentOS Linux release 7.0.1406 (Core)
@nomadster

+1 This "feature" actually makes docker+fig a tool to make a deploy in just two days :)

Docker version 1.3.2, build 39fa2fa/1.3.2
fig 1.0.1
CentOS Linux release 7.0.1406 (Core)
@ddliu
ddliu commented Feb 11, 2015

+1

fig 1.0.1
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8
@aanand
Contributor
aanand commented Feb 11, 2015

Can anyone produce a minimal failing case? I'm currently unable to reproduce. Here's my test case:

$ cat docker-compose.yml
test:
  image: busybox
  command: cat /data/hello.txt
  volumes:
    - ./data:/data

$ docker-compose --version
docker-compose --version

$ docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): darwin/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

$ docker-compose rm --force
$ mkdir -p data
$ echo "hello world" > data/hello.txt

$ docker-compose up
Creating 622_test_1...
Attaching to 622_test_1
test_1 | hello world
622_test_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)

$ docker inspect -f '{{.Volumes}}' 622_test_1
map[/data:/Users/aanand/work/docker/compose-bugs/622/data]

$ docker-compose up
Recreating 622_test_1...
Attaching to 622_test_1
test_1 | hello world
622_test_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)

$ docker inspect -f '{{.Volumes}}' 622_test_1
map[/data:/Users/aanand/work/docker/compose-bugs/622/data]
@cpuguy83

@aanand 1.5 has some fixes in it that would resolve this, and maybe one more in 1.5.1 for some fig users.

@dnephin
Member
dnephin commented Feb 11, 2015

I suspect (based on the early comments on this thread, and my personal experience helping people new to fig) that this is often an issue with copying old volumes during fig up instead of creating things from scratch.

If you have an existing container with data volumes, and you change the content of those volumes, when you fig up, you'll still get the old data, which makes it seem like the volume is not mounted. The workaround of fig rm seems to have worked for a few people.

I think #858 helps with this.

There is still a case where the volume path hasn't changed but the contents or structure of the data volume have changed. That one is harder to deal with and maybe just needs to be called out better in the docs.

@nomadster

@dnephin it's not my case.
fig stop && fig up

doesn't mount the volume. If i go to the folder where the volume should be mounted I don't have any data at all.

@dnephin
Member
dnephin commented Feb 11, 2015

@nomadster can you docker inspect <container name> and include a link to a pastebin with the output?

fig stop && fig up is actually what I described, what if you fig stop; fig rm --force; fig up (this will remove data volume containers).

@nomadster

Cannot reproduce right now, I've switched to docker 1.1.2 and fig 0.5.2 to be able to work :)

@morvans
morvans commented Feb 13, 2015

@aanand, I've got a scenario with fig where volumes aren't re-mounted correctly upon fig up when docker daemon has been restarted between stop/kill and up. I can provide you (privately preferably) the full log with docker inspect outputs. It forces me to fig rm my containers every morning after booting my workstation (event if containers where "gracefully" fig-stopped before the reboot or docker daemon restart)

@aanand
Contributor
aanand commented Feb 13, 2015

@morvans Does the problem only surface when the docker daemon has been restarted? If so, that's a separate issue, and it's worth investigating whether that behaviour can be reproduced using Docker without Fig.

Can you produce a failing case with private information stripped out?

@morvans
morvans commented Feb 13, 2015

Ok I'll try to get back to you with a stripped down fig.yml and/or a test case involving docker alone.

@dmitry-saritasa

I can't mount relative folder through fig on centos7:

[root@localhost using-fig]# uname -a
Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost fig-master]# fig --version
fig 1.0.1

[root@localhost fig-master]# docker version
Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa/1.3.2
OS/Arch (client): linux/amd64
Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 39fa2fa/1.3.2
[root@localhost fig-master]#

fig.yml:

/home/apps/using-fig/fig.yml
es:
    image: library/elasticsearch:1.4.3
    volumes:
        - elasticsearch:/data
    ports:
        - "9200:9200"
        - "9300:9300"

[root@localhost using-fig]# pwd
/home/apps/using-fig

[root@localhost using-fig]# ls -l
total 4
drwxr-xr-x. 3 root root 41 Feb 13 17:30 elasticsearch
-rwxr-xr-x. 1 root root 141 Feb 13 17:16 fig.yml
drwxr-xr-x. 2 root root 59 Feb 13 15:32 web

[root@localhost using-fig]# ls -l elasticsearch/
total 4
drwxrwxrwx. 2 root root 6 Feb 13 16:59 data
-rwxr-xr-x. 1 root root 49 Feb 13 16:23 elasticsearch.yml

so when I run fig up -d

[root@localhost using-fig]# fig up -d
Recreating usingfig_es_1...

[root@localhost using-fig]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e7f8547a4ad1 library/elasticsearch:1.4.3 "elasticsearch" 5 seconds ago Up 4 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp usingfig_es_1

[root@localhost using-fig]# docker exec -ti usingfig_es_1 bash
root@e7f8547:/# ls -l /data/
ls: cannot open directory /data/: Permission denied

however if I change fig to absolute path

fig.yml:

/home/apps/using-fig/fig.yml
es:
    image: library/elasticsearch:1.4.3
    volumes:
        - /elasticsearch/data:/data
    ports:
        - "9200:9200"
        - "9300:9300"

everything works just fine

@nomadster

@dnephin I've managed to find some time to check for the bug:

$ docker version
Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa/1.3.2
OS/Arch (client): linux/amd64
Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 39fa2fa/1.3.2
----
$ fig --version
fig 1.0.1

I've deleted every single image I had and built them from scratch then run fig up (some container use links, I can provide the fig.yml privately if needed).
Everything is ok:

$ nsenter -m -u -p -n -i -t $PID
container$ ls /var/lib/mysql
my_db_from_volume_1
...
my_db_from_volume_N

Now I fig stop all the containers and fig up them again (not running fig rm --force)
And the bugs doesn't happen: everything just works fine.
All the data is where I wanted it to be.

Just in case this will happen again later on I've pasted the output of the command you asked here

@dnephin
Member
dnephin commented Feb 18, 2015

Ok, this sounds a lot like re-using old volumes, and might be fixed by #858

@morvans
morvans commented Feb 18, 2015

@nomadster, if you restart the docker daemon between fig stop and fig up, are the volumes re-mounted correctly ?

@nomadster

@morvans

fig stop
systemctl stop docker #Stopped Docker Application Container Engine
systemctl start docker #Started Docker Application Container Engine.
fig up -d container1 #(ups other container via links, same command as above)

Some of the container doesn't start correctly. I suppose because the volumes are not mounted correctly.

@morvans
morvans commented Feb 26, 2015

FWIW, I updated to docker 1.5.0 and the issue seems to be gone.
I'm able to fig up stopped containers without fig rm them beforehand, and volumes are re-mounted correctly.

@UnknownHero

yes, 1.4.0 not work , but 1.5.0 work fine

@Vrakfall
Vrakfall commented Mar 3, 2015

@morvans Yeah, I think the problem was mainly coming from docker itself and the last big update was meant to solve big mounting issues.

@aanand
Contributor
aanand commented Mar 3, 2015

I'm closing this issue as it seems to be a confusion of different reports - if anyone can produce a minimal failing case, please open a new one.

@aanand aanand closed this Mar 3, 2015
@CWSpear
CWSpear commented Jan 28, 2016

I'm seeing some behavior that seems related and I have a reduced test case here: #2776

@pavankumarkatakam

Hi, im experienceing with same problem, after restarting my server the containers are starting but not mounting the volumes, the changes i have done to my server is updated to Ubuntu server 15.10 and docker-compose version is 1.6.0, but when i goto my docker-compose folder and start them then volumes are mounting correctly.

@dnephin
Member
dnephin commented Feb 16, 2016

You might be hitting #2912

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment