Fig data-only containers

[ajrouvoet]

Using this simple configuration:

data:
  image: base/arch
  volumes:
    - /var/lib/mysql
  command: true

mysql:
  build: .
  volumes_from:
    - data

I would expect that the mysql data is persisted on the data-only container 'data'. But writing anything to /var/lib/mysql is NOT persisted across runs. Doing this with plain docker does work.

Am I overlooking something, or is this either a bug or a feature of fig?

The fig commands I'm trying to use are:

fig run mysql init
fig run mysql start

The mysql image has an entrypoint that takes init and start command. The first one populates /var/lib/mysql, the second one starts the mysql server with /var/lib/mysql as it's datadir. Both run, but the second one fails because /var/lib/mysql is empty.

The following is a minimal example of it failing:

fig run mysql /bin/bash -c "touch /var/lib/mysql/testfile && ls -l /var/lib/mysql"
fig run mysql /bin/bash -c "ls -l /var/lib/mysql"

Where the first ls -l shows that the file was created, where as the second one returns empty.

Using docker 1.3.0 and fig 1.0.0

[dnephin]

Could you include the fig commands you're using as well? I believe I've seen this work

[ajrouvoet]

Ofcourse, added both my goal and the minimal test. Thank you for taking a look so soon!

[ajrouvoet]

The minimal example can be made more minimal ofcourse, since you don't really need the mysql image. Replacing the build: . with image: base/arch would work as well.

[dnephin]

I think I was able to reproduce this. I had to run fig up -d data to get it to recreate the container, otherwise it will just use the last container (which didn't have the volume). After doing that it worked.

I think docker will also silently ignore volumes if the directory doesn't exist in the container.

I ran docker inspect <container id> on the containers to see which volumes were attached.

[mhubig]

I had the same issues, updating to 1.0.1 fixed it ...

[ajrouvoet]

Confirming that @dnephin's proposed solution worked for me as a workaround.

[TwoD]

I'm having a similar issue as fig rebuilds my db container every time I fig up some other container linking to it. However, it does not rebuild any containers linked to by db in turn.

I've got all my database files mounted as host volumes so I don't really lose any data when this happens, but it's still annoying as other containers rely on db too.

[wid]

On fig up the doc tells
By default if there are existing containers for a service, fig up will stop and recreate them (preserving mounted volumes with volumes-from), so that changes in fig.yml are picked up.

I think we should add a parameter to fig.yml: no-recreate. This parameters should tell that fig up must never recreate this container.

[SvenDowideit]

please don't use --no-something style negative flags - --no-recreate=false is hard to write about - and yes, it will happen :)

[TwoD]

A parameter like that would not fix the underlying bug, which assumes linked containers should be recreated too when specifying only one or a few containers to up. What would be the reason to do that?

I get that Fig recreates changed containers, and/of those you specify in the fig up command, but why recreate any other containers those are linking to when they don't need to? Especially without also recreating other containers also linking there. (Please don't make it do that, it would be even worse and make fig up servicename pointless.)

That makes no sense and breaks connections from other existing containers linked to the same container as those are not recreated. Recreating one website container forcibly recreates my DB container and thus breaks all other sites or node using the same DB containers until I manually recreate those too.

[dnephin]

This issue is quite similar to #447, I've proposed a solution there (which doesn't use negative flags).

[wid]

Actually I don't think #447 is the same need but I agree on negative flags.

Maybe we could tag a container as "permanent: true" or "data-container: true" so that this container is never changed or re-created by fig and require manual docker action.

[rocketraman]

On a related note, fig up fails when re-run with a truly minimal data container image such as tianon/true that does not have the /bin/echo executable:

datacontainer:
  image: tianon/true
  volumes:
    - /foo/

The first time it works fine, but then the second time fig up is run:

 $ fig up
Recreating testdata_datacontainer_1...
Cannot start container e0e8a126d57a1a356470df3c0f21b78ce6b4bd9612f3d63a9b32a18345b0dc1d: exec: "/bin/echo": stat /bin/echo: no such file or directory

[aanand]

@rocketraman see #516

[rocketraman]

rocketraman see #516

Thanks!

[hxu]

Sorry, ignore the below -- turns out I was inspecting the wrong container due to a naming error. D'oh! Though I also still have the issue of getting volumes from data only containers to mount correctly in other containers. Still investigating the exact nature of the problem.

I'm not sure if this is related, but my data only containers defined in fig don't even seem to have their volumes mounted correctly.

My image Dockerfile looks like this:

FROM debian:7.7
VOLUME ["/logs"]

and the fig config looks like this:

logdata:
  image: myimage
  command: echo

when I run with fig up (after building and tagging) and inspect, there is no Volumes key or VolumesRW key in the main inspect object.

Running with docker run myimage results in what looks like a correct volumes block:

    "Volumes": {
        "/logs": "/var/lib/docker/vfs/dir/7fcd1c12f6b8e15dcd29748c27c3e390359a8903bfa75c92bfee8
4df165cc288"
    },
    "VolumesRW": {
        "/logs": true
    }

I also noticed that in the container created by fig, the command that is being run as shown by inspect doesn't look correct:

        "Cmd": [
            "/bin/sh",
            "-c",
            "#(nop) VOLUME [/logs]"
        ],

[hxu]

I also confirm that @dnephin's workaround works for me. @mhubig I have version 1.0.1 but still have the issue.

[thaJeztah]

Reading this back, I don't think data-only containers even need a command. At least since Docker 1.4, volumes are initialised at docker create (release notes).

So having an option to only create a service without starting would solve at least some cases.

[aanand]

@thaJeztah if we get #630 merged, that would be possible with initial_scale: 0

[thaJeztah]

@aanand exactly; and it would prevent having to add "magic" no-op commands, so a much cleaner solution.

[aanand]

@thaJeztah hmmm, actually that's not right: initial_scale defines how many containers should be created.

[drags]

Ran into a similar issue while debugging with a coworker and wanted to leave this comment in case it helped anyone else.

One of our users had an issue with data not persisting between fig runs. We found that the docker daemon running inside of boot2docker seemed to be ignoring volume configs completely (whether we used volumes_from with a data container, or even a static volume mount). The container that wound up getting spawned would always have a docker-daemon created volume mount (similar to when a Dockerfile contains a VOLUME line and no other volume information is provided at run time, and that image did contain a VOLUME line for that dir). Also, it would create a new version of the data container each run, rather than re-using the initially created one.

While we didn't find the root cause of the issue we noticed that the docker client (on the OSX side) was 1.3.2, whereas the docker server was running 1.4.0. After upgrading boot2docker and getting the OSX docker client up to 1.4.0 the volumes are mounting as expected and data is persisting.

Hope this helps someone with a similar issue.

[aanand]

@drags Docker 1.4.0 causes an issue with Fig where volumes aren't mounted - you should upgrade to 1.4.1 as soon as possible.

[Starefossen]

I'm having a hard time getting persistent volumes from my data only containers. I had to use @dnephin's workaround fig up -d data in order for fig/docker to keep the volume persistent between fig up.

Ubuntu 14.04, Docker version 1.3.3 (build d344625), Fig version 1.0.1 (master@55095e)

[dnephin]

#711 has been merged which I believe resolves this issue. It will be in the next release

[zhengjia]

@dnephin Hi, I just tried a build from master
(55095ef) and docker 1.4.1, and it appears the issue is not fixed; if I do a fig up -d to rebuild containers, the data volume is not persisted. Also tried fig up -d data and it seems it doesn't work in master either.

[dnephin]

@zhengjia Ok, it sounds like there are still some issues with 1.4.1 compatibility. Tracking it down in #711

[pwaller]

Is it possible that the no-op is no longer required since moby/moby#8942 was fixed in moby/moby#9089 ? That would mean that starting the intermediate container would be unnecessary.

[pwaller]

I've just hit this when trying to run a single-binary go container.

[caleb]

Is there a workaround to use data containers with docker-compose?

Would downgrading docker (I'm on 1.4.1 now) help?

I'd like to be able to use data containers but they get wiped every time I do a docker-compose up even though I haven't changed anything in my docker-compose.yml file.

If not I can use host mounted volumes for now.

Thanks for the hard work on docker-compose I really feel this will change how I develop forever.

[Starefossen]

Using docker v1.3.3 and docker-compose@master appears to have working data-only containers.

[caleb]

I tried Docker 1.4.1 again, and I found a bug in one of my containers that was wiping the data container on every start... I have fixed that bug and now it seems like everything is working!

Apparently there was just a misfortunate bit of googling that led me to believe this problem was something other than my own! Thanks for the help!

[kojiromike]

@aanand said

Docker 1.4.0 causes an issue with Fig where volumes aren't mounted - you should upgrade to 1.4.1 as soon as possible.

Is that #723?

[aanand]

@kojiromike Yes.

[aanand]

OK, this thread is a bit of a jumble of issue reports, and it's not clear if there's an actual bug here or not. I'm going to close it until someone can come up with a reproducible error.