How to refresh symlinks mounted to a docker data volume?

[carn1x]

I'm running a docker data volume as per the method documented at https://docs.docker.com/userguide/dockervolumes/#creating-and-mounting-a-data-volume-container

One of the volumes I'm mounting is a symlinked folder linking to the latest version of a codebase. Currently I destroy the container and run it again to refresh the mounted folder but is there a more graceful method to deal with this?

So I have the following directory structure:

drwxr-xr-x release_2015-04-08-145116_HEAD/
drwxr-xr-x release_2015-04-08-145223_HEAD/
drwxr-xr-x release_2015-05-20-160620_HEAD/
lrwxrwxrwx  live -> /var/www/mysite/release_2015-05-20-160620_HEAD/

And the docker data volume runs with docker run --name mydata -t -v /var/www/live:/data/www mydata

Destroying the container has the downside of losing the non external data of course.

Another method I suppose would be to mount a level above the symlink although then really I'm exposing more to the data container than I really need to and it seems sloppy.

Thanks for any ideas.

[cpuguy83]

This won't work because docker is resolving this symlink when it mounts it in.

[carn1x]

OK, so my only choice is to mount the parent?

Alternatively would it be possible to image/run the original data container so that the --volume switch can be reapplied but the internally stored changes are maintained?

[cpuguy83]

Hmm, this might work.

[carn1x]

Glad to hear you're confident :D, I'll give it a whirl then, thanks!

[thaJeztah]

I'm going to close this issue, because this is more of a "question" than a bug in Docker, but feel free to continue the discussion.

[C-Duv]

I am experiencing similar issue using data volume container and symlinks but not sure if @carn1x's workaround would work for me...

It seems to me that Docker de-references symlinks pointing to data volume container at second run.

Goal: I wanted to set up an application container and store the applications' data in a data volume container.

Installation of softwares is done in Dockerfile, and I have a small script (init.sh) that moves configurations files and data folders to specific locations (eg. /conf, /data) and place strategic symbolic links in the application container.
I find it cleaner to let the softwares uses their default config/storage locations, move it where I want it to (id. /conf, /data) and put symlink in-place.

The data volume container have the volume /data, so when I run application container with the data volume container using --volumes-from for the first time, the data volume container should be empty.
My init.sh script will move some stuff into it and create the symlinks.

At second run, symlinks in application container should still be there and data volume container should contain the data produced by the application.

The issue is that content of /data (path to my volume) seems to be copied to the application container, overwriting the symbolic link /opt/app1/storage.

Steps to Reproduce:

Here is "context creation" code (this will create Dockerfile, the run.sh docker container start point and the demo.sh that builds and highlights the issue) :

#!/bin/bash

# Create files
cat >> Dockerfile <<EOT
FROM debian:jessie

RUN apt-get update && apt-get install tree

RUN mkdir -p /opt/app1/conf
RUN echo "foo" > /opt/app1/conf/main.cfg
RUN mkdir -p /opt/app1/storage
RUN echo "bar" > /opt/app1/storage/datafile-bar
RUN mkdir -p /opt/app2

RUN mkdir -p /conf
# Will also be done from run.sh for when data-volume container is used
RUN mkdir -p /datas/app1

COPY run.sh /run.sh
RUN chmod u+x /run.sh
EOT

cat >> run.sh <<EOT
#!/bin/bash

mkdir --parents /datas/app1

if [ ! -e "/conf/main.cfg" ] ; then
    echo "Symlink to \"/conf/main.cfg\": CREATING..."
    mv --no-target-directory /opt/app1/conf/main.cfg /conf/main.cfg
    ln --symbolic --no-target-directory /conf/main.cfg /opt/app1/conf/main.cfg
else
    echo "Symlink to \"/datas/app1/storage\": OK"
fi

if [ ! -e "/datas/app1/storage" ] ; then
    echo "Symlink to \"/datas/app1/storage\": CREATING..."
    mv --no-target-directory /opt/app1/storage /datas/app1/storage
    ln --symbolic --no-target-directory /datas/app1/storage /opt/app1/storage
else
    echo "Symlink to \"/datas/app1/storage\": OK"
fi

# Pretend application as worked and produced data...
echo "baz" > /opt/app1/storage/datafile-baz

tree -CasFp /{conf,opt,datas}
EOT

cat >> demo.sh <<EOT
#!/bin/bash

docker build --quiet=true --tag="test-symlink" . > /dev/null

echo "Without data-container (1st time):"
docker run \
    "test-symlink" \
    /run.sh

echo "Without data-container (2nd time):"
docker run \
    "test-symlink" \
    /run.sh

echo ""
echo ""

docker kill "test-symlink-data" > /dev/null
docker rm "test-symlink-data" > /dev/null
docker create \
    --volume=/datas \
    --name "test-symlink-data" \
    debian:jessie \
    /bin/true > /dev/null

echo "With data-container (1st time):"
docker run \
    --volumes-from="test-symlink-data" \
    "test-symlink" \
    /run.sh

echo "With data-container (2nd time):"
docker run \
    --volumes-from="test-symlink-data" \
    "test-symlink" \
    /run.sh

echo "Explore data volume container directly"
docker run -it --volumes-from="test-symlink-data" debian:jessie ls -l /datas/app1/storage

EOT
chmod u+x demo.sh

# Explain how to test
echo "Files created, run demo.sh to experience my issue"

When running demo.sh I get

Without data-container (1st time):
Symlink to "/conf/main.cfg": CREATING...
Symlink to "/datas/app1/storage": CREATING...
/conf
`-- [-rw-r--r--           4]  main.cfg
/opt
|-- [drwxr-xr-x        4096]  app1/
|   |-- [drwxr-xr-x        4096]  conf/
|   |   `-- [lrwxrwxrwx          14]  main.cfg -> /conf/main.cfg
|   `-- [lrwxrwxrwx          19]  storage -> /datas/app1/storage/
`-- [drwxr-xr-x        4096]  app2/
/datas
`-- [drwxr-xr-x        4096]  app1/
    `-- [drwxr-xr-x        4096]  storage/
        |-- [-rw-r--r--           4]  datafile-bar
        `-- [-rw-r--r--           4]  datafile-baz

6 directories, 4 files
Without data-container (2nd time):
Symlink to "/conf/main.cfg": CREATING...
Symlink to "/datas/app1/storage": CREATING...
/conf
`-- [-rw-r--r--           4]  main.cfg
/opt
|-- [drwxr-xr-x        4096]  app1/
|   |-- [drwxr-xr-x        4096]  conf/
|   |   `-- [lrwxrwxrwx          14]  main.cfg -> /conf/main.cfg
|   `-- [lrwxrwxrwx          19]  storage -> /datas/app1/storage/
`-- [drwxr-xr-x        4096]  app2/
/datas
`-- [drwxr-xr-x        4096]  app1/
    `-- [drwxr-xr-x        4096]  storage/
        |-- [-rw-r--r--           4]  datafile-bar
        `-- [-rw-r--r--           4]  datafile-baz

6 directories, 4 files


With data-container (1st time):
Symlink to "/conf/main.cfg": CREATING...
Symlink to "/datas/app1/storage": CREATING...
/conf
`-- [-rw-r--r--           4]  main.cfg
/opt
|-- [drwxr-xr-x        4096]  app1/
|   |-- [drwxr-xr-x        4096]  conf/
|   |   `-- [lrwxrwxrwx          14]  main.cfg -> /conf/main.cfg
|   `-- [lrwxrwxrwx          19]  storage -> /datas/app1/storage/
`-- [drwxr-xr-x        4096]  app2/
/datas
`-- [drwxr-xr-x        4096]  app1/
    `-- [drwxr-xr-x        4096]  storage/
        |-- [-rw-r--r--           4]  datafile-bar
        `-- [-rw-r--r--           4]  datafile-baz

6 directories, 4 files
With data-container (2nd time):
Symlink to "/conf/main.cfg": CREATING...
Symlink to "/datas/app1/storage": OK
/conf
`-- [-rw-r--r--           4]  main.cfg
/opt
|-- [drwxr-xr-x        4096]  app1/
|   |-- [drwxr-xr-x        4096]  conf/
|   |   `-- [lrwxrwxrwx          14]  main.cfg -> /conf/main.cfg
|   `-- [drwxr-xr-x        4096]  storage/
|       |-- [-rw-r--r--           4]  datafile-bar
|       `-- [-rw-r--r--           4]  datafile-baz
`-- [drwxr-xr-x        4096]  app2/
/datas
`-- [drwxr-xr-x        4096]  app1/
    `-- [drwxr-xr-x        4096]  storage/
        |-- [-rw-r--r--           4]  datafile-bar
        `-- [-rw-r--r--           4]  datafile-baz

6 directories, 6 files

Output analysis:

• The two first trees (no data volume container used) seems about right :
  I have a symlink from /opt/app1/conf/main.cfg to /conf/main.cfg and another one from /opt/app1/storage/ to /datas/app1/storage/
• The third one (with data volume container) is also OK.
• But the fourth one (second run on the same data volume container) is missing the /opt/app1/storage/ symlink (the /opt/app1/conf/main.cfg is present).

I am clearly missing something here that makes Docker, at second run, overwrite the symlink and/or copy files from data volume container /datas/app1/storage to application container?

docker version:

Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

docker info:

Containers: 153
Images: 1514
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1820
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-57-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 4
Total Memory: 15.54 GiB
Name: mybox
ID: 2OQQ:3X7V:BE2M:WPL5:WLWX:DRFF:X3G6:4MIJ:MY3N:FCUF:NSYF:YLSO
WARNING: No swap limit support

uname -a:

Linux mybox 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[hcpss-banderson]

@carn1x, can you expand on your workaround? I am trying to solve the same problem, but I do not understand how you achieved this:

Alternatively would it be possible to image/run the original data container so that the --volume switch can be reapplied but the internally stored changes are maintained?

[carn1x]

@hcpss-banderson I did not achieve it, sorry to say. I ended up maintaining live/ as a mounted folder and then rsync changes on-top of it.

[trompx]

Hello,

I am trying to achieve the same.

@carn1x did you mean that you tried to mount the parent and both changing the symlink on the host or in the container failed?

So the only solution to this problem is to mount the live/ and on each code update, remove live folder content and update with the new one (with rsync or other commands)? With the inconvenient that the update is not atomic and introduces a downtime.