The mtime for folders within a VOLUME is inconsistent with mtime of files within same VOLUME #17018

Open
tomfotherby opened this Issue Oct 14, 2015 · 9 comments

Comments

Projects
None yet
5 participants
@tomfotherby
Contributor

tomfotherby commented Oct 14, 2015

If you create a container image with some data exposed in a volume, when you run the container, the mtime of folders within the volume is set to when the container was run rather than when the container image was created. I'm not sure if it's a bug or a feature but I think it perhaps violates the Principle of least astonishment? For example, if I created a container 2 weeks ago, I would imagine the mtime of it's folders to be 2 weeks old, not brand new. From some limited testing, I think files work as expected - their mtime is not reset when the container is run. Ideally, the behaviour of the mtime of folders should be consistent with files.

I blogged about this issue here: http://l33t.peopleperhour.com/2015/10/19/docker-file-modified-time-gotcha/


docker version:
I reproduced on 1.8.3 and 1.7.1 (1.7.1 is the current coreos stable version).

docker info:

Containers: 275
Images: 415
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 967
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-16-generic
Operating System: Ubuntu 15.04
CPUs: 8
Total Memory: 5.809 GiB
Name: pochama
ID: 6VMD:Z57I:QDFF:TBSU:FNSH:Y433:6KDS:2AXU:CVMH:JUKU:TVQQ:7UMR
Username: tomfotherby
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

uname -a: Linux pochama 3.19.0-16-generic #16-Ubuntu SMP Thu Apr 30 16:09:58 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

I can reproduce on my laptop without any virtualisation and also on CoreOS stable channel running on AWS.

How reproducible: Easy

Steps to Reproduce:

# Build the test container:
touch afile
echo -e "FROM ubuntu\nADD afile /afolder/afile\nVOLUME /afolder" > Dockerfile
docker build -t mtimetest .
# Run it once and stat the folder:
docker run -it --rm mtimetest stat afolder
# Take a note of the Modify time.
sleep 2
# Run it again and stat the same folder:
docker run -it --rm mtimetest stat afolder
# Take a note of the Modify time - see it's not the same.

# Notice if you stat the file, the Modify time is aways the same (the time the container image was created)
docker run -it --rm mtimetest stat afolder/afile
sleep 1
docker run -it --rm mtimetest stat afolder/afile

# Also notice if you stat a folder that is not in the VOLUME, the Modify time is aways the same
docker run -it --rm mtimetest stat bin
sleep 1
docker run -it --rm mtimetest stat bin

# Note: if you remove the VOLUME statement from the Dockerfile, and re-run the test, there are no problems.

Actual Results:

$ docker run -it --rm mtimetest stat afolder
  File: 'afolder'
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: 801h/2049d  Inode: 4592900     Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-10-14 11:02:02.458513192 +0000
Modify: 2015-10-14 11:02:02.494513310 +0000
Change: 2015-10-14 11:02:02.494513310 +0000

$ docker run -it --rm mtimetest stat afolder
  File: 'afolder'
  Size: 4096        Blocks: 8          IO Block: 4096   directory
Device: 801h/2049d  Inode: 4592900     Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2015-10-14 11:02:04.126518656 +0000
Modify: 2015-10-14 11:02:04.162518774 +0000
Change: 2015-10-14 11:02:04.166518788 +0000

Expected Results:

I would expect the "Modify" time of folders to be the same every time I run the container.

Additional info:

Maybe a real use-case would be useful to explain how this issue is causing us a headache...

We run a website on a cluster of 3 CoreOS machines. When we have a new version of the website the deployment pipeline will build a data container of the PHP code and start a new web-server container on each of the machines in the cluster that uses --volumes-from thedatacontainer to get access to the new code. The assets of the website (js,css,images) get generated dynamically and stored in a folder which is a hash of their name and the filemtime of their folder. The problem is that if our 3 machines each start the container at slightly different times, the mtime of the asset folders will be different and they will all generate assets in different folders leading to broken urls in the website. One workaround is to manually reset the mtime to midnight after creating the data container so every server will end up with a consistent container, e.g.

/usr/bin/docker create --name yolo company/datacontainer:tag;
/usr/bin/docker run --rm --volumes-from yolo ubuntu bash -c 'touch -d $(date +%Y%m%d) /var/www/pathtoassets'
@tomfotherby

This comment has been minimized.

Show comment
Hide comment
@tomfotherby

tomfotherby Oct 14, 2015

Contributor

This could be a better example (it's a bit shorter):

echo -e "FROM ubuntu\nRUN mkdir /afolder\nRUN touch /afolder/afile"  | docker build -t mtimetest -

docker run -it --rm mtimetest stat afolder
sleep 2
docker run -it --rm mtimetest stat afolder
# Result: OK - Modify Time isn't changing

docker run -it --rm -v /afolder mtimetest stat afolder
sleep 2
docker run -it --rm -v /afolder mtimetest stat afolder
# Result: KO - Modify Time changes to the time the container was run

This shows using -v on docker run will change the mtime of folders.

Consistency is a cornerstone of docker - a container run on one server should be identical to another run somewhere else if they are from the same image. This issue breaks that promise as the mtime will not be consistent across different runs.

Contributor

tomfotherby commented Oct 14, 2015

This could be a better example (it's a bit shorter):

echo -e "FROM ubuntu\nRUN mkdir /afolder\nRUN touch /afolder/afile"  | docker build -t mtimetest -

docker run -it --rm mtimetest stat afolder
sleep 2
docker run -it --rm mtimetest stat afolder
# Result: OK - Modify Time isn't changing

docker run -it --rm -v /afolder mtimetest stat afolder
sleep 2
docker run -it --rm -v /afolder mtimetest stat afolder
# Result: KO - Modify Time changes to the time the container was run

This shows using -v on docker run will change the mtime of folders.

Consistency is a cornerstone of docker - a container run on one server should be identical to another run somewhere else if they are from the same image. This issue breaks that promise as the mtime will not be consistent across different runs.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Oct 14, 2015

Contributor

A volume is a dir that is created on the host and then mounted into the container.
I would not expect mtimes to be consistent here.

Contributor

cpuguy83 commented Oct 14, 2015

A volume is a dir that is created on the host and then mounted into the container.
I would not expect mtimes to be consistent here.

@pmoust

This comment has been minimized.

Show comment
Hide comment
@pmoust

pmoust Oct 14, 2015

Contributor

Ok, I was confused as well. docker inspect reveals the data dir of a volume.

@cpuguy83 In the scenario described by @tomfotherby, does setting mktime makes sense or is there another way that is more docker-ish?

Contributor

pmoust commented Oct 14, 2015

Ok, I was confused as well. docker inspect reveals the data dir of a volume.

@cpuguy83 In the scenario described by @tomfotherby, does setting mktime makes sense or is there another way that is more docker-ish?

@tomfotherby

This comment has been minimized.

Show comment
Hide comment
@tomfotherby

tomfotherby Oct 19, 2015

Contributor

I would not expect mtimes to be consistent here.

I get your point @cpuguy83 , that as soon as you use VOLUME, different rules apply. But you know docker more than the average person and I'd wager that most people would expect consistent mtimes. If I could share data between containers without using VOLUME then I would, but there isn't another way. It is the documented way.

Contributor

tomfotherby commented Oct 19, 2015

I would not expect mtimes to be consistent here.

I get your point @cpuguy83 , that as soon as you use VOLUME, different rules apply. But you know docker more than the average person and I'd wager that most people would expect consistent mtimes. If I could share data between containers without using VOLUME then I would, but there isn't another way. It is the documented way.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 20, 2015

Member

@cpuguy83 would it be possible to preserve the mtime when propagating a volume?

Member

thaJeztah commented Oct 20, 2015

@cpuguy83 would it be possible to preserve the mtime when propagating a volume?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Oct 23, 2015

Contributor

AFAICT it is preserving mtimes, actually.
@tomfotherby Can you provide the output of docker version?
Thans.

Contributor

cpuguy83 commented Oct 23, 2015

AFAICT it is preserving mtimes, actually.
@tomfotherby Can you provide the output of docker version?
Thans.

@tomfotherby

This comment has been minimized.

Show comment
Hide comment
@tomfotherby

tomfotherby Oct 26, 2015

Contributor

@cpuguy83 , here you go:

$docker --version
Docker version 1.8.3, build f4bf5c7

AFAICT it is preserving mtimes, actually.

Docker preserves mtimes of files, but not folders. Here is a single line test that shows the issue:

See that folders have the problem: echo -e "FROM ubuntu\nRUN mkdir /afolder\nRUN touch /afolder/afile" | docker build -t mtimetest - && docker run -v /afolder mtimetest stat afolder;sleep 2;docker run -v /afolder mtimetest stat afolder

After running, compare the 2 Modify times - they are different on each of the container runs.

See that files don't have the same problem: echo -e "FROM ubuntu\nRUN mkdir /afolder\nRUN touch /afolder/afile" | docker build -t mtimetest - && docker run -v /afolder mtimetest stat afolder/afile;sleep 2;docker run -v /afolder mtimetest stat afolder/afile

Contributor

tomfotherby commented Oct 26, 2015

@cpuguy83 , here you go:

$docker --version
Docker version 1.8.3, build f4bf5c7

AFAICT it is preserving mtimes, actually.

Docker preserves mtimes of files, but not folders. Here is a single line test that shows the issue:

See that folders have the problem: echo -e "FROM ubuntu\nRUN mkdir /afolder\nRUN touch /afolder/afile" | docker build -t mtimetest - && docker run -v /afolder mtimetest stat afolder;sleep 2;docker run -v /afolder mtimetest stat afolder

After running, compare the 2 Modify times - they are different on each of the container runs.

See that files don't have the same problem: echo -e "FROM ubuntu\nRUN mkdir /afolder\nRUN touch /afolder/afile" | docker build -t mtimetest - && docker run -v /afolder mtimetest stat afolder/afile;sleep 2;docker run -v /afolder mtimetest stat afolder/afile

@LK4D4 LK4D4 added the area/volumes label Sep 16, 2016

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Sep 16, 2016

Contributor

@cpuguy83 psst, still an issue?

Contributor

LK4D4 commented Sep 16, 2016

@cpuguy83 psst, still an issue?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Sep 17, 2016

Contributor

Still an issue.

Contributor

cpuguy83 commented Sep 17, 2016

Still an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment