New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Base Images & Volume & mkdir #3639
Comments
I'm not sure I understand what the expected behavior should be, can you give an example dockerfile to reproduce? |
i've run into this too and here's an example: first Dockerfile:
this Dockerfile is then used to build image "base":
... looks good, works. second Dockerfile, should add more config and create a directory for later use in /etc:
again we build:
... looks good, but did not work:
as you see above, the edit: same problem with |
so I ran into this again with a different scenario just now and it's making me doubt if i am using
that causes all kinds of weird behaviour of the ldap server. eg. the domain reconfiguration is applied, but other default objects are not there in the new base dn. anyway, point is: as soon as i remove the volumes really sound great the way the docs and articles like this http://crosbymichael.com/advanced-docker-volumes.html describe them, but if i cannot modify their contents when "inheriting" them |
+1 on this issue, I think this is clearly a bug |
I've made simplest example to demonstrate this issue: Parent image - create file [e-max@e-max docker]$ cat ./parent/Dockerfile
FROM ubuntu
RUN mkdir /tmp/docker/
RUN echo "hello" > /tmp/docker/hello
VOLUME ["/tmp/docker/"] Child image - remove file [e-max@e-max docker]$ cat ./child/Dockerfile
FROM parent
RUN rm /tmp/docker/hello Build [e-max@e-max docker]$ docker build -t parent ./parent/
Sending build context to Docker daemon 2.56 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu
---> 5cf8fd909c6c
Step 1 : RUN mkdir /tmp/docker/
---> Using cache
---> 0925b684892f
Step 2 : RUN echo "hello" > /tmp/docker/hello
---> Using cache
---> c63b75c84ecc
Step 3 : VOLUME ["/tmp/docker/"]
---> Using cache
---> 71a00868d3a2
Successfully built 71a00868d3a2
[e-max@e-max docker]$ docker build -t child ./child/
Sending build context to Docker daemon 2.56 kB
Sending build context to Docker daemon
Step 0 : FROM parent
---> 71a00868d3a2
Step 1 : RUN rm /tmp/docker/hello
---> Using cache
---> dcde5cf04e6b
Successfully built dcde5cf04e6b And test - file still exist in child image. [e-max@e-max docker]$ docker run -i -t --rm child cat /tmp/docker/hello
hello
[e-max@e-max docker]$ |
@e-max Thanks. I could reproduce with master. |
Okay, I did little research and it seems that volume is non-mutable between
On each instruction we create new volume and copy content from original volume. |
During builds when `VOLUME` is declared it is not treated any differently than a normal volume. As such, for each new container created as part of the build there is a new volume created for that container. Because of the way volumes work, this essentially makes a delcared volume an immutable directory within the image since changes to the data in the volume are not commited back to disk. This in turn makes VOLUME delcarations highly positional and they must be declared at the end of the Dockerfile. The current behavior can also create a lot of added overhead if there is any signficant amount of data in the image at the declared volume location since this data gets automatically copied out onto the host as part of the volution initialization. This change introduces for the first time a differentiation from a build container and a normal container. This means a new Config field called `BuildOnly` has been added. This field is a boolean that can be used to change behavior of the container creation depending on if it is for a build or not. In the case of this change it checks if `container.Config.BuildOnly` is true, and if so, does not fully initialize the volume and instead just creates the dir within the container's fs (if it doesn't exist). Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 is this fixed? |
@jfrazelle No, see #7133 |
ah ok bummmeerrrrr |
I just ran into this issue. I was trying to extend from the official mongodb image from the hub and seed the database with my apps skeleton data, but because the parent container uses |
Relates #8647 |
@gotgenes It doesn't look like jhister is an official image. |
VOLUME makes it impossible to add additional data to that directory while building downstream docker image. There is a 3 year old docker bug about that (see below) and I really like the following advise: Basically, don't declare VOLUME unless you are done with that directory. moby/moby#3639 (comment) In particular, culprit for me is /opt/puppetlabs/server/data/puppetserver/ where I want to install additional gems. For all kinds and purposes, the only VOLUME here is ssldir (and maybe code to some extent). But since volumes can be defined at run time (and mostly everybody would do it anyway), I would just remove the VOLUME directive completely if I were you. The problem was introduced by PR puppetlabs-toy-chest#15 (author @ms1111)
VOLUME makes it impossible to add additional data to that directory while building downstream docker image. There is a 3 year old docker bug about that (see below) and I really like the following advise: Basically, don't declare VOLUME unless you are done with that directory. moby/moby#3639 (comment) In particular, culprit for me is /opt/puppetlabs/server/data/puppetserver/ where I want to install additional gems. For all kinds and purposes, the only VOLUME here is ssldir (and maybe code to some extent). But since volumes can be defined at run time (and mostly everybody would do it anyway), I would just remove the VOLUME directive completely if I were you. The problem was introduced by PR puppetlabs-toy-chest#15 (author @ms1111)
+1 for this feature |
Would it be feasible to error or at least warn in this situation? Silently failing is unfortunate. |
I've been relying on the behavior described in #21728 for several months; I had no idea it was a bug until helped a colleague figure out why the data he was putting in He might end up having to just copy mysql's Dockerfile into his own instead of basing his image on theirs. Maybe I'm supposed to do the same, in case #21728 ever gets "fixed". At minimum, child Dockerfiles should have a way to un-VOLUME a directory that was declared as a VOLUME by the parent. At least long enough to modify its contents. |
@pjweisberg see #3465 for that topic |
So this issue is like what... 5 years old... just lost 2h 45m trying to figure out what's wrong. Is it ever going to be fixed? |
Use DOCKER_BUILDKIT=1 |
Thank you very much for your answer/hint. This resolves all of my issues! As you can see below, the default behavior does not work, and the one you proposed does: That's a very simple Dockerfile that I'm using:
I wish this was standing out from the documentation more. I'm developing for quite a while using docker and only now I started using this: https://docs.docker.com/develop/develop-images/build_enhancements/ Perhaps it's better if this 'new' builder was the default? |
I thought we had a note in the documentation about this behaviour, but perhaps it got lost during a rewrite (can't find it in a quick search https://docs.docker.com/engine/reference/builder/)
That's definitely the goal |
If the base image Dockerfile has a VOUME /var/lib directive, then an image derived from the base image cannot do a RUN mkdir -p /var/lib/somedir
The mkdir command has no effect. I think it is convenient to have the base image export a VOLUME and the derived images can create their own structure without having to explicitly calling their own VOLUME.
As-is, the base image must not issue the VOLUME /var/lib or the derived images won't be able to create a directory in /var/lib
The text was updated successfully, but these errors were encountered: