flatten images - merge multiple layers into a single one #332

Closed
unclejack opened this Issue Apr 4, 2013 · 231 comments

Projects

None yet
@unclejack
Contributor

There's no way to flatten images right now. When performing a build in multiple steps, a few images can be generated and a larger number of layers is produced. When these are pushed to the registry, a lot of data and a large number of layers have to be downloaded.

There are some cases where one starts with a base image (or another image), changes some large files in one step, changes them again in the next and deletes them in the end. This means those files would be stored in 2 separate layers and deleted by whiteout files in the final image.

These intermediary layers aren't necessarily useful to others or to the final deployment system.

Image flattening should work like this:

  • the history of the build steps needs to be preserved
  • the flattening can be done up to a target image (for example, up to a base image)
  • the flattening should also be allowed to be done completely (as if exporting the image)
@justone justone referenced this issue May 1, 2013
Merged

Builder #472

@unclejack
Contributor

@shykes How would you like this to work? Could you provide an example of how this should work, please?

It looks like AUFS has some limit with 39-41 layers. We should really have image flattening in order to allow commit->run->commit to be used after deployment as well.

@bortels
bortels commented Aug 1, 2013

Ping.

My dockerfiles grow as I find neat stuff like this https://gist.github.com/jpetazzo/6127116

and IIRC, each RUN line makes a new level of AUFS, no?

I'm basically ignorant about many things, happy to admit it - if a "docker flatten" isn't coming down the pipe soon, does anyone have a reference for how to do it by hand? Or a reason it can't be?

(I guess I could workaround by moving all of the RUN lines into a single shell script, so it's not vital; but I can't do that with someone else's image. Hmm. Is there a way to "decompile" an image, recreating the Dockerfile used for it (assuming it was done entirely by a Dockerfile, of course).

@dqminh
Contributor
dqminh commented Aug 5, 2013

I encountered this recently too when building images. Will something like http://aufs.sourceforge.net/aufs2/shwh/README.txt help here ?

@vieux
Member
vieux commented Aug 5, 2013

I made a small tool to flatten images: https://gist.github.com/vieux/6156567

You have to use full ids, to flatten dhrp/sshd : sudo python flatten.py 2bbfe079a94259b229ae66962d9d06b97fcdce7a5449775ef738bb619ff8ce73

@mhennings
Contributor

+1

i see the need too.
if possible i would like a command that allows both, flatten all and squashing some selected layers.

if a container is flattened we should think about what happends when it is pushed. the registry / index could remove unneeded / duplicated layers, if enough inormation is sent during the push.
like "replaces Xxxxxxxxx, yyyyyyy, zzzzzzz"

@jpetazzo
Contributor

FWIW, the "aubrsync" (in aufs-tools package) might be useful for that,
since it aims at synchronizing and merging AUFS branches.

@shykes
Contributor
shykes commented Aug 21, 2013

From my answer in a different thread:

Currently the only way to "squash" the image is to create a container from it, export that container into a raw tarball, and re-import that as an image. Unfortunately that will cause all image metadata to be lost, including its history but also ports, env, default command, maintainer info etc. So it's really not great.

There are 2 things we can do to improve the situation:

  1. A short-term solution is to implement a "lossless export" function, which would allow exporting an image to a tarball with all its metadata preserved, so that it can be re-imported on the other side without loss. This would preserve everything except history, because an image config does not currently carry all of its history. We could try to plan this for 0.7 which is scheduled for mid-September. That is, if our 0.7 release manager @vieux decides we have time to fit it in the release :)

  2. A 2nd step would be to add support for history as well. This is a little more work because we need to start storing an image's full history in each image, instead of spreading it out across all the aufs layers. This is planned for 0.8.

@ykumar6
ykumar6 commented Aug 21, 2013

Hey guys, here's an idea we are prototyping. Let's say an image consists of 4 layers

L1<-L2<-L3<-L4

When we start a container off L4, we make changes in L5. Once the changes are complete, we commit back to get a new image

L1<-L2<-L3<-L4<-L5

At this point, we do a post-commit merge step where we start a new container, L4A from L3. We copy L5 & L4 into L4A and create a new image like this

L1<-L2<-L3-<-L4A

This way, we preserve the impermutable nature of the image but can compress layers when necessary to create new images

@dqminh
Contributor
dqminh commented Aug 22, 2013

@shykes @ykumar6 i did some experiments on exporting the image and trying to preserve metadata last night here https://github.com/dqminh/docker-flatten . Would love to know if the approach is reasonable.

What it does is that it will try to compress all image's layers into a tarfile, generate a dockerfile with as much metadata as possible, and create a new image from that.

@jpetazzo
Contributor

Question: do we really want to flatten existing images, or to reduce the number of layers created by a Dockerfile?

If we want to flatten existing images, it could be the job of an external tool, which would download layers, merge them, upload a new image.

If we want to reduce the number of layers, we could have some syntactic sugar in Dockerfiles, meaning "don't commit between those steps because I want to reduce the number of layers or the first steps are creating lots of intermediary files that I clean up later and don't want to includee in my layers".

@unclejack
Contributor

@jpetazzo Removing commits done between two steps of a Dockerfile would be useful, but we might still want to be able to flatten images. There are some use cases which require "-privileged" to be provided during a run and that's not possible with a Dockerfile, so you have to script a Dockerfile run, some docker run -privileged steps and then commit.
We might also want to craft custom images which have one layer and one single parent layer (a common image such as ubuntu, centos, etc).

@dkulchenko

@jpetazzo I would say both, as they address separate issues.

Flattening existing images allows you to work around the AUFS branch limit (you can only stack so many images), in the case where you're building on someone else's image, and someone else builds on yours, and your stack ends up hitting the limit pretty quick.

The syntactic sugar in the Dockerfile would allow building docker images that necessitate large toolchains to build and produce a comparatively small result (which I would argue is the more pressing of the two issues). Without it, a 2GB toolchain building a 10MB image will result in a 2058MB image.

@bortels
bortels commented Sep 11, 2013

I second the syntactic sugar - but I'd flip-flop it, in that I do a bunch of stuff (package building), and I really only want to commit the last step.

Maybe simply having an explicit "COMMIT imagename" in the dockerfile? And an implicit one right at the end? (I actually think commit at the end is sufficient - I'm not sure what use I'd have for an intermediate image, where I wouldn't just do it with a seperate dockerfile...)

I'll admit the AUFS limit was floating around in the back of my brain, but being able to flatten an arbitrary dockerfile is perfectly adequate for me there. (Doing so AND keeping history would be even nicer).

@jeffutter

I am somewhat fond of @bortels idea. I can see use cases where you would want the intermediate steps when building the dockerfile (incase something fails, like apt-get due to networking). You would want to be able to resume at that step. However it would be nice to say "When this is done" or "When you get to point A" squash the previous layers.

@tomgruner

An idea and script by Maciej Pasternacki:
http://3ofcoins.net/2013/09/22/flat-docker-images/

Docker looks really exciting, but the limit of 42 layers could cause some issues if a docker needs to be updated over a few years. Flattening every now and then doesn't sound so bad though.

@a7rk6s
a7rk6s commented Sep 26, 2013

When I started using Docker I soon wished for a "graft" command for image maintenance. Something like this:

$ docker graft d093370af24f 715eaaea0588
67deb2aef0e0

$ docker graft d093370af24f none
e4e168807d31

$ docker graft -t repo:8080/ubuntu12 d093370af24f 715eaaea0588
67deb2aef0e0

In other words it would basically change the parent of an image, or make it into a parent-less base image, and then return the new ID (possibly tagging/naming it). Would it be really slow because it'd have to bring both images into existence and compare them?

I like the "COMMIT" idea too. Or better, a "make a flattened image" flag when building, since this is really is more of a build option.

(Confession: I love Docker but the concept of the Dockerfile never clicked with me. Why add extra syntax just to run some shell commands? Why commit intermediate steps? So I've been making containers 100% with shell scripts. It's nice because it forces me to create build/setup scripts for my code, which is useful outside of Docker).

@jpetazzo
Contributor

Re "why commit intermediate steps": I find it very convenient when I have
longer Dockerfiles; when I modify one line, it only re-executes from that
line, thanks to the caching system; that saves me time+bandwidth+disk
space, since the first steps are usually those big "apt-get install" etc.;
of course, I could do the apt-get install and other big steps in a separate
Dockerfile, then commit that, then start another Dockerfile "FROM" the
previous image; but the Dockerfile caching system makes the whole thing way
easier. At least, to me :-)

On Wed, Sep 25, 2013 at 5:25 PM, a7rk6s notifications@github.com wrote:

When I started using Docker I soon wished for a "graft" command for image
maintenance. Something like this:

$ docker graft d093370af24f 715eaaea0588
67deb2aef0e0

$ docker graft d093370af24f none
e4e168807d31

$ docker graft -t repo:8080/ubuntu12 d093370af24f 715eaaea0588
67deb2aef0e0

In other words it would basically change the parent of an image, or make
it into a parent-less base image, and then return the new ID (possibly
tagging/naming it). Would it be really slow because it'd have to bring both
images into existence and compare them?

I like the "COMMIT" idea too. Or better, a "make a flattened image" flag
when building, since this is really is more of a build option.

(Confession: I love Docker but the concept of the Dockerfile never clicked
with me. Why add extra syntax just to run some shell commands? Why commit
intermediate steps? So I've been making containers 100% with shell scripts.
It's nice because it forces me to create build/setup scripts for my code,
which is useful outside of Docker).

β€”
Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/332#issuecomment-25135724
.

@jpetazzo https://twitter.com/jpetazzo
Latest blog post: http://blog.docker.io/2013/09/docker-joyent-openvpn-bliss/

@shykes
Contributor
shykes commented Sep 26, 2013

On Wed, Sep 25, 2013 at 5:25 PM, a7rk6s notifications@github.com wrote:

When I started using Docker I soon wished for a "graft" command for image
maintenance. Something like this:

$ docker graft d093370af24f 715eaaea0588
67deb2aef0e0

$ docker graft d093370af24f none
e4e168807d31

$ docker graft -t repo:8080/ubuntu12 d093370af24f 715eaaea0588
67deb2aef0e0

In other words it would basically change the parent of an image, or make
it into a parent-less base image, and then return the new ID (possibly
tagging/naming it). Would it be really slow because it'd have to bring both
images into existence and compare them?

I like the "COMMIT" idea too. Or better, a "make a flattened image" flag
when building, since this is really is more of a build option.

This problem will go ahead on its own once each image carries its full
history (currently history is encoded in the chain of aufs layers, which
avoids duplication of data, but means you can't get rid of one without
getting rid of the other, hence the problem we're discussing).

Once that's in place, whether you commit at each build step or only at the
end will be entirely up to you (the person running the build). Depending on
the granularity you want. More granularity = more opportunities to re-use
past build steps and save bandwidth and disk space on upgrades. Less
granularity = you can remove build dependencies from the final image,
export to a single tarball without losing context, etc. I doubt we'll add
any syntax to the Dockerfile to control that.

(Confession: I love Docker but the concept of the Dockerfile never clicked
with me. Why add extra syntax just to run some shell commands? Why commit
intermediate steps? So I've been making containers 100% with shell scripts.
It's nice because it forces me to create build/setup scripts for my code,
which is useful outside of Docker).

That's a common misunderstanding. Dockerfiles are not a replacement for
shell scripts. They provide context for running shell scripts (or any
other kind of script) from a know starting point (hence the FROM keyword)
and a known source code repository (hence the ADD keyword).

@shykes
Contributor
shykes commented Sep 26, 2013

s/the problem will go ahead/the problem will go away/

On Wed, Sep 25, 2013 at 5:36 PM, Solomon Hykes
solomon.hykes@dotcloud.comwrote:

On Wed, Sep 25, 2013 at 5:25 PM, a7rk6s notifications@github.com wrote:

When I started using Docker I soon wished for a "graft" command for image
maintenance. Something like this:

$ docker graft d093370af24f 715eaaea0588
67deb2aef0e0

$ docker graft d093370af24f none
e4e168807d31

$ docker graft -t repo:8080/ubuntu12 d093370af24f 715eaaea0588
67deb2aef0e0

In other words it would basically change the parent of an image, or make
it into a parent-less base image, and then return the new ID (possibly
tagging/naming it). Would it be really slow because it'd have to bring both
images into existence and compare them?

I like the "COMMIT" idea too. Or better, a "make a flattened image" flag
when building, since this is really is more of a build option.

This problem will go ahead on its own once each image carries its full
history (currently history is encoded in the chain of aufs layers, which
avoids duplication of data, but means you can't get rid of one without
getting rid of the other, hence the problem we're discussing).

Once that's in place, whether you commit at each build step or only at the
end will be entirely up to you (the person running the build). Depending on
the granularity you want. More granularity = more opportunities to re-use
past build steps and save bandwidth and disk space on upgrades. Less
granularity = you can remove build dependencies from the final image,
export to a single tarball without losing context, etc. I doubt we'll add
any syntax to the Dockerfile to control that.

(Confession: I love Docker but the concept of the Dockerfile never
clicked with me. Why add extra syntax just to run some shell commands? Why
commit intermediate steps? So I've been making containers 100% with shell
scripts. It's nice because it forces me to create build/setup scripts for
my code, which is useful outside of Docker).

That's a common misunderstanding. Dockerfiles are not a replacement for
shell scripts. They provide context for running shell scripts (or any
other kind of script) from a know starting point (hence the FROM keyword)
and a known source code repository (hence the ADD keyword).

@a7rk6s
a7rk6s commented Sep 26, 2013

They provide context

Makes sense. Though, the Dockerfiles I've seen in the wild have been all over the place (as are the ones I've created, since I'm still trying to find the best way to lay things out so it's easy to develop / maintain / repurpose chunks to make different images).

once each image carries its full history

Out of curiosity, will it be possible to do, e.g., "apt-get clean" after the image has been built, and end up with less disk space used?

@mattwallington

I am assuming this didn't make it into .7 as previously mentioned. Any plans for the next release?

@vmadman
vmadman commented Dec 28, 2013

Am I understanding this correctly? An image can only have a maximum of ~40 RUN/ADD statements in its entire lifetime.. including inheritance?

@shykes
Contributor
shykes commented Dec 28, 2013

The limit is now 127 layers. That's the hardcoded maximum layers in an aufs mount.

We are working an lifting this by separating commit history from logical history.

On Sat, Dec 28, 2013 at 4:23 AM, Luke Chavers notifications@github.com
wrote:

Am I understanding this correctly? An image can only have a maximum of ~40 RUN/ADD statements in its entire lifetime.. including inheritance?

Reply to this email directly or view it on GitHub:
#332 (comment)

@vmadman
vmadman commented Dec 29, 2013

Ah... great. Any limit in that regard would be reaallly bad, I think, as it completely eliminates the ability to put anything that updates inside of Docker. i.e. A website deployment..

@shykes
Contributor
shykes commented Dec 29, 2013

You can easily update your website any number of times without increasing the number of layers. Just re-run "docker build" for each version of the website. Build caching will make sure it's fast and doesn't waste disk space.

On Sat, Dec 28, 2013 at 9:06 PM, Luke Chavers notifications@github.com
wrote:

Ah... great. Any limit in that regard would be reaallly bad I think as it completely eliminates the ability to put anything that updates inside of Docker without losing the benefits of Docker. i.e. A website deployment..

Reply to this email directly or view it on GitHub:
#332 (comment)

@monokrome

See #3116 for another potential user interface suggestion for solving this same problem.

@garo
garo commented Jan 21, 2014

I'm fine with Dockerfile build committing each RUN/ADD command into another layer, it makes really fast to develop the Dockerfile. But after the build command completes without error I'd really much like that it would flatten all steps so that the end result would be one image which is added on top of the FROM image, instead of having to ship and push all images in between.

@daviddyball

@garo, I think it should be optional as to how much you quash the image. That way people could choose to inherit from base-images or not. Choosing which image ID you'd like to compress down to would be of benefit

e.g. Given the following tree:

└─ad18ff9f83df Virtual Size: 484.7 MB Tags: myimage:latest
    └─f45f88e50248 Virtual Size: 552.7 MB
        └─3e5747d65960 Virtual Size: 552.8 MB
            └─8c381ae7a086 Virtual Size: 563.6 MB
                └─13d909f018b8 Virtual Size: 563.6 MB Tags: ubuntu:12.04```

```docker squash [IMAGE] [FROM] [TO]```

```docker squash myimage:latest 13d909f018b8 ad18ff9f83df```

The above example would result in there only being two images at the end, the base image `13d909f018b8` and `ad18ff9f83df` (myimage:latest).

Just an idea. I've not even looked into the way AUFS works, so this is purely an idea from an end-user perspective.

EDIT: Fixing tree formatting
@monokrome

@garo @davidrobertwhite Given the syntax that I recommended in #3116, you can easily perform this explicitly by asking for only one COMMIT at the end of the Dockerfile. Without any COMMIT rules, it will use the current solution (one layer per RUN), and if you provide multiple COMMIT rules then you will be explicitly saying "I want these pieces as separate layers".

The benefits of this approach is that it's more forward compatible in some ways. For instance, it's possible that Docker might allow for (assuming it doesn't already) parallelized downloading of layers from the index. If you make everything into one giant layer, then you've effectively reduced the usefulness of such a feature. It's better for those kinds of things to be explicitly requested rather than implicitly. Even in the case without parallel downloads from the index, it's nice to have a few smaller layers than one giant one. That way, if a download fails, you don't have to re-download everything again.

Furthermore, providing this also allows for people to say "This layer updates the system", "This layer is where I installed Java", "This is where I installed the services for this machine" by putting a separate COMMIT rule at each point in the Dockerfile process. The suggestion that makes development more difficult is a bit NIL, because there could be a flag to --commit-all or something similar. This could effectively allow someone to ignore the COMMIT rules. Manually ignoring vs manually requesting in this case is better, because the Dockerfile should represent what it is doing by default unless explicitly requested not to.

TLDR: It's important to have some way of specifically asking for AUFS to commit at specific points, because the number of changes that one RUN command can make is very arbitrary, and squashing everything can cause problems as easily as not allowing a user to squash anything.

@mattwallington

What about the case where you aren't using a dockerfile but instead have multiple commits on an image that update the same files. So you have multiple layers of edits of the same file and you need to compact them into a single layer and don't want to keep a history of all of the prior changes/layers?

On Jan 23, 2014, at 4:24 PM, "Brandon R. Stoner" notifications@github.com wrote:

@garo @davidrobertwhite Given the syntax that I recommended in #3116, you can easily perform this explicitly by asking for only one COMMIT at the end of the Dockerfile. Without any COMMIT rules, it will use the current solution (one layer per RUN), and if you provide multiple COMMIT rules then you will be explicitly saying "I want these pieces as separate layers".

The benefits of this approach is that it's more forward compatible in some ways. For instance, it's possible that Docker might allow for (assuming it doesn't already) parallelized downloading of layers from the index. If you make everything into one giant layer, then you've effectively reduced the usefulness of such a feature. It's better for those kinds of things to be explicitly requested rather than implicitly. Even in the case without parallel downloads from the index, it's nice to have a few smaller layers than one giant one. That way, if a download fails, you don't have to re-download everything again.

Furthermore, providing this also allows for people to say "This layer updates the system", "This layer is where I installed Java", "This is where I installed the services for this machine" by putting a separate COMMIT rule at each point in the Dockerfile process. The suggestion that makes development more difficult is a bit NIL, because there could be a flag to --commit-all or something similar. This could effectively allow someone to ignore the COMMIT rules. Manually ignoring vs manually requesting in this case is better, because the Dockerfile should represent what it is doing by default unless explicitly requested not to.

TLDR: It's important to have some way of specifically asking for AUFS to commit at specific points, because the number of changes that one RUN command can make is very arbitrary, and squashing everything can cause problems as easily as not allowing a user to squash anything.

β€”
Reply to this email directly or view it on GitHub.

@monokrome

@mattwallington For that case, you could have a command-line flag to list specific commits to merge down or something similar. Some possible usages could be:

--commits=layer1,layer5,layer7
--commits=all

Once again, emphasizing explicit sqashing of all instead of using it as a (potentially harmful) default. When not provided, we can assume to keep all commits.

@mattwallington

Definitely. It shouldn't be default. It should be an option. But the idea is, if you have to "edit" an image by creating a container, modifying a file, committing back to an image, and doing that cycle a few times, after awhile you end up with a ton of layers.

Especially if you are editing the same file over and over again, you have layer after layer of the same file overwriting itself. It would be nice if after that whole process you could merge the images all down into a single layer losing all of the history of changes but saving tons of space and time to push/pull that image to other devices.

@kryztoval

I would really like this to be possible, and to some extent to keep the IDs so we can reuse the
images.

for the example granted

└─ad18ff9f83df Virtual Size: 484.7 MB Tags: myimage:latest
    └─f45f88e50248 Virtual Size: 552.7 MB
        └─3e5747d65960 Virtual Size: 552.8 MB
            └─8c381ae7a086 Virtual Size: 563.6 MB
                └─13d909f018b8 Virtual Size: 563.6 MB Tags: ubuntu:12.04

if you flatten one layer that it may end as:

└─ad18ff9f83df Virtual Size: 484.7 MB Tags: myimage:latest
    (└─f45f88e50248 Virtual Size: 0 B) Flattened
        └─3e5747d65960 Virtual Size: 552.8 MB
            └─8c381ae7a086 Virtual Size: 563.6 MB
                └─13d909f018b8 Virtual Size: 563.6 MB Tags: ubuntu:12.04 

So you can build a container from any but the flattened images, and also it will be interchangeable in case someone has the full tree or the flattened tree.

Too much to ask? I am sure it is quite hard to do.

@vmadman
vmadman commented Jan 30, 2014

I'm still too underqualified to weigh in on this, but feel compelled to do so anyway. What you guys are suggesting is really awesome, but wouldn't the need for such granular control of the layers only exist in edge cases?

I just mean, I'd be happy with any mechanism for flattening layers, firstly, and then let the more advanced stuff come down the road over the next few years. For my/our part, we're just wanting 1.0 to go gold, and this layer thing is a potential show stopper.

I have very little experience with open source projects, so far all I know one of us nerds will get bored one day and build what you're asking for overnight.. but with color.. but in commercial software development everything has to come in thin layers and we systematically reject any complex requests solely on the fact that it's complex, regardless of whether or not its a good idea, because complexity inevitably brings instability, confusion, and eventually panic.

With that said, it is a wonderful idea that I hope they consider.

Thanks,

@monokrome

@vmadman This particular issue represents a very bothersome issue in Docker.

The Docker team can decide if/when it should be implemented. I would think that whether or not it happens before version 1.0 is a completely separate discussion - assuming this does happen at all.

Either way, the feature will need to be collaboratively discussed (hopefully?) and that is what is happening in here right now.

@shykes
Contributor
shykes commented Jan 31, 2014

Hi guys, flattening images is a desirable and we (docker maintainers) will
definitely implement it. It's only a matter of finding the time to do it :)

If you want to volunteer to help out ping me here or on #docker-dev.
Otherwise it will happen soon-ish (and in the meantime you can craft tiny
images from the outside, although you lose the benefit of 'docker build' so
I know it's not ideal... soon!)

On Thu, Jan 30, 2014 at 2:40 PM, Brandon R. Stoner <notifications@github.com

wrote:

@vmadman https://github.com/vmadman This particular issue represents a
very bothersome part of Docker. The Docker team can decide if/when it
should be implemented. I would think that whether or not it happens before
version 1.0 is a completely separate discussion - assuming this does happen
at all.

Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/332#issuecomment-33742579
.

@vmadman
vmadman commented Jan 31, 2014

@monokrome No I understand, and like I said, its a good idea. I'm only speculating that it might be more useful to scale down the complexity of the initial solution talks, just for the sake getting a solution to central problem of layer limits more quickly. Unless I'm misunderstanding, and I probably am, the largest problem is the fact that there is a finite layer limit, which excludes the possibility of an "updating image" (one that continuously updates) and only leaves room for rebuilds, which Shykes mentioned above.

But, feel free to ignore me, I'm not trying to criticize anyone... and apologize if it came across that way.

@shykes I'd love to volunteer, but know little to nothing about Go. Hopefully I'll be able to stick with the Docker community and some of it will rub off on me over time. Either way, I'm happy to help in any way that I can, and as time permits.

@kryztoval

@shykes I'm volunteering.

@svenfuchs

What about having RUN and RUNC, C meaning "commit after this step"? I reckon that would be an API change, but I can't think of a one-letter description of "don't commit after this step".

@justincampbell

If we could just use a heredoc or similar form in a run command, I feel like it would mitigate the problem:

RUN <<-SCRIPT
    yum install gcc-c++ make -y
    ./configure && make && make install && make clean
    yum erase gcc-c++ make -y
SCRIPT

Or quoted?

RUN 'yum install gcc-c++ make -y
     ./configure && make && make install && make clean
     yum erase gcc-c++ make -y'

The use case I keep running into is wanting to install dependencies for a single thing and remove them after, and not be stuck with the dependencies in commit(s).

I suppose the best practice is to ADD a script to do this, but now I have to later remove that temporary script anyway.

@scjudd
scjudd commented Feb 6, 2014

@justincampbell: You can do the following:

RUN \
    yum install gcc-c++ maky -y ;\
    ./configure && make && make install && make clean ;\
    yum erase gcc-c++ make -y ;\
# END RUN

I use this syntax quite a bit. I think @jpetazzo posted it originally.

As an added bonus, the winky meh face at the end of each line pretty well sums up the feeling one gets from this workaround. "that's clever! ;)" + "it's also sort of terrible that there's not a better way to do this. :\" = ;\

@justincampbell

@scjudd @jpetazzo I tend to use command && \ instead of command; \. While using the semicolon, you could have a failure in the middle of your script, but the rest of the script would still attempt to execute.

@monokrome

@svenfuchs RUNC is less explicit, and adds a lot of confusion since it potentially implies that it's some type of RUN command.

@tsutsu
tsutsu commented Apr 28, 2014

It seems to me that all the use-cases discussed thusfar could be subsumed into the API of docker save and docker load, like so:

docker save [--flatten=from..to] [--clean] [--remove=base-image] image > patch-layer.tar

cat patch-layer.tar > docker load [--reparent-onto=new-parent-image]

In docker save:

  • --flatten would be a parameter that could be specified zero or more times, and would take a pair of image IDs, and collapse the layers in those ranges into single (temporary) layers, creating a new (temporary) image-chain.
  • --clean would be an optional parameter, taking effect after --flatten (and thus operating on its temporary chain), that would iterate through the layers that would be saved, and, for each layer, remove any files that currently exist with the same content-hash further up the chain, and haven't been whited out since. This would create another temporary image-chain.
  • --remove would be an optional parameter that takes an image ID, taking effect after --clean, and would remove any layers that exist in base-image from the resulting tarball, making the direct child of base-image into the new root layer of the saved image-chain.

Using all three options like so:

docker save --flatten=parent-image..child-image --clean --remove=parent-image child-image > patch-layer.tar

You'd get a tarball containing one layer, with child-image as the root layer (not parented on parent-image), where all redundant data from parent-image has been stripped. This would be the minimal patch required to recreate child-image on top of parent-image.

Meanwhile, in docker load:

  • --reparent-onto would be an optional parameter, specifying a parent to "replay" the provided layers onto, with the root layer in the tarball becoming the direct child of new-base-image. Since patches created by docker save --remove have the patch as their root layer, you must specify what to attach them to when you re-load them. Interestingly, this does not have to be the same image that was specified in --remove -- you could save a patch given one parent, and then attach it to a different parent on re-load.

This would allow for:

  • merging everything into a single flattened image
  • merging everything in a Dockerfile into an "app" layer that keeps its parentage on the FROM layer it specifies
  • unflattening a flattened image, by first loading it, and then re-saving it with --remove set to the image it was originally constructed under (alternately, --remove could also be an option on docker load, with the same diffing semantics it has on docker save, to allow this to be one step)
  • in an image tree base-image < toolchain-image < build-results-image, generating a patch from build-results-image, and then loading it using --reparent-onto=base-image to remove the toolchain from the resulting image
  • submitting a patch layer to a mailing list to discuss using it, in the same way code patches are submitted now
  • everything docker export and docker import do now. (Save for the fact that you'd get the json metadata along with the flattened layer, which would actually be nice. You could still extract just the layer image by piping the export through an extra invocation of tar -x, if you liked.)
@tailhook

+1 for apply-patch or similar command. Unfortunately, my similar proposal #5174 was rejected.

@skolos
skolos commented Apr 29, 2014

I really like proposal by @tsutsu

@borromeotlhs

I saw @TravisCardwell post an interesting solution here:
https://github.com/extellisys/docker-image-apt-cacher-ng

If we're throwing out ideas, I believe this gentleman has an interesting (and working) workflow : )

@akoeplinger akoeplinger added a commit to akoeplinger/docker-mono-aspnetvnext that referenced this issue Jun 4, 2014
@akoeplinger akoeplinger compile/install Mono in a single RUN step to avoid creating many inte…
…rmediate container commits (see docker/docker#332)
8927171
@vmadman
vmadman commented Jun 4, 2014

Yes, I think @tsutsu has a good proposal. I did not follow all of it and do not have time to really dive in and read it, but it sounded nice on the surface. It sort of reminds me of deleting snapshots in VirtualBox.. kinda.

I wonder how hard that would be to code?

@mpasternacki

Thanks for the link, @borromeotlhs - I have wrapped @TravisCardwell approach into a Ruby script that can either "rebase" an existing image, or run docker build and rebase the resulting image as a single layer on top of the Dockerfile's FROM image. There's a bit of a hack to support aufs deletions, and the script needs to run under fakeroot to support permissions and device files. It could probably be improved by working on tar archive contents rather than by extracting them and working with the filesystem (which would also eliminate need for fakeroot).

You can get the script at https://github.com/3ofcoins/docker-images/blob/master/script/docker-rebase.rb

@borromeotlhs

Do you mean work on a tar archive's content without deflating it?

Otherwise, that sounds amazing.

Is Ruby the only script docker excepts? Or can any scripting language
which runs on a Linux image acceptable?

R/
TJ
On Jun 5, 2014 1:56 AM, "Maciej Pasternacki" notifications@github.com
wrote:

Thanks for the link, @borromeotlhs https://github.com/borromeotlhs - I
have wrapped @TravisCardwell https://github.com/TravisCardwell approach
into a Ruby script that can either "rebase" an existing image, or run docker
build and rebase the resulting image as a single layer on top of the
Dockerfile's FROM image. There's a bit of a hack to support aufs deletions,
and the script needs to run under fakeroot
http://fakeroot.alioth.debian.org/ to support permissions and device
files. It could probably be improved by working on tar archive contents
rather than by extracting them and working with the filesystem (which would
also eliminate need for fakeroot).

You can get the script at
https://github.com/3ofcoins/docker-images/blob/master/script/docker-rebase.rb

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@fangpenlin

+1 on this feature. I just found myself in need to clean sensitive data after building the image. I ADD private keys into that container and pull some repo from Github, then I use RUN rm ... to remove those private keys. However, I realized that it doesn't really works since AFUS keeps all the file history, my private key is still there in the image. So, a squash or flatten command could be really helpful in my situation.

@borromeotlhs

You would need more than that, actually, you'd need an equivalent of a git
history rewrite :(
On Jun 9, 2014 9:38 AM, "Victor Lin" notifications@github.com wrote:

+1 on this feature. I just found myself in need to clean sensitive data
after building the image. I ADD private keys into that container and pull
some repo from Github, then I use RUN rm ... to remove those private
keys. However, I realized that it doesn't really works since AFUS keeps all
the file history, my private key is still there in the image. So, a squash
or flatten command could be really helpful in my situation.

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@fangpenlin

@borromeotlhs why I would need that? My idea is something like

  • Layer 1 Initial
  • Layer 2 Add private key
  • Layer 3 Git pull
  • Layer 4 Remove private key

As long as I can squash layers from 2 to 4, so that result would be

  • Layer 1 Initial
  • Layer 2 squashed from 2 to 4

then the private key shouldn't be in the AUFS history, or did I miss something?

@borromeotlhs

Because flattening, in this instance, enables you to excise the
intermediate steps while only retaining those steps that you care about.
If you have:
Step A - install Ubuntu
Step b - set up packages to include ssh
Step c - commit private key
Step d - some other important command

And you then wanted to get to:
Step A - install Ubuntu
Step c' - install and configure all packages to include ssh
Step d - other stuff

You could do that, but grafting the filesystem from step d onto the end
result of old step c will not erase the commit of the private key ( at
least, I see no guarantee of it)

Squashing or rebasing will make less commits, but won't undo the actions
each individual commit achieved. Rebasing or squashing or flattening won't
get your private keys out of there, I don't think. . . at least not as the
example shows.
On Jun 9, 2014 9:56 AM, "Victor Lin" notifications@github.com wrote:

@borromeotlhs https://github.com/borromeotlhs why I would need that? My
idea is something like

  • Layer 1 Initial
  • Layer 2 Add private key
  • Layer 3 Git pull
  • Layer 4 Remove private key

As long as I can squash layers from 2 to 4, so that result would be

  • Layer 1 Initial
  • Layer 2 squashed from 2 to 4

then the private key shouldn't be in the AUFS history, or did I miss
something?

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@borromeotlhs

I should mention that, on second thought, you could, similarly to not
'pick'-ing a commit for rebase in git, choose to not 'pick' commit c.
Again though, flattening isn't exactly that. Flattening is just a way to
regroup what is the important commit group ( I think of it like a git
commit --amend) without actually altering what occurred.

Cheers,
TJ
On Jun 9, 2014 10:02 AM, "Theodore Borromeo" borromeotlhs@gmail.com wrote:

Because flattening, in this instance, enables you to excise the
intermediate steps while only retaining those steps that you care about.
If you have:
Step A - install Ubuntu
Step b - set up packages to include ssh
Step c - commit private key
Step d - some other important command

And you then wanted to get to:
Step A - install Ubuntu
Step c' - install and configure all packages to include ssh
Step d - other stuff

You could do that, but grafting the filesystem from step d onto the end
result of old step c will not erase the commit of the private key ( at
least, I see no guarantee of it)

Squashing or rebasing will make less commits, but won't undo the actions
each individual commit achieved. Rebasing or squashing or flattening won't
get your private keys out of there, I don't think. . . at least not as the
example shows.
On Jun 9, 2014 9:56 AM, "Victor Lin" notifications@github.com wrote:

@borromeotlhs https://github.com/borromeotlhs why I would need that?
My idea is something like

  • Layer 1 Initial
  • Layer 2 Add private key
  • Layer 3 Git pull
  • Layer 4 Remove private key

As long as I can squash layers from 2 to 4, so that result would be

  • Layer 1 Initial
  • Layer 2 squashed from 2 to 4

then the private key shouldn't be in the AUFS history, or did I miss
something?

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@shin- shin- added the Distribution label Jul 1, 2014
@rvs
rvs commented Jul 2, 2014

Ran into the very same problem: accidentally created multiple layers and seem to be stuck now.

What I am wondering though, is whether there's any chance for the following workaround:

  1. export the points in the image history I am interested in as tarballs
  2. iterate over 'docker build' with the FROM being a sort of iteration variable pointing at the current image
  3. the only RUN command in that build would be something that would completely overwrite the image of the container FS

The trouble is -- I don't quite know how to make #3 happen reliably. Thoughts?

@borromeotlhs

Well,
The biggest issue is that images (opaque) and dockerfiles ( transparent)
are not equivalent. So though you could perhaps squash your filesystem
manually, there wouldn't be a way to have the multiple run commands you
used to get to the end result also be 'squashed' without just chaining them
on your dockerfile. So your image could be compressed to just those steps
you wanted, but you wouldn't be able to reproduce that action in docker
files, only transmit the image via a pull.

Now, from what you are talking about, I do believe that the post I linked
to previously does have some pretty good details for just such an
operation. I don't know, as I don't claim to take credit for that idea :)

-TJ
On Jul 1, 2014 5:08 PM, "Roman V Shaposhnik" notifications@github.com
wrote:

Ran into the very same problem: accidentally created multiple layers and
seem to be stuck now.

What I am wondering though, is whether there's any chance for the
following workaround:

  1. export the points in the image history I am interested in as
    tarballs
  2. iterate over 'docker build' with the FROM being a sort of iteration
    variable pointing at the current image
  3. the only RUN command in that build would be something that would
    completely overwrite the image of the container FS

The trouble is -- I don't quite know how to make #3
#3 happen reliably. Thoughts?

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@mitar
mitar commented Jul 2, 2014

I see separate layers for each run command great when developing a Dockerfile, because of the caching. But after I get to the right file, I often combine all RUN commands into one big command so that I can then base other images upon that file.

I think published images should behave the same. Locally they can have multiple levels, but when pushing you should have an option to combine them together.

@txomon
txomon commented Jul 28, 2014

I had an idea: How about specifying the following

FROM xxx
COMMIT OFF
ADD apt-sources /etc...
RUN apt-get install ...
RUN ap
COMMIT
COMMIT ON

This way, it would be backwards compatible, and would ease install deps, compile, install and clean all in one, having just one layer!

@jerng
jerng commented Jul 29, 2014

Docker clients vs Docker registries now has the same pattern that Git solved, by making every Git client a complete repository. Perhaps that could be done in a future release of Docker?

Meanwhile, I like this idea: #332 (comment)

I'd say there's merit to the having these features:

Given a tree of images:
L1 <- L2 <- L3 <- L4 <- L5

It would be great to have [Git rebase/squash-stye] operations to output the following:

Operation A (use case 2 adjacent nodes).
L1 <- (L2,L3: filesystems merged, metadata from L3) <- L4 <- L5

Operation A (use case N adjacent nodes):
L1 <- (L2,L3,L4: filesystems merged, metadata from L4) <- L5

Operation B (updating config, like the environment).
L1 <- L2 <- L3 <- L4 <- (L5, filesystem preserved, metadata updated)

Operation C (diff reports/GUI):
=> a screen/report showing the difference between any two adjacent nodes
=> a "step through" screen/report showing the difference between N adjacent nodes

All that being said, this note on avoidance of core feature bloat on related features, is noted. https://groups.google.com/forum/#!topic/docker-user/Y0z-BnTlogo

@l3iggs
l3iggs commented Aug 16, 2014

Not having a clean way to flatten images is currently a huge issue for me. At the moment many of my intermediate RUN steps create very large layer sizes, files which are removed in subsequent RUN steps.
The consumer of the images cares not about the history of how they were created, but in order to use the image it must pull the entire creation history. This is supremely wasteful of my time, my bandwidth and my disk space as well as the bandwidth and disk space of the docker image registry.

@txomon's idea for adding a COMMIT command seems like a perfect solution to the problem especially since it is backwards compatible, +1.

@monokrome

The COMMIT ON and COMMIT OFF style seems a bit superflous when you just need COMMIT to denote a commit.

See #3116 for an example.

@txomon
txomon commented Aug 16, 2014

The idea under COMMIT ON and COMMIT OFF was to denote that you didn't
wanted to have automatic commits. If you wish, you can say just:

COMMIT OFF


COMMIT
...
...
COMMIT

i.e.

The idea was not to disturb current functionality, and the need to be
explicit on having autocommit disabled, or sth like that

On Sat, Aug 16, 2014 at 9:28 PM, Brandon R. Stoner <notifications@github.com

wrote:

The COMMIT ON and COMMIT OFF style seems a bit superflous when you just
need COMMIT to denote a COMMIT. See #3116
#3116 for an example.

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

Javier Domingo Cansino

@mhennings
Contributor

I think for your usecase you could export as a tar and import again. This
flattens history and makes the image smaller. But you lose metadata as well.

Still I think there is a need for a kind of rebase command. Like I have
image x and want it to be based on y.
Calculation of the delta could allow avoiding full downloads on updates.
Am 16.08.2014 23:53 schrieb "Javier Domingo" notifications@github.com:

The idea under COMMIT ON and COMMIT OFF was to denote that you didn't
wanted to have automatic commits. If you wish, you can say just:

COMMIT OFF


COMMIT
...
...
COMMIT

i.e.

The idea was not to disturb current functionality, and the need to be
explicit on having autocommit disabled, or sth like that

On Sat, Aug 16, 2014 at 9:28 PM, Brandon R. Stoner <
notifications@github.com

wrote:

The COMMIT ON and COMMIT OFF style seems a bit superflous when you just
need COMMIT to denote a COMMIT. See #3116
#3116 for an example.

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

Javier Domingo Cansino

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@jdmarshall

I have a suggestion I was going to submit as its own issue, but here works.

Similar to the 'LAYER X' command and a few others I saw elsewhere, but hopefully a little more straightforward to read.

"Step" starts a new AUFS layer. Everything until the next step stays in the same layer. No need to indent. No need for brackets. It's just a section heading. Also I chose "Step" because I think it will make a lot more sense to beginners than "Layer" (what's a layer?), which is kind of out of keeping with the dead-obvious instruction names that make up the bulk of your typical Docker file.

Note: I don't feel very strongly about the colons, but I thought they were a nice touch.

STEP 1:
RUN apt-get update
RUN apt-get install somepackage xvfb git
RUN apt-get clean

STEP 2: Put some stuff on the filesystem
RUN curl http://www.example.com/index.html
ADD my_files /tmp

STEP 3:
RUN chown -R user:group /tmp/my_files

@phemmer
Contributor
phemmer commented Oct 6, 2014

@jdmarshall How would you get it back into the mode where every line is a layer? With your syntax, it seems that once you start grouping, you have to continue doing so for the rest of the file.

Personally I'm in favor of the syntax txomon recommended, or something similar to it.
Many people are familiar with the concept of database transactions & automatic/explicit commits. It makes sense to use a paradigm people are familiar with rather than creating a new one.

@docbill
docbill commented Oct 11, 2014

docker-rebase is my attempt to resolve this issue:

https://github.com/docbill/docker-scripts

I find this essential for containers running databases, where you want to regularly snapshot the database, but you don't want an ever growing image history.

I'm not too thrilled about the name... As rebase doesn't also imply commit. However, it seemed better than flatten, as that would seem to imply I was working on an image, not a container.

I'll probably extend the script to have the option of taking attributes from the running container instead of the image. e.g. Sometimes, I really want to keep the original command, and sometimes I want to update an image to have a new default cmd. So that is probably something that would be useful to be able to select when running the script.

@vladia vladia added a commit to vladia/docker that referenced this issue Oct 16, 2014
@vladia vladia Attempt to implement #332 - flattening layers cf4c5d8
@monokrome

@txomon The COMMIT OFF can be implicit in your example.

If there isn't a COMMIT provided, then one can assume COMMIT OFF for the Dockerfile. This 100% works for backward compatibility. Docker can start flattening commits only once it sees a COMMIT rules, and then flatten into a new layer for each successive COMMIT that follows. The OFF and ON conditions just make things more complicated with no value add.

It might make sense to use other formats as well, for instance you might want to use LAYER at the start of a layer instead of COMMIT at the end if that's a preferred syntax.

I like @jdmarshall's idea here of ignoring the rest of the line as well, although the name STEP seems odd. I don't think that a user who doesn't know what a layer is really would care about this problem in the first place, but even if they did - it still doesn't read right to me. The choice of colons would be up to the user if everything after LAYER was simply ignored which would allow the command to be used with or without additional notes, too.

Doing this would suffice all previously mentioned cases as far as I can tell:

RUN apt-get update -yy
RUN apt-get upgrade -yy

LAYER 1: Install services
RUN apt-get install nginx -yy
RUN apt-get install postgres-server -yy

LAYER 2: Index file
WORKDIR /tmp/
RUN curl -O http://example.com/index.html

CMD ["cat", "/tmp/index.html"]

Note that if there were RUN commands before a LAYER, it could just create a separate layer for each of them to ensure things are backwards compatible. This avoids the need for ON and OFF style rules.

The only case missing here would be if someone wanted to do LAYER OFF in the middle of a Dockerfile, but that seems like an odd use case to me.

@txomon
txomon commented Oct 17, 2014

I really like that idea Brandon!! Anyway, is there any plans to implement
it?

On Fri, Oct 17, 2014 at 7:19 AM, Brandon R. Stoner <notifications@github.com

wrote:

@txomon https://github.com/txomon The COMMIT OFF can be implicit in
your example. If there isn't a COMMIT provided yet, then one can assume COMMIT
OFF. This 100% works for backward compatibility. Docker can start
flattening commits once it sees a COMMIT rule, and then flatten into a
new layer for each successive COMMIT that follows. The OFF and ON
conditions just make things more complicated with no value add.

It might make sense to use other formats as well, for instance you might
want to use LAYER at the start of a layer instead of COMMIT if that's a
preferred syntax.

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

Javier Domingo Cansino

@cpuguy83
Contributor

What if... we don't change how the caching itself works, but rather on push/save the image is squashed up to the first layer after the parent image.

So:

FROM ubuntu
RUN apt-get update && apt-get install build-essential
ADD . /opt/myapp
WORKDIR /opt/myapp
RUN ./configure && make build
RUN make install
RUN apt-get remove -y build-essential

When that image is built, it's built as normal. You can rebuild, use the cache, etc.
Then when you push it up, it is sqaushed all the way to RUN apt-get update && apt-get install build-essential, so all the new stuff is on a single layer and preserves the layers from the parent image.

@phemmer
Contributor
phemmer commented Oct 17, 2014

@cpuguy83 Perhaps. However even locally there might be a benefit. When I do a docker build, docker often takes several seconds for each layer to be created. I imagine there's a lot of background stuff being done such as converting the intermediate container to an image, and then starting a new container with that image. This gets rather annoying when your layers are trivial operations such as ENV ... and CMD .... If disabling layering for these changes speeds up the build, I would welcome it.

@cpuguy83
Contributor

@phemmer I believe no-op stuff, in an enhanced version of the image format, will no longer create filesystem commits.
So with stuff like VOLUME, ENV, etc. there won't actually be an fs snapshot created.

@thaJeztah
Member

@cpuguy83 I like your idea, it's a fresh look at the idea, and produces the expected result.

The only downside I see with caching the intermediate layers is disk usage, for example if an intermediate layer is generating a huge file, and is removed in a later layer. I gues that --no-cache would solve that.

So, nice!

@cpuguy83
Contributor

Sure, but that's the downside of pretty much any caching.

@thaJeztah
Member

Yup πŸ˜„ just thought it would be worth mentioning, because this issue is about saving space (primarily in the final image).

So, again, all +1! With this change, people will be able to write clean Dockerfiles, not having to concat RUNs to save space. Count me in!

Possibly enable/disable the new behavior via a flag for backwards compatibility?

@monokrome

@borromeotlhs I think that the COMMIT syntax is only redundant in the previous example, but without it it the possibility of the following is impossible:

INCLUDE Dockerfile.dependencies
INCLUDE Dockerfile.base
COMMIT

INCLUDE Dockerfile.web
COMMIT

The last paragraph in my previous comment explains a few of the pitfalls of assuming a COMMIT. I don't feel like INCLUDE being coupled to COMMIT syntax would make much sense at all. An INCLUDE is just directly injecting the contents of another file - which may also have it's own COMMIT statements.

@tcurdt
tcurdt commented Apr 29, 2015

Please - no separate files (if required as such).

ADD file.tar.gz /root/
WORKDIR /root/file
RUN make
SQUASH
RUN make install
RUN rm -rf /root/file
SQUASH

Why not just SQUASH (or COMMIT)

@borromeotlhs

Squash would flatten images, but my original goal with flattening was to
hide some necessary steps to create a layer that I don't want others to
peer into. hence, the whole inclusion of another dockerfile that, say,
injects some public and private key info into containers so they are
automatically able to access another external site (s3,etc) is good as I
can then dictate, via multiple FROMS to incorporate another Dockerfile that
I as an end user may not have control over but could pull like images.

Commit on/off will definitely flatten images, though the need to constrain
layers to fit under arbitrary limited seems to be going away.

As of right now, we also "assume commit" regardless, so I again don't see
how it's bad or dangerous.

Being able to add opacity akin to the from line seems to be a win as it
also doesn't force brackets, on/off keywords or delimiters of any sort.

Cheers,
TJ

On Tue, Apr 28, 2015, 18:55 Torsten Curdt notifications@github.com wrote:

Please - no separate files (if required as such).

ADD file.tar.gz /root/
WORKDIR /root/file
RUN make
SQUASH
RUN make install
RUN rm -rf /root/file
SQUASH

Why not just SQUASH (or COMMIT)

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@jdmarshall

@monokrome dozen? Kind of hyperbolic don't you think?

I can't imagine anyone ever using squash and still ending up with twelve layers at the end of the process. I was thinking three or four (before, secret, after). What sort of docker image are you making?

@monokrome

@jdmarshall I think that it's not dramatic to suggest that a dozen Dockerfiles would be used in a complicated production system isn't outlandish. With that said, I also think that the only rhetoric here is that of the person pointing out a single word in a previous comment instead of trying to understand the suggestion which I was trying to make.

I don't see 4 Dockerfiles solving the problem any more elegantly, and I feel like any number greater than 1 potentially exhibits the same result. Possibly even 1 does.

@jdmarshall

@monokrome

It seems to me that having a dozen Dockerfiles seems like less maintainable then just dealing with all the unnecessary layers, so it kind of brings us back to the original issue here.

Of course later you talked about another way to have multiple files so now I don't know what to think.

A dozen is a lot. In most software that is code for "too many" and also seems to be the argument you're making, so I'm not hanging on a word, I'm hanging on an inequality and your unstated but very clearly implied "no".

Many of us are only concerned about a couple of layers. For some that's apt-get churn, for others it's an authenticated GET request, or a password. "A couple" is not just doable, it's reasonable.

Far more reasonable than introducing keywords that affect the semantics of the statements between them (known generally as Blocks) and not calling a spade a spade.

There are no block semantics in Docker now, aside from base images. Please think twice before setting a precedent that will encourage people to ask for more block semantics.

But if you still must add them in the end, don't half-ass it. Call a spade a spade.

Also, if you are honestly concerned about too few layers slowing transfer time, I wouldn't worry too much. First we're kinda here talking about trying to make smaller images, so it should be a net gain. But if push comes to shove we could probably figure out how to do simultaneous Range requests (like the DownloadThemAll browser addon does), but if everyone is behaving the same way it's all a wash.

Me, I have a crappy network connection at home and often so do the people I'm trying to pitch Docker to, as part of their toolchain. So I care more about total size than throughput. I also care about credentials but I work around that (would be nice to get rid of the workarounds though)

@jdmarshall

@borromeotlhs if you want to run your sensitive code at the beginning of someone else's docker build that might work, but if you run it in the middle of theirs you can't protect your credentials.

I could just swap out curl and sftp and ssh and git on my image and copy your keys without you being able to detect it. Stay with the base image idea if you want to avoid trouble.

@borromeotlhs

@jdmarshall, I disagree that more docker files somehow equates to more
total bandwidth. if anything, you end up using less bandwidth, especially
if those dozen or so dockerfiles just to common idioms that are necessarily
private.

I agree that I don't like block syntax though, and it is under this precept
that I offer up the multiple FROM idiom. it is succinct, there are no new
keywords, it works at first glance how one would suspect from visual
inspection of a parent dockerfile, it is reusable, and I feel that it could
be pulled (and stored locally like images === no extra bandwidth ) into
whatever containers needed it.

Now, I'm again only interested in deleting layers due to the presence of
things that I don't want in a shipping dockerfile. I don't necessarily
care about minimizing layers as I think later limitations are a problem
that will solve itself in the future.

On Tue, Apr 28, 2015, 22:53 Jason Marshall notifications@github.com wrote:

@monokrome https://github.com/monokrome

| It seems to me that having a dozen Dockerfiles seems like less
maintainable then just dealing with all the unnecessary layers, so it kind
of brings us back to the original issue here.

Of course later you talked about another way to have multiple files so now
I don't know what to think.

A dozen is a lot. In most software that is code for "too many" and also
seems to be the argument you're making, so I'm not hanging on a word, I'm
hanging on an inequality and your unstated but very clearly implied "no".

Many of us are only concerned about a couple of layers. For some that's
apt-get churn, for others it's an authenticated GET request, or a password.
"A couple" is not just doable, it's reasonable.

Far more reasonable than introducing keywords that affect the semantics of
the statements between them (known generally as Blocks) and not calling a
spade a spade.

There are no block semantics in Docker now, aside from base images. Please
think twice before setting a precedent that will encourage people to ask
for more block semantics.

But if you still must add them in the end, don't half-ass it. Call a spade
a spade.

Also, if you are honestly concerned about too few layers slowing transfer
time, I wouldn't worry too much. First we're kinda here talking about
trying to make smaller images, so it should be a net gain. But if push
comes to shove we could probably figure out how to do simultaneous Range
requests (like the DownloadThemAll browser addon does), but if everyone is
behaving the same way it's all a wash.

Me, I have a crappy network connection at home and often so do the people
I'm trying to pitch Docker to, as part of their toolchain. So I care more
about total size than throughput. I also care about credentials but I work
around that (would be nice to get rid of the workarounds though)

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@jdmarshall

@borromeotlhs re: lower bandwidth, I agree entirely. Sorry if that wasn't clear.

@borromeotlhs

no prob. I just love the serious consideration this feature is getting :)

On Tue, Apr 28, 2015, 23:11 Jason Marshall notifications@github.com wrote:

@borromeotlhs https://github.com/borromeotlhs re: lower bandwidth, I
agree entirely. Sorry if that wasn't clear.

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@jdmarshall

@monokrome I think essentially I'm saying that I've come around to the --one-layer-per-run solution brought up back at the end of January (I forgot that had been put forward). Are you no longer in favor with this solution?

@jdmarshall

@blaggacao +1 on the list either flattening the whole thing or making the entire lifecycle explicit would solve a bunch of problems. I didn't assume we could know a priori what those stages would be but maybe we actually can...

I'm not at all happy with the heavy handed usage of && in RUN. Everybody does it and everyone who writes about docker tells more people to do it as well. Chaining has all of the problems of squashed layers and is harder to read to boot.

@txomon
txomon commented Apr 29, 2015

Just remember that if I have to compile from source various libs and the
program, i may now want to have there at the end nothing bur binaries,
deleting private keys i might have used, sources etc.

You can check for use cases in previous posts, as i think we are failing in
the same arguments once and another.

On Wed, Apr 29, 2015 at 8:07 AM TJ Borromeo notifications@github.com
wrote:

@jdmarshall, I disagree that more docker files somehow equates to more
total bandwidth. if anything, you end up using less bandwidth, especially
if those dozen or so dockerfiles just to common idioms that are necessarily
private.

I agree that I don't like block syntax though, and it is under this precept
that I offer up the multiple FROM idiom. it is succinct, there are no new
keywords, it works at first glance how one would suspect from visual
inspection of a parent dockerfile, it is reusable, and I feel that it could
be pulled (and stored locally like images === no extra bandwidth ) into
whatever containers needed it.

Now, I'm again only interested in deleting layers due to the presence of
things that I don't want in a shipping dockerfile. I don't necessarily
care about minimizing layers as I think later limitations are a problem
that will solve itself in the future.

On Tue, Apr 28, 2015, 22:53 Jason Marshall notifications@github.com
wrote:

@monokrome https://github.com/monokrome

| It seems to me that having a dozen Dockerfiles seems like less
maintainable then just dealing with all the unnecessary layers, so it
kind
of brings us back to the original issue here.

Of course later you talked about another way to have multiple files so
now
I don't know what to think.

A dozen is a lot. In most software that is code for "too many" and also
seems to be the argument you're making, so I'm not hanging on a word, I'm
hanging on an inequality and your unstated but very clearly implied "no".

Many of us are only concerned about a couple of layers. For some that's
apt-get churn, for others it's an authenticated GET request, or a
password.
"A couple" is not just doable, it's reasonable.

Far more reasonable than introducing keywords that affect the semantics
of
the statements between them (known generally as Blocks) and not calling a
spade a spade.

There are no block semantics in Docker now, aside from base images.
Please
think twice before setting a precedent that will encourage people to ask
for more block semantics.

But if you still must add them in the end, don't half-ass it. Call a
spade
a spade.

Also, if you are honestly concerned about too few layers slowing transfer
time, I wouldn't worry too much. First we're kinda here talking about
trying to make smaller images, so it should be a net gain. But if push
comes to shove we could probably figure out how to do simultaneous Range
requests (like the DownloadThemAll browser addon does), but if everyone
is
behaving the same way it's all a wash.

Me, I have a crappy network connection at home and often so do the people
I'm trying to pitch Docker to, as part of their toolchain. So I care more
about total size than throughput. I also care about credentials but I
work
around that (would be nice to get rid of the workarounds though)

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@gdm85
Contributor
gdm85 commented Apr 29, 2015

I've been tinkering with Docker for ~1 year now, and at some point I have removed this image layers limit (as I am not using AUFS either) because yes, to aim to production you need a lot of steps in your images and packing instructions with concatenation '&&' or scripts is not elegant, nor readable (and against providing all the instructions in a Dockerfile, for that matter).

If this feature gets in (as SQUASH/COMMIT, with or without the block initiator/braces), I'll probably use it at the end of each Dockerfile that leads to a production image, while never do it for a high-flux/development image (as you want to benefit maximally from caching).

Although I'd keep any day the removal of the layers limit, I would start using SQUASH/COMMIT because that puts back me (the user) in control of how I want to use/misuse layers.

@muayyad-alsadi
Contributor

I don't like blocks { }

dockerfiles are sequence of commands so AUTOCOMMIT ON/OFF is not a block because it's meaning less to nest them.

the use case would be like this:

AUTOCOMMIT OFF
ADD PRIVATEKEY
RUN git clone ...
RUN rm PRIVATEKEY
RUN rm -rf .git
RUN make
RUN find . -name '*.o'
COMMIT
  1. the objective is to not include the private key used to clone the code (we don't want to ship a layer with sensitive data)
  2. we need to save space by omitting the layer with big .git directory and building intermediate files (we don't want to ship layers with useless files: .o files, .git directory, yum/apt cache ...etc)

I would think of layers to be

  1. base distro layer
  2. base dependencies
  3. base app with config...etc.
  4. the latest milestone release of the app
  5. the latest release of the app

but I don't think they should be

  1. base distro
  2. distro with some changed config file (ex. repo)
  3. package installed
  4. package configured (ex. put /etc/httpd/conf/httpd.conf)
  5. another packages installed
  6. another package configured
  7. touch a log file
  8. chmod a log file
  9. add user
  10. add key
  11. compile object files
  12. link object files
    ....

the layers are just bunch of command results of no high level meaning

@duglin
Contributor
duglin commented Apr 29, 2015

@gdm85 your comment got me wondering about something. If you plan on squashing production-target images, but not dev-target images how do you see controlling that? Simply adding a COMMIT to your Dockerfile after you're done developing it? Or do you see the same Dockerfile being used in both environments but have the need for some kind of if-statement around the COMMIT to enable/disable it based on what you're doing at that moment?

@thaJeztah
Member

while never do it for a high-flux/development image (as you want to benefit maximally from caching).

@gdm85 the MARK..SQUASH instructions still make optimum use of the cache; see #12198 (comment). Both individual layers and the squashed result are cached, so that's a win-win.

@gdm85
Contributor
gdm85 commented Apr 29, 2015

@thaJeztah nice one! Thanks for pointing it out. Yes, that would be best because one could tinker around on the registry/development server, and then staging/production peers get the squashed images :)

@duglin right now I am splitting images (and common ancestors) by using different Dockerfile's, trying to organize them hierarchically as much as it is reasonable to do. I'd say yes, I would like a conditional usage - depending on whether I want to consume or develop on the image, so what @thaJeztah mentioned seems like a good feature for such use-case.

So - even if technically the fully-squashed image is an actual duplicate of the image made of N combined layers, I think it would be best suited for the consumption of the images (in staging/production) - also for the secrets/privacy management concerns that were mentioned by others here.

@borromeotlhs

why even bother making commit on/off syntax visible in production
dockerfiles? I mean, layers in images aren't even really related to a
dockerfile as one approach would be to squash the images layers apart from
messing with a dockerfile.

so, the call to take control of commit syntax in development, and then only
deal with squash or merge on production deployment points to the fact that
we should have control over layers, but that end users of our dockerfile
can't get a dockerfile at all if there are any steps that contain private
information (layers which contain cruft aren't really an issue, they just
add up overall size).

So again,I think that going with multiple Dockerfiles, and treating sub
Dockerfiles as potentially binary like images, is still the right way to
go. akin to distributing source (base Dockerfile) and then linking them to
objects when compiling ( build command with sub Dockerfile.whatever files )
would solve all use cases.

On Wed, Apr 29, 2015, 05:40 Sebastiaan van Stijn notifications@github.com
wrote:

while never do it for a high-flux/development image (as you want to
benefit maximally from caching).

@gdm85 https://github.com/gdm85 the MARK..SQUASH instructions still
make optimum usage of the cache; see #12198 (comment)
#12198 (comment). Both
individual layers and the squashed result are cached, so that's a
win-win.

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@jdmarshall

@borromeotlhs are you tracking #9176 ?

If they can sort out the cache determination I think it fulfills your concerns.

@monokrome

@muayyad-alsadi Wouldn't this fulfill your request more elegantly?

ADD PRIVATEKEY
RUN git clone ...
RUN rm PRIVATEKEY
RUN rm -rf .git
RUN make
RUN find . -name '*.o'
COMMIT # Nothing commits until the first COMMIT when a COMMIT is present
@muayyad-alsadi
Contributor

Yes it does fulfill. Is it more elegant? I'm not sure because this way the
meaning of "commit" is overloaded with an implied condition. And explicit
is better than implicit. If some sort of include feature got introduced I
won't be able to see if there is commit in them or not.

On Wed, May 6, 2015, 12:56 AM Brandon R. Stoner notifications@github.com
wrote:

@muayyad-alsadi https://github.com/muayyad-alsadi Wouldn't this fulfill
your request more elegantly?

ADD PRIVATEKEYRUN git clone ...RUN rm PRIVATEKEYRUN rm -rf .gitRUN makeRUN find . -name '*.o'
COMMIT # Nothing commits until the first COMMIT when a COMMIT is present

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@sleaze
sleaze commented May 7, 2015

It's been over 2 years since this functionality was filed and there is still no working solution for this merged.. yikes!

Guess I'll just let my images grow unbounded.

@PierreR
PierreR commented May 8, 2015

πŸ‘

@monokrome

@muayyad-alsadi I think that your conception that there are actually 2 conditions is a misconception. Either you tell it when to make a commit explicitly, or else it would assume a commit for every action (for backwards compatibility reasons). What condition is the meaning overloaded with?

@morallo
morallo commented May 12, 2015

+1 for the COMMIT / AUTOCOMMIT ON / AUTOCOMMIT OFF proposal from @txomon.
It feels like the simplest solution, a lot more explicit than disabling AUTOCOMMIT with a COMMIT.

If I understand @monokrome alternative correctly, if you want to disable autocommit only for a specific set of instructions (adding private keys and removing them, or installing build tools and removing them), you would have to use COMMIT explicitly for every other instruction from the beginning of the file.

@zoechi
Contributor
zoechi commented May 13, 2015

+1 MARK..SQUASH

I like

  • AUTOCOMMIT OFF/COMMIT/AUTOCOMMIT ON for it's simplicity
  • MARK..SQUASH for preserving cache usage

So in overall I think MARK..SQUASH is the better solution because it has the benefits of both, but might be more costly to implement.

I really hope to soon get rid of these cumbersome and error-prone

RUN \ 
  abc && \ 
  def && \ 
  ghi && \ 
  jkl && \ 
  mno && \ 
  pqr && \ 
  stu 
@ewindisch
Contributor

Rather than 'MARK..SQUASH', you could just do 'SQUASH $N' to squash $N previous layers. This would be easier to parse and would continue to treat the Dockerfile as a RPN / layered format rather than introducing branches.

@tiborvass
Contributor

Review with @duglin @crosbymichael @diogomonica @LK4D4 @ewindisch @cpuguy83 @vishh

The problem with this issue is that it provides a solution to a problem that yet has to be defined

The bigger question is what is the end goal for users who need to squash layers? Faster pull/push? Removing some layers (with secrets)? Maximum number of layers?

It feels like all these problems are implementation details and solving them by exposing squashing is probably the wrong way.

We suggest to address each independent issue you guys have, in more focused separate issues.

For relieving the immediate pain, we also suggest updating the documentation on how to manually squash layers if people really need it, with external tools. @crosbymichael volunteered on writing such a tool.

We're closing this issue. Would love to continue the debate on more focused issues.

@tiborvass tiborvass closed this May 14, 2015
@sleaze
sleaze commented May 19, 2015

@ewindisch how am I supposed to know how many layers have been created? Lol..just imagine if SQL forced you to recall how many statements had been executed.. COMMIT 5 would commit the past 5 statements.

What a terrible idea, mate.

@sleaze
sleaze commented May 19, 2015

@tiborvass Why should we have to continue to debate with you? Why hasn't this been implemented? What can we do to get this functionality into docker?

@stevenschlansker

I also have been frustrated at the tendency of the Docker project to close entirely legitimate issues such as this one with non-fixes or "open a different bug later and let's start the whole process over"...

Any future discussion will either be referring to this ticket or rehashing the exact same ideas again. Closing this ticket with "we'll split this up into smaller tickets" would be a palatable resolution if anyone had actually created said tickets, or if there was any indication that anyone actually cared to reach resolution for the numerous problems faced by people trying to run Docker, but alas we continue to despair...

@thaJeztah
Member

@sleaze please watch your language, and don't turn this in a "personal" issue.

We don't have to all agree on decisions that are made (I would have liked this feature myself), and you are absolutely free to express your opinion, but please do so in a reasonable way.

@thaJeztah
Member

Closing this ticket with "we'll split this up into smaller tickets" would be a palatable resolution if anyone had actually created said tickets,

Agreed. Let's do so. Can you create a bullet list of topics you think make sense as separate issues here first? We can discuss them shortly and then create those, and link them from here.

I have to go in a few minutes, but will watch this issue, if needed, I can be on IRC tomorrow

@ewindisch
Contributor

@sleaze A Dockerfile is not SQL. We know what the previous statements are... they would be directly preceding the SQUASH command in the Dockerfile. I agree it's somewhat ugly looking and not as user-friendly.

The main disadvantage on technical merit would be that we wouldn't know how many layers were created from ONBUILD instructions.

@ewindisch
Contributor

@stevenschlansker I think that's a fair criticism. This issue was closed because it was seen as a proposed solution, where we might instead like to see the problem proposed and discussed. If/once squashing is decided as a solution, we might discuss the details of implementation. Reading the original report, the use-case for this particular issue was:

@unclejack:
There are some cases where one starts with a base image (or another image), changes some large files in one step, changes them again in the next and deletes them in the end. This means those files would be stored in 2 separate layers and deleted by whiteout files in the final image.

These intermediary layers aren't necessarily useful to others or to the final deployment system.

There are a number of ways to solve this problem which are NOT squashing. Alternative image formats, for instance may solve this natively, of which I have several ideas (and we've even seen one bittorrent-based PoC offer one potential solution).

I hope this helps.

@tcurdt
tcurdt commented May 20, 2015

To break it down I think the people discussing here want to to get rid of running everything in one single RUN just to avoid inflating the size of the image by intermediate layers and/or try to deal with the maximum number of layers limitation. Is that a fair description of the problem?

Frankly speaking I don't see how a different image format or bittorrent(??) could help with the size inflation - either you store intermediate steps or you don't.

In fact if you compare the current model of operation with a git workflow - usually you would not want a commit after each and every modification either. You group modifications in commits. Same would be nice for layers. This isn't really squashing - but controlling when layers are being created.

And I agree with @stevenschlansker - closing this issue without opening new ones that actually address the way forward was not very helpful.

@TravisCardwell

@tiborvass @crosbymichael

My goals in managing layers are as follows:

Improved Dockerfiles

In order to minimize layers, it is common practice to combine commands using
backslashes, which significantly reduces the utility of the file format. It
is error-prone: if you insert a line and forget to add the backslash, the
build will break. It promotes poor organization: separation of unrelated
commands results in additional layers, so they are often thrown together. It
promotes poor documentation: adding comments in between continued lines is not
possible.

With better layer management, we would be able to write better Dockerfiles.

Organization

I want every layer that I create to exist for a reason. Every layer should be
one that is run and/or a parent for other layers. As a general rule of
thumb, I think that a layer probably shouldn't exist if there is no need to
tag it.

COW-Filesystem Layer Minimization

Increased layers increases the depth of the copy-on-write filesystem mount. I
have never had a problem due to this, but having many unnecessary layers (such
as layers that contain only metadata that is later overridden) just seems
wasteful and avoidable.


Personally, I am not worried about the following:

Faster Push/Pull

I do not think that "squashing" layers would significantly effect transmission
speed, at least for my use case.

Removing Layers With Secrets

I do not use layers to maintain a history of how they were built, as I am of
the opinion that using Docker like Git is an anti-pattern. (The deprecation
of docker images --tree supports this feeling.) My images are built from
revision-controlled source files.

Maximum Number Of Layers

I currently avoid this issue by combining commands with backslashes.


For me, an optimal solution would be a flag for docker build that causes it
to process a Dockerfile and build a single layer instead of a layer per
command. A separate "squash" tool would be sufficient, however, as long as it
can squash a specified range of layers (keeping a specific parent layer); I
would simply make a docker-build script that squashes every build that I
perform.

@borromeotlhs

the reduction of writes to disk for unnecessary layers is a huge concern to
reduce io and general wear on hard disks. less layers needlessly pulled
decreases time to deploy and should manner to everyone who is charged to
host their cloud runtime.

So many reasons to implement this other than getting rid of the irritating
(RUN * &&)* syntax.

On Wed, May 20, 2015, 04:50 Travis Cardwell notifications@github.com
wrote:

@tiborvass https://github.com/tiborvass @crosbymichael
https://github.com/crosbymichael

My goals in managing layers are as follows:

Improved Dockerfiles

In order to minimize layers, it is common practice to combine commands
using
backslashes, which significantly reduces the utility of the file format. It
is error-prone: if you insert a line and forget to add the backslash, the
build will break. It promotes poor organization: separation of unrelated
commands results in additional layers, so they are often thrown together.
It
promotes poor documentation: adding comments in between continued lines is
not
possible.

With better layer management, we would be able to write better Dockerfiles.

Organization

I want every layer that I create to exist for a reason. Every layer should
be
one that is run and/or a parent for other layers. As a general rule of
thumb, I think that a layer probably shouldn't exist if there is no need to
tag it.

COW-Filesystem Layer Minimization

Increased layers increases the depth of the copy-on-write filesystem
mount. I
have never had a problem due to this, but having many unnecessary layers
(such
as layers that contain only metadata that is later overridden) just seems

wasteful and avoidable.

Personally, I am not worried about the following:

Faster Push/Pull

I do not think that "squashing" layers would significantly effect
transmission
speed, at least for my use case.

Removing Layers With Secrets

I do not use layers to maintain a history of how they were built, as I am
of
the opinion that using Docker like Git is an anti-pattern. (The deprecation
of docker images --tree supports this feeling.) My images are built from
revision-controlled source files.

Maximum Number Of Layers

I currently avoid this issue by combining commands with backslashes.

For me, an optimal solution would be a flag for docker build that causes
it
to process a Dockerfile and build a single layer instead of a layer per
command. A separate "squash" tool would be sufficient, however, as long as
it
can squash a specified range of layers (keeping a specific parent layer); I
would simply make a docker-build script that squashes every build that I
perform.

β€”
Reply to this email directly or view it on GitHub
#332 (comment).

@thaJeztah
Member

External tooling also cannot be used for automated builds on the Docker Hub Registry (#12198 (comment))

@txomon
txomon commented May 20, 2015

This issues have been already explained multiple times and yet I see this is the 3rd or 4th iteration in the loop:
1.- Explain problems
2.- Discuss a solution
3.- Bikeshed on the exact form of the solution
4.- Say the issue is not relevant because of not defined problem
5.- Start over.

I have seen there is no intention in 2 years this issue has been opened to implement it, so I will just unsubscribe, please don't @mention me, as I want to avoid getting notifications.

@foxx
foxx commented Nov 26, 2015

Shame this isn't supported

@monokrome

I think that txomon's (not mentioning by request) example of the process happening here may be a bit more negative than the situation actually is, but I think that it's still a valid concern that this hasn't gone anywhere yet.

There are a lot of issues where this is referenced and then they are closed, but now we are all hanging out in the kitchen sink talking about 20 different things. It may be worth considering leaving the other issues open - at least until this is figured out.

Assuming that it goes anywhere, it is completely possible that those issues aren't going to all be solved by the result of this thread now that this thread is expected to account for so many different use cases My other concern here is that someone may have a pull request related to another issue which maybe doesn't completely solve this issue's case but solves one of the other ones.

I think that it's possible that having so many different problems being solved in a single thread isn't helping help solve any of them.

@zerthimon

I'd like to have a way to unite multiple FS layers into a single layer.

@foxx
foxx commented Dec 29, 2015

Get in line, it ain't gonna happen any time soon, at least not properly. There are third party tools which claim to support this, but they are flakey at best.

For now you can use something like the following, it's pretty nasty but it's the closest you'll get for now.

#!/bin/bash
echo "Flattening base image.."
SRC=myrepo:version
DST=myrepo:version_flat
ID=$(docker run -d ${SRC} /bin/bash)
docker export $ID | docker import - ${DST}

If you want a better solution, perhaps take a look at rkt and acbuild

@zerthimon

Thanks @foxx

@monokrome

Well, this is an extremely disappointing way to leave dozens of GitHub issues unresolved without any suggestion of following up in the future. πŸ‘Ž

@thaJeztah
Member

Flattening images / nested builds is still on the radar. Making changes in that area is on hold until the builder is split from the daemon; https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax, which is actively worked on, but requires a lot of refactoring.

@TomasTomecek
Contributor

@tiborvass wrote:

For relieving the immediate pain, we also suggest updating the documentation on how to manually squash layers if people really need it, with external tools. @crosbymichael volunteered on writing such a tool.

Did @crosbymichael write such tool? Will it be even possible to write such tool after 1.10 is out?

@bmeng
bmeng commented Jan 19, 2016

@foxx cool!

@foxx
foxx commented Jan 19, 2016

@TomasTomecek The other third party tool was docker-squash, but it's /very/ unstable. The best you can hope for right now is the approach I mentioned earlier.

Sadly the docker export command does not support ranged extraction either, therefore you can only merge from start to X, rather than X to X. This means that if you perform a merge on every build release, you'll have to re-upload the entire image, rather than a small portion of it.

Another option is to use Docker in combination with pip, where you package up your application as a pypi package and then only push new Docker containers when you need to update your system libs/deps (which ideally should be at least once a day, to ensure you're getting security patches). However, this means having a bootstrapper inside your container which is then running on your production boxes, a particularly nasty devops pattern.

You can reduce the impact of slow uploads slightly by using something like quay.io or Amazon ECS, and so long as you're building the containers on a reliable and speedy CI service such as CircleCI or Travis CI, then you can /just about/ achieve "github to production" in ~10 mins. This will require you to use a lot of provisioning optimizations (such as apt-fast) as well as segregated Dockerfile builds so that you can push application updates without having to re-build the system container.

None of this advice changes the fact that Docker is fundamentally flawed, but you at least have some knowledge on how to workaround these problems if you so choose to put Docker into production. As for development usage, you'll have to put up with the slow speeds or move to a different solution such as Vagrant.

I'll be touching on this topic more in my next blog post, see here for previous discussion.

tl;dr - It's unlikely that you'll see any improvement in this area for at least another 18 months, by which time Rocket should have reached production maturity.

@TomasTomecek
Contributor

@TomasTomecek The other third party tool was docker-squash, but it's /very/ unstable. The best you can hope for right now is the approach I mentioned earlier.

The unstability of docker-squash was the reason we wrote our own tool: https://github.com/goldmann/docker-scripts#squashing and we are using it in production now.

Sadly the docker export command does not support ranged extraction either, therefore you can only merge from start to X, rather than X to X. This means that if you perform a merge on every build release, you'll have to re-upload the entire image, rather than a small portion of it.

This is the exact reason we discarded the solution.

@foxx
foxx commented Jan 20, 2016

Interesting, your library seems to have a decent amount of tests as well. I'll give this a mention in the upcoming article, as it looks to serve as a decent workaround.

@beorn
beorn commented Jan 31, 2016

I think flattening of layers is something that should be a standard part of Docker, via syntax in the Dockerfile, so that all of the vendor-provided docker images on Docker Hub can use it. It's important to provide small base images, and it'd be a shame if that had to be done through external tools importing/exporting, losing the transparency we have in Dockerfiles describe how images came to being.

Personally, I think a simple syntax that should work is just to allow any Dockerfile command (or any where it makes sense) to be preceded with an AND keyword, a slightly generalized version of the above feature request, e.g.,

WORKDIR /app
AND COPY requirements.txt /app/
AND RUN apt-get update
AND RUN apt-get install some-build-dependencies
# Install app
# (comments are ignored, so following commands are still in the same layer)
AND RUN pip install -f requirements.txt
AND RUN apt-get purge --auto-remove -y some-build-dependencies
AND RUN apt-get autoremove && apt-get clean

# following command not part of above layer
RUN apt-get install

It's easy enough to use # comments to document each layer and what's going on - that's what comments are for anyways.

Also, for when you're troubleshooting builds, it'd probably be good to have a way to tell Docker to ignore the AND keyword and create layers anyways (through a command line option or similar). So you can develop docker images with the full power of the per-line cache, but production/distributed images are not built that way.

@cgrandsjo

My opinion is that when working with a Dockerfile the default behaviour should be to only create one additional layer on top of the base image, independent of how many commands the Dockerfile contains.

If you for some reason want to add an extra layer then there should be an ADDLAYER command available that you could insert at any line in the Dockerfile to separate the layers.

Default behaviour, creates one layer on top of the Ubuntu image:

FROM ubuntu
MAINTAINER Me me@host.com
INCLUDE Dockerfile.dependencies
INCLUDE Dockerfile.base
INCLUDE Dockerfile.web

Using ADDLAYER to create two layers on top of the Ubuntu image:

FROM ubuntu
MAINTAINER Me me@host.com
INCLUDE Dockerfile.dependencies
ADDLAYER
INCLUDE Dockerfile.base
INCLUDE Dockerfile.web

@justincampbell

@cgrandsjo That would cause the cache to not be used by default.

@cgrandsjo

@justincampbell: Sorry, please elaborate your answer. Omitting the ADDLAYER command actually means that ADDLAYER is added to the end of the file "silently" and next time you build with the Dockerfile the cache will be used because an additional layer was created.

Update:
Actually I just realized that it depends on how you modify the Dockerfile, whether the cache will be used or not. Maybe the default behaviour should be as it is right now and for those who know what they are doing, there should be a docker build option to "squash" intermediate layers if that is desired.

@mishunika

Yay, now I have images of around 30Gigs in size, and their actual size should not be more than 10G!

So yeah, just stepped into the same difficulty, and I was thinking that Dockerfile is really lacking some kind of COMMIT action (Inspired from the DB transactions) to decide where a layer should end.
Then I found this issue and I've read other commit related ideas that are in fact the same. I think that user should be able to specify explicitly what the layers should contain and when they should start/end.

Furthermore, in my opinion, the issue with caching is not a big one though. It can be the same as now, but delimited by the commit/addlayer levels, if something has changed in a such block, then no cache is used at all for this level. And even more, the commit thing can be optional, and the default behavior can be maintained as it is now.

@foxx
foxx commented Feb 17, 2016

@mishunika See my previous answer, and also @TomasTomecek, for a workaround. Don't bother trying to push this proposal with Docker, it ain't going to happen any time soon (see previous comments from core devs)

@campbel
campbel commented Feb 19, 2016

@TomasTomecek is https://github.com/goldmann/docker-scripts#squashing a suitable tool for reducing image size?

For instance given a docker file:

FROM baseimage

RUN apt-get install buildtools
ADD / /src

RUN  buildtools build /src

RUN apt-get remove buildtools
RUN rm -rf /src

After building and squashing, would the resulting image lose the size of the src and buildtools?

@goldmann
Contributor

@campbel That's correct. This tool will remove unnecesary files. I haven't tested it with ADDing root filesystem (/) to the image (it's generally, a very bad idea), but I understand that this is just an example.

Please note that Docker 1.10 is still in works (see v2 branch). Feel free to open any issues.

@yoshiwaan

I think the AND and ADDLAYER options mentioned above are useful for certain situations (such as controlling what to and what not to cache), but if you are chaining builds from images you control and later builds are removing things from the upstream builds then they don't help with the size problem.

Something as simple as a --squash option to docker build which looks through the layers and removes whiteout files and all underlying files in above layers (correct me if I'm wrong but that's my understanding of how it works) would be extremely useful.

It's the same as when you use git really, sometimes you want to rebase, sometimes you want full commit history and sometimes you just want to squash all that noise out of there.

@sivang
sivang commented May 26, 2016

So, is this going to be a feature in docker or already solved in the stable release somehow?

@cpuguy83
Contributor

@sivang Maybe, now that the image format has been changed: #22641.

Please don't spam the PR, though.

@foxx
foxx commented May 26, 2016 edited

@sivang I'll be surprised if you see this feature in a release before 2017. If you need a quick fix, read previous suggestions or check out the far more superior option, rkt

@thaJeztah
Member

Thanks for the commercial break, @foxx

@sivang
sivang commented May 26, 2016

Well, I used jwilder's docker-squash, it seemed to have done the flatten
job but loading the image back doesn't show it on the docker images list...

On Thu, May 26, 2016 at 6:24 PM, Sebastiaan van Stijn <
notifications@github.com> wrote:

Thanks for the commercial break, @foxx https://github.com/foxx

β€”
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#332 (comment)

@JonathonReinhart
JonathonReinhart commented Jul 1, 2016 edited

Yet another disappointment from the Docker team; not because some feature doesn't exist, but because of a dismissive attitude by the maintainers.

@tiborvass said (#332 (comment)):

The problem with this issue is that it provides a solution to a problem that yet has to be defined
...
We're closing this issue. Would love to continue the debate on more focused issues.

Perhaps he didn't read the original issue (which was opened over three years ago), which very clearly stated:

There are some cases where one starts with a base image (or another image), changes some large files in one step, changes them again in the next and deletes them in the end. This means those files would be stored in 2 separate layers and deleted by whiteout files in the final image.

These intermediary layers aren't necessarily useful to others or to the final deployment system.

I don't understand what is "yet to be defined" or not focused about that, but in case you need something concrete:

FROM debian

# This line produces an intermediate layer 400 MB in size
ADD local_400MB_tarball_im_about_to_install.tar /tmp

# This line installs some software, and removes the tarball.
# Lets say it produces a layer with 20 MB of binaries
RUN cd /tmp && tar xf local_400MB_tarball_im_about_to_install.tar && cd foo && make install && cd /tmp && rm local_400MB_tarball_im_about_to_install.tar

The end result is that this image is sizeof(debian) + 420MB in size, when 400 MB of it were removed.

Perhaps if issues were addressed instead of dismissed, this project wouldn't have nearly as many issues in its history as it does commits.

@cpuguy83
Contributor
cpuguy83 commented Jul 1, 2016

@JonathonReinhart This problem is this issue is discussing a particular solution rather than the problems.
In reality, squashing is a stop-gap to a particular problem that is an implementation detail of the current storage subsystem... ie, we don't need squashing if/when the storage subsystem is replaced with a better solution.

Thank you for your kind and thoughtful comments.

@JonathonReinhart

@cpuguy83 Sarcasm isn't necessary when someone is expressing frustration.

Regardless of whether or not this is the right solution, people will find this issue when looking for a solution to a very common problem. When you see that the issue is closed, you'll immediately wonder "Why was this closed? Was it fixed?", and when you see that was closed with a message essentially stating, "Sorry, too vauge, try again", that is a good way to frustrate and alienate users.

I am a big supporter of Docker, and advocate many different types of projects to use it. I think that it would greatly help the project if issues like this were handled better. Specifically, I think when @tiborvass closed this issue, it should have been locked (so the "resolution" of the issue didn't get burried in the middle of the page), and included a reference to other issue(s) where the problem(s) could be discussed in the "more focused" fashion he was advocating for.

@foxx
foxx commented Jul 1, 2016

to a particular problem that is an implementation detail of the current storage subsystem

@cpuguy83 The entire implementation of Docker is fundamentally flawed, and this issue is just one of many such issues. So unless you are planning on rewriting the entire Docker platform from scratch, then flattening images is the best you're going to get.

The problem with this issue is that it provides a solution to a problem that yet has to be defined

@tiborvass I think it's pretty clear what the problem is, don't you?

@monokrome
monokrome commented Jul 1, 2016 edited

Problem: We have n layers when the results of actions performed in order to create each one only need to be in 1 layer.
Solution: ?!?!?!?!?

@ohjames
ohjames commented Jul 14, 2016

when you see that was closed with a message essentially stating, "Sorry, too vauge, try again", that is a good way to frustrate and alienate users.

I see hundreds of people defining a very very clear problem... Hundreds of users all in unanimous agreement that docker handles layers in a way that doesn't make sense to them. Yet the people on the inside actually developing it are the only ones who feel that hundreds of community members all agreeing with each other and stating the same thing haven't "defined" themselves.

Even if I did agree that the problem wasn't clearly well defined (and I definitely don't) the way the core developers have responded to the community basically shows contempt. As for the solutions, how docker-squash manages to be so slow and delay our build time for so long on such a tiny set of layers, I don't know... can't wait for rkt.

@zerthimon

@ohjames +1
I feel the same thing. I asked for a few features before, and they all were rejected with the following reasons:

  1. It will hurt portability
  2. it will hurt security

When will this project realize, users don't like to be FORCED to have portability and security at the price of productivity.
How about adding the feature users ask for, so USER HAS THE CHOICE and DECIDES FOR HIMSELF if he wants to use it even if it hurts protability and security.

Can't wait for someone fork this project and make it more friendly to the users.

@justincormack
Member

There is an open PR for flattening #22641

@vdemeester
Member
vdemeester commented Jul 14, 2016 edited

It took me a while to decide to answer something here, but I feel I need to pin-point some stuff.

First, as @justincormack there is a PR for flattening (#22641) β€” thus maybe we could reopen that issue as we are trying to, maybe, have it built-in.

How about adding the feature users ask for, so USER HAS THE CHOICE and DECIDES FOR HIMSELF if he wants to use it even if it hurts protability and security.

I'm gonna quote Nathan Leclaire here (from The Dockerfile is not the source of truth for your image).

The Dockerfile is a tool for creating images, but it is not the only weapon in your arsenal.

Dockerfiles and docker build is only one way to build Docker images. It is the default/built-in one, but you definitely have the choice to build your image with other tooling (and there is some : packer, rocker, dockramp, s2i… to only list a few). One of the focus of Dockerfile is portability and thus this is one of the main concern when discussing features on Dockerfile and docker.

If you don't care about portability or if the Dockerfile possibilities are too limited for your use cases, again, repeating myself, you are free to use other tooling to build images. Docker does not force you to use Dockerfiles to build image β€” it's just the default, built-in way to do it.

On the Dockerfile and image building subject, I highly recommend people to watch a talk from Gareth Rushgrove at DockerCon16 : The Dockerfile Explosion and the need for higher level tools.

@rvs
rvs commented Jul 14, 2016

@justincormack Justin, what I really would love to see is very similar to #22641 but with a full power of git rebase (especially git rebase --interactive). Do you think it is feasible?

@docbill
docbill commented Jul 15, 2016

It is important to distinguish what is needed per this request, and what is
desired. What is desired is to refactor docker images into a git or git
like repository view, so all the cool features one does with git would be
possible in docker. For example, I have docker image for plex. It is
basically a fedora base, with a download of the plex build onto. I have
the container set to autobuild on the docker hub. So every time the
Fedora base changes, it rebuilds, even though most of the time the plex
download does not change. What annoys me about that one, is when I pull
the update the layer for adding the plex download is treated as brand new,
even though byte per byte is identical to the original delta. With a git
like repository that could be handled, making a docker pull a much much
more efficient operation.

That is the desired...

However, the ask is simply to have a standard way to flatten an image. So
lets say as part of my build I downloaded the plex source installed the
developer dependencies, compiled it, and then deleted the everything except
the actual build. All the tools for the build would still be layers in
the image even though they would be inaccessible from my final image. It
is a huge waste... If the container could be flattened to remove unneeded
layers it would be much much more efficient use of space and bandwidth.

That is the required...

Don't say you can't do the required, because the desired is too much
work... Just hit the low hanging fruit first, and everyone will be much
happier.

On 14 July 2016 at 19:03, Roman V Shaposhnik notifications@github.com
wrote:

@justincormack https://github.com/justincormack Justin, what I really
would love to see is very similar to #22641
#22641 but with a full power of
git rebase (especially git rebase --interactive). Do you think it is
feasible?

β€”
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#332 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADBcWBAHV30oG5yogvUifDjqAFINyFsLks5qVsBMgaJpZM4AjRHk
.

@jdmarshall
jdmarshall commented Jul 19, 2016 edited

@vdemeester

| The Dockerfile is not the source of truth for your image

I think some of the people asking for features like this one are more comfortable with the truth in this statement than some of the people with 'Docker member' after their names. There are, for instance, a number of security related issues that have been closed-won't-fix with the reason that docker build should be repeatable.

@cpuguy83
Contributor
cpuguy83 commented Nov 2, 2016

For those interested, we just merged --squash on docker build.
This will squash the final result of the build to it's parent image (ie. the FROM).
#22641

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment