Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't stack more than 42 aufs layers #1171

Closed
Soulou opened this issue Jul 9, 2013 · 66 comments
Closed

Can't stack more than 42 aufs layers #1171

Soulou opened this issue Jul 9, 2013 · 66 comments

Comments

@Soulou
Copy link
Contributor

@Soulou Soulou commented Jul 9, 2013

Hello, thanks for this fabulous project

I'm encountering quite often an issue that I associate to broken images. In fact sometimes when I create a container, I'm getting the following error :

docker run -i -t #image# bash
2013/07/09 14:40:06 Error: Error starting container 042c2188714f: Unable to mount using aufs

However AUFS is working fine, using base image is just ok. The only way to use that image is to remove it and rebuild it again. What can be the causes of this image corruption ?

Thank you.

However, aufs is completely working

@vieux
Copy link
Contributor

@vieux vieux commented Jul 9, 2013

Could you start the deamon in debug mode (with -D) and paste here the debug ?

Thanks.

@Soulou
Copy link
Contributor Author

@Soulou Soulou commented Jul 9, 2013

As soon as I'm able to reproduce it I'll post (next broken image) Thank you !

@Soulou
Copy link
Contributor Author

@Soulou Soulou commented Jul 9, 2013

[debug] api.go:883 Calling POST /containers/create
2013/07/09 16:11:25 POST /v1.3/containers/create
[debug] api.go:883 Calling POST /containers/{name:.*}/start
2013/07/09 16:11:25 POST /v1.3/containers/64e6343f4d63/start
2013/07/09 16:11:25 Kernel does not support AUFS, trying to load the AUFS module with modprobe...
2013/07/09 16:11:25 ...module loaded.

Nothing special in this output, aufs module is always loaded, the error is not there event if docker tries to modprobe again the driver. As I said before with another image it's working.

I'm using Ubuntu 12.04 LTS with PPA docker repository.

@creack
Copy link
Contributor

@creack creack commented Jul 9, 2013

/cc @jpetazzo

@jpetazzo
Copy link
Contributor

@jpetazzo jpetazzo commented Jul 9, 2013

I don't see any useful clue here, unfortunately!
Would there be a way to tar the broken image (and its dependencies) so we
can have a look?

@Soulou
Copy link
Contributor Author

@Soulou Soulou commented Jul 11, 2013

Ok next case I'll export the container if that's possible ! (Maybe the same error will occure)

@Soulou
Copy link
Contributor Author

@Soulou Soulou commented Jul 22, 2013

Broken Image :

docker run -i -t 298bd447a161 bash
2013/07/22 14:48:37 Error: Error starting container 2bdc2487652c: Unable to mount using aufs

Working Image :

docker run -i -t base bash
root@29dd2712d871:/# 
docker inspect 298bd447a161
[{
    "id": "298bd447a161615bd6cadb0a7e1a6646ae982d5ff46f676b873676498675bcb9",
    "parent": "a72f54bf6432c5961cbc777cc4cdbea54d11d6fae7483cd5e0d999d9f4aaf2fa",
    "created": "2013-07-22T14:38:37.039340267Z",
    "container": "8ff61bd27b30f5da3806c8f635ba6dc6aef489c247eceee276c5e566cff07985",
    "container_config": {
        "Hostname": "8ff61bd27b30",
        "User": "",
        "Memory": 0,
        "MemorySwap": 0,
        "CpuShares": 0,
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "PortSpecs": null,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": [
            "/buildpacks/builder"
        ],
        "Dns": null,
        "Image": "app/testapp",
        "Volumes": null,
        "VolumesFrom": "",
        "Entrypoint": null
    },
    "docker_version": "0.5.0",
    "config": {
        "Hostname": "",
        "User": "",
        "Memory": 0,
        "MemorySwap": 0,
        "CpuShares": 0,
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "PortSpecs": null,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": null,
        "Dns": null,
        "Image": "",
        "Volumes": null,
        "VolumesFrom": "",
        "Entrypoint": null
    },
    "architecture": "x86_64",
    "Size": 94881024

What is the best way to export the image ?

Thank you !

@Soulou
Copy link
Contributor Author

@Soulou Soulou commented Jul 22, 2013

I'm tarring and uploading /var/lib/docker/graph/[IMAGE_ID] will you be able to do something with that ?

@Soulou
Copy link
Contributor Author

@Soulou Soulou commented Jul 22, 2013

In fact this is the at least the 20th layers. (Each parent has a parent on more than 20 levels) Is their a limitation with AUFS on the number of layers we can mount ?

(Because the command which fails is directly moun -t aufs ... )

@vieux
Copy link
Contributor

@vieux vieux commented Jul 22, 2013

@Soulou no, because the parents arent' there. could you docker push it ?

@shykes
Copy link
Contributor

@shykes shykes commented Jul 22, 2013

@Soulou yes there is a limit in the number of aufs layers, I think it's 42 (at least in the default build on ubuntu kernels).

@solomonstre
@getdocker

On Mon, Jul 22, 2013 at 8:32 AM, Victor Vieux notifications@github.com
wrote:

@Soulou no, because the parents arent' there. could you docker push it ?

Reply to this email directly or view it on GitHub:
#1171 (comment)

@apatil
Copy link
Contributor

@apatil apatil commented Jul 23, 2013

I'm getting this also, 38 RUN commands plus one EXPOSE above the base image, on Ubuntu 13.0.4. Could it have something to do with the brs=1|0 option of http://manpages.ubuntu.com/manpages/karmic/man5/aufs.5.html?

              If the number of your branches is large or their  path  is  long
              and  you  meet the limitation of mount(8) ro /etc/mtab, you need
              to enable CONFIG_SYSFS and set aufs module parameter brs=1.   If
              your  linux  version  is  linux-2.6.24  and earlier, you need to
              enable CONFIG_AUFS_SYSAUFS too.

If it isn't feasible to increase aufs's maximum depth, implementing #332 would help a lot, especially if we could put some kind of FLATTEN command in dockerfiles.

One stopgap solution would be to let us group several shell commands into a single RUN command, so that we can decrease the number of layers we create without sacrificing too much readability:

RUN
    git clone git://github.com/apatil/repo
    cd repo
    ./configure
    make install

@jpetazzo
Copy link
Contributor

@jpetazzo jpetazzo commented Jul 23, 2013

When you get the "Unable to mount using aufs" error, do you get something
more in the kernel logs or the output of the docker daemon itself?
(That would be very helpful to pinpoint the root cause of the problem.)

On Mon, Jul 22, 2013 at 10:50 PM, Anand Patil notifications@github.comwrote:

I'm getting this also, 38 RUN commands plus one EXPOSE above the base
image, on Ubuntu 13.0.4. Could it have something to do with the brs=1|0option of
http://manpages.ubuntu.com/manpages/karmic/man5/aufs.5.html?

          If the number of your branches is large or their  path  is  long
          and  you  meet the limitation of mount(8) ro /etc/mtab, you need
          to enable CONFIG_SYSFS and set aufs module parameter brs=1.   If
          your  linux  version  is  linux-2.6.24  and earlier, you need to
          enable CONFIG_AUFS_SYSAUFS too.

If it isn't feasible to increase aufs's maximum depth, implementing #332https://github.com/dotcloud/docker/issues/332would help a lot, especially if we could put some kind of FLATTEN command
in dockerfiles.


Reply to this email directly or view it on GitHubhttps://github.com//issues/1171#issuecomment-21394503
.

@apatil
Copy link
Contributor

@apatil apatil commented Jul 24, 2013

From the Docker daemon:

[debug] api.go:64 [error 500] Unable to mount using aufs
2013/07/23 19:07:45 http: multiple response.WriteHeader calls

This is with revision a93a87f. Haven't found anything in the kernel logs.

@apatil
Copy link
Contributor

@apatil apatil commented Jul 24, 2013

I can reproduce the problem by putting

FROM base
RUN "ls"
RUN "ls" # repeat 40-50 times

in a dockerfile, and doing docker build . inside its folder.

@keeb
Copy link
Contributor

@keeb keeb commented Jul 24, 2013

@apatil there's a limit to how many AUFS layers can be created. I believe it's 42.

@crosbymichael
Copy link
Contributor

@crosbymichael crosbymichael commented Jul 24, 2013

Should we have an error message that is returned before doing docker build or when the maximum is reached?

@pinhao
Copy link

@pinhao pinhao commented Jul 24, 2013

I'm having the same problem on Ubuntu 12.04 LTS with PPA docker repository.
I think some FLATTEN command like is described in #332 would be a good solution.

@metalivedev
Copy link
Contributor

@metalivedev metalivedev commented Jul 26, 2013

Reproduced. Gist with sample Dockerfile and build output:
https://gist.github.com/metalivedev/6085112

@vieux
Copy link
Contributor

@vieux vieux commented Jul 26, 2013

This is weird, once you reached the limit (use the Dockerfile fom @metalivedev)
You can't mount any more, doesn't look to be per image, but global.

ping @shykes @jpetazzo

@laurentpetit
Copy link

@laurentpetit laurentpetit commented Jul 31, 2013

Same problem here, also.

I understand why creating an image per step in the Dockerfile helps, noticeably while growing / debugging an image.
But maybe we should have the option to choose when this occurs, so that for the production of the final image, there are less intermediate ones produced.

Maybe either a flag passed to the command line to prevent the commit of intermediate images. Or directives inside the Dockerfile. (the former seems more interesting since you won't have to change the Dockerfile everytime you want to rewrite on it - and just before you want to create the final production image - )

@fsouza
Copy link
Contributor

@fsouza fsouza commented Jul 31, 2013

@laurentpetit I believe #332 is the answer for that.

@laurentpetit
Copy link

@laurentpetit laurentpetit commented Jul 31, 2013

@fsouza It's answer proposal which would solve the same problem, but not the same proposal :-). I like it also, but am not seeing it make progress :-/

@pdwinkel
Copy link

@pdwinkel pdwinkel commented Aug 17, 2013

I had the same problem

2013/08/17 20:59:59 Error: Error starting container af267a84f9c4: Unable to mount using aufs

Merging a few RUN commands, and the container starts normally.

@ykumar6
Copy link

@ykumar6 ykumar6 commented Aug 20, 2013

Hey Guys - We're experiencing the same problem with some images. We do not believe we have a layer issue (is there a way to check)?

In our kernel logs, we see this error
aufs opt_add:714:docker[15256]: lookup failed /var/lib/docker/graph/32737f8072d0 (-2)

@ykumar6
Copy link

@ykumar6 ykumar6 commented Aug 20, 2013

so docker history shows that we are hitting the limit :(
any good ideas on compressing the number of steps to a single layer?

@solomonstre
Copy link

@solomonstre solomonstre commented Aug 20, 2013

Yash, currently the only way to "squashing" the image is to create a
container from it, export that container into a raw tarball, and re-import
that as an image. Unfortunately that will cause all image metadata to be
lost, including its history but also ports, env, default command,
maintainer info etc. So it's really not great.

There are 2 things we can do to improve the situation:

  1. A short-term solution is to implement a "lossless export" function,
    which would allow exporting an image to a tarball with all its metadata
    preserved, so that it can be re-imported on the other side without loss.
    This would preserve everything except history, because an image config does
    not currently carry all of its history. We could try to plan this for 0.7
    which is scheduled for mid-September. That is, if our 0.7 release manager
    @vieux decides we have time to fit it in the release :)

  2. A 2nd step would be to add support for history as well. This is a little
    more work because we need to start storing an image's full history in each
    image, instead of spreading it out across all the aufs layers. This is
    planned for 0.8.

On Mon, Aug 19, 2013 at 9:00 PM, Yash Kumar notifications@github.comwrote:

so docker history shows that we are hitting the limit :(
any good ideas on compressing the number of steps to a single layer?

Reply to this email directly or view it on GitHubhttps://github.com//issues/1171#issuecomment-22921077
.

@ykumar6
Copy link

@ykumar6 ykumar6 commented Aug 20, 2013

Thanks solomon. We're also playing around with aubrsync to copy down branches. Do you not recommend this approach?

We're expecting our commit history to get 100-150 layers long.

@solomonstre
Copy link

@solomonstre solomonstre commented Aug 21, 2013

This is related to issue #332.

On Mon, Aug 19, 2013 at 9:24 PM, Yash Kumar notifications@github.comwrote:

Thanks solomon. We're also playing around with aubrsync to copy down
branches. Do you not recommend this approach?

We're expecting our commit history to get 100-150 layers long.

Reply to this email directly or view it on GitHubhttps://github.com//issues/1171#issuecomment-22921792
.

@lox
Copy link

@lox lox commented Sep 17, 2014

We are running Docker 1.2.0 under Ubuntu 14.04 and are still seeing our images fail when theny have more than 42 layers. Is there anything we need to do to enable the increased image count?

@jessfraz
Copy link
Contributor

@jessfraz jessfraz commented Sep 17, 2014

@lox can you post the output of docker info

@lox
Copy link

@lox lox commented Sep 17, 2014

[debug] server.go:1036 Calling GET /info
[info] GET /v1.14/info
[5a41a262] +job info()
[5a41a262] +job subscribers_count()
[5a41a262] -job subscribers_count() = OK (0)
[5a41a262] -job info() = OK (0)
Containers: 0
Images: 45
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 45
Execution Driver: native-0.2
Kernel Version: 3.13.0-35-generic
Operating System: Ubuntu 14.04.1 LTS
Debug mode (server): true
Debug mode (client): false
Fds: 9
Goroutines: 12
EventsListeners: 0
Init Path: /usr/bin/docker
Username: 99designs
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support

@lox
Copy link

@lox lox commented Sep 17, 2014

@xzyfer
Copy link

@xzyfer xzyfer commented Sep 17, 2014

#7863 looks to be related to @lox's issue

@lox
Copy link

@lox lox commented Sep 17, 2014

After some testing, this appears to specifically affect the kernel used in EC2, can't reproduce the issue elsewhere, but on Linux ip-172-31-32-231 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15 01:58:42 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux it happens consistently.

@lox
Copy link

@lox lox commented Sep 17, 2014

Possibly also related to? #4338

We'll do some testing on whether it's related to having docker's layers stored in /mnt.

@erikh
Copy link
Contributor

@erikh erikh commented Sep 17, 2014

I seem to remember this being hard coded at 42 due to some AUFS limitations but after searching the code a tad now I’m not as sure.

On Sep 16, 2014, at 8:16 PM, Lachlan Donald notifications@github.com wrote:

Possibly also related to? #4338

We'll do some testing on whether it's related to having docker's layers stored in /mnt.


Reply to this email directly or view it on GitHub.

@jessfraz
Copy link
Contributor

@jessfraz jessfraz commented Sep 17, 2014

Its 127 now
On Sep 16, 2014 8:35 PM, "Erik Hollensbe" notifications@github.com wrote:

I seem to remember this being hard coded at 42 due to some AUFS
limitations but after searching the code a tad now I’m not as sure.

On Sep 16, 2014, at 8:16 PM, Lachlan Donald notifications@github.com
wrote:

Possibly also related to? #4338

We'll do some testing on whether it's related to having docker's layers
stored in /mnt.


Reply to this email directly or view it on GitHub.


Reply to this email directly or view it on GitHub
#1171 (comment).

@lox
Copy link

@lox lox commented Sep 17, 2014

Strangely, the problem occurs with docker mounted on /mnt/docker but works fine with /mnt/anything_else_here/docker. Truly bizarre, any insights into what might be causing that?

tonistiigi added a commit to tonistiigi/docker that referenced this issue Nov 13, 2014
Fixes moby#1171
Fixes moby#6465

Data passed to mount(2) is clipped to PAGE_SIZE if its bigger. Previous 
implementation checked if error was returned and then started to append layers 
one by one. But if the PAGE_SIZE clipping appeared in between the paths, in the 
permission sections or in xino definition the call would not error and 
remaining layers would just be skipped(or some other unknown situation).

This also optimizes system calls as it tries to mount as much as possible with 
the first mount.


Signed-off-by: Tõnis Tiigi <tonistiigi@gmail.com> (github: tonistiigi)
yoheiueda added a commit to yoheiueda/docker that referenced this issue Nov 21, 2014
Fixes moby#1171
Fixes moby#6465

Data passed to mount(2) is clipped to PAGE_SIZE if its bigger. Previous
implementation checked if error was returned and then started to append layers
one by one. But if the PAGE_SIZE clipping appeared in between the paths, in the
permission sections or in xino definition the call would not error and
remaining layers would just be skipped(or some other unknown situation).

This also optimizes system calls as it tries to mount as much as possible with
the first mount.

Signed-off-by: Tõnis Tiigi <tonistiigi@gmail.com> (github: tonistiigi)
@agherzan
Copy link
Contributor

@agherzan agherzan commented Nov 10, 2016

I can reproduce this on 1.10.3 . Any reason why this is closed?

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Nov 10, 2016

@agherzan If you are reproducing a 42-layer issue, then most likely it's due to how your aufs kernel module is compiled.
We do have a 127 layer hard limit for all storage drivers, however.

What's your setup?

@agherzan
Copy link
Contributor

@agherzan agherzan commented Nov 10, 2016

Are you referring to the max branch config of aufs? @cpuguy83

EDIT: I run docker 1.10.3 on an armv7 board, aufs, linux 4.1.15, 42 layers

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Nov 10, 2016

@agherzan Yes.

@agherzan
Copy link
Contributor

@agherzan agherzan commented Nov 10, 2016

@cpuguy83 I have 127

CONFIG_AUFS_FS=y
CONFIG_AUFS_BRANCH_MAX_127=y

@agherzan
Copy link
Contributor

@agherzan agherzan commented Nov 11, 2016

Just for reference I will tell here what happened in my case for people to benefit from it.

I was using golang 1.6 which hardcodes the page size of arm64 to 64. This made docker generate aufs mount commands that would make options not fit in 4K. In my case the kernel is configured without 64K page size support so mount command would return invalid argument due to string truncation as it uses a page size as maximum option size.

There are some patches in master (of golang) which removes this static behaviour and detects the pagesize at runtime. (https://go-review.googlesource.com/#/c/25021/) Backporting these fixed my problem.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Nov 11, 2016

@agherzan Thanks for the report!
I assume you mean go 1.6 and not docker 1.6?

@agherzan
Copy link
Contributor

@agherzan agherzan commented Nov 11, 2016

Edited to clarify @cpuguy83 . Typo.

@cpuguy83
Copy link
Collaborator

@cpuguy83 cpuguy83 commented Nov 11, 2016

Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.