New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker build hangs/crashes when useradd with large UID #5419

Open
mcieslik-mctp opened this Issue Apr 26, 2014 · 26 comments

Comments

Projects
None yet
@mcieslik-mctp

mcieslik-mctp commented Apr 26, 2014

When I try to add a user during a "docker build ." the process hangs for approx 2-3 min and crashes with a

$ docker build .
Uploading context  5.12 kB
Uploading context 
Step 0 : FROM ubuntu:14.04
 ---> 99ec81b80c55
Step 1 : RUN useradd -u 99900000 -g users mcieslik
 ---> Running in 3ba3c92673fd
2014/04/26 14:58:55 write /var/lib/docker/devicemapper/mnt/.../rootfs/var/log/lastlog: no space left on device

Dockerfile

FROM ubuntu:14.04 
RUN useradd -u 99900000 -g users mcieslik

If I change the above to

FROM ubuntu:14.04 
RUN useradd -u 1001 -g users mcieslik

or run

useradd -u 99900000 -g users mcieslik

in a

docker run -i -t ubuntu:14.04 /bin/bash

everything works fine

@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Apr 26, 2014

We're probably doing the wrong thing with sparse files which /var/log/lastlog is, it looks like you are using the devicemapper storage backend

Can you update this the output of docker info

I'd also be interested if you can you explain your need for such a large uid?

@mcieslik-mctp

This comment has been minimized.

mcieslik-mctp commented Apr 26, 2014

Thanks for your prompt answer. These user ids are given to each user (university wide) by our IT overlords. Setting a proper UID is needed to write to NAS mounts.

Containers: 58
Images: 30
Storage Driver: devicemapper
 Pool Name: docker-254:3-12587682-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 14230.2 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 11.4 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.1                                                                                                                                                                                                                   
Kernel Version: 3.14.1-1-ARCH                                                                                                                                                                                                                  
WARNING: No swap limit support   
@pnasrat

This comment has been minimized.

Contributor

pnasrat commented Apr 26, 2014

As a work around can you try the -l or --no-log-init in the Dockerfile

RUN useradd -l -u 99900000 -g users mcieslik
@mcieslik-mctp

This comment has been minimized.

mcieslik-mctp commented Apr 26, 2014

Thanks! This worked.

@unclejack

This comment has been minimized.

Contributor

unclejack commented Apr 30, 2014

The underlying issue is that a large sparse file is created (approximately 32 GB), but it's not exactly a Docker bug. Docker could make an attempt to handle large sparse files better, but that's a subject for the #docker-dev mailing list.

I'll close this issue now. Please feel free to comment.

@unclejack unclejack closed this Apr 30, 2014

zzak added a commit to zzak/mruby_hello_world_cli that referenced this issue Jun 28, 2015

hone added a commit to hone/mruby_hello_world_cli that referenced this issue Jun 28, 2015

Merge pull request #2 from zzak/master
Use --no-log-init when creating a user to avoid moby/moby#5419
@rhvgoyal

This comment has been minimized.

Contributor

rhvgoyal commented Sep 29, 2015

So is this an issue with "docker commit" that it can't handle sparse files and blots the file to the full size during commit. If yes, then I guess this is something which should be fixed in docker.

I created a 1G sparse file in a container (with overlay backend) and then did docker commit and top most layer size was 1G. So docker did inflate the file to 1G and that seems very inefficient.

@rhvgoyal

This comment has been minimized.

Contributor

rhvgoyal commented Sep 29, 2015

@unclejack Has this issue been discussed since then any where else? I tested this with latest docker so it has not been fixed yet.

@trntv

This comment has been minimized.

trntv commented Oct 26, 2015

usermod causes the same problem and it doesn't have --no-log-init
Storage Driver: aufs

@runcom

This comment has been minimized.

Member

runcom commented Oct 28, 2015

It's happening the same on overlay.. I'm going to reopen this to track this bug and see if Docker can better handle this scenario

@runcom runcom reopened this Oct 28, 2015

@vbatts

This comment has been minimized.

Contributor

vbatts commented Oct 28, 2015

golang's archive/tar only supports extracting from archives with sparse files, not creating an archive with sparse files.
From my searches, only GNU tar boasts creation of archives with sparse files. Many implementations barely support extracting these archives created by GNU tar

@wontonst

This comment has been minimized.

wontonst commented Mar 1, 2016

This issue still occurs on docker 1.10.2

Step 7 : RUN echo "start" && echo $(useradd -m -u $uid -g $gid rdsdb) && echo "done"
 ---> Running in d1d07c9f2316
start

done

Hangs after printing done.

Docker info:

Containers: 6
 Running: 0
 Paused: 0
 Stopped: 6
Images: 12
Server Version: 1.10.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 86
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.13.0-79-generic
Operating System: Ubuntu precise (12.04.5 LTS)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.58 GiB
Name: xxx
ID: xxx
WARNING: No swap limit support
@sdwolfz

This comment has been minimized.

sdwolfz commented Mar 8, 2016

I can also confirm this is still occurring with 1.10.2

I am trying to create the following image:

FROM node:5.7.1

ARG HOST_USER_UID=1000
ARG HOST_USER_GID=1000

RUN DEBIAN_FRONTEND=noninteractive                            && \
                                                                 \
    echo 'Creating notroot user and group from host'          && \
    addgroup --gid $HOST_USER_GID notroot                     && \
    adduser --uid $HOST_USER_UID --gid $HOST_USER_GID notroot && \
                                                                 \
    echo 'Installing testing tools'                           && \
    npm install -g mocha phantomjs testem

USER notroot

WORKDIR /work

EXPOSE 9876

CMD testem --host 0.0.0.0 --port 9876

And I am building with the following command:

docker build -t testem --build-arg HOST_USER_UID=`id -u` --build-arg HOST_USER_GID=`id -g` .

The build freezes after displaying all output from RUN and writes to /var/lib/docker/aufs/diff/a66b81c2371267f6e749ecc667ed9c39dfabdb6f8d767c88d80fe07525ac6ade/var/log/lastlog until the disk is full.

docker info output:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 9
Server Version: 1.10.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 13
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.19.0-51-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.679 GiB
Name: cgi
ID: E4J7:MZX3:WSEP:CEVQ:HGMD:GNQT:IW5L:NJYX:GTUV:6HRV:LB3L:36EK
WARNING: No swap limit support
@sdwolfz

This comment has been minimized.

sdwolfz commented Mar 8, 2016

I also can confirm that @pnasrat's workaround fixes this problem. I am now using:

groupadd -g $HOST_USER_GID notroot                           && \
useradd -l -u $HOST_USER_UID -g $HOST_USER_GID notroot       && \

instead of:

addgroup --gid $HOST_USER_GID notroot                     && \
adduser --uid $HOST_USER_UID --gid $HOST_USER_GID notroot

And the build succeeded.

My $HOST_USER_UID and $HOST_USER_GID contain 10 digit IDs

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Nov 29, 2016

linking golang/go#13548 to this issue

@gbraad

This comment has been minimized.

gbraad commented Nov 29, 2016

Reproducible as an automated build on Docker Hub: https://hub.docker.com/r/gbraad/issue-dockerfile/builds/bsdmngknf8dhfreh8sgfmyu/ :-s

While adduser -l can work around the issue, when using something like Ansible inside the container to configure an environment or some other use of su - or sudo, will cause the same issue.

@justincormack

This comment has been minimized.

Contributor

justincormack commented Nov 29, 2016

Presumably disk quotas will fix the issue with the host running out of space and just leave a runtime error?

pkarashchenko pushed a commit to pkarashchenko/sonic-buildimage that referenced this issue Feb 1, 2017

Petro Karashchenko
[sonic-slave]: Fix issue "no space left on disk" while trying to add …
…user in docker

Note: related to moby/moby#5419

Signed-off-by: Petro Karashchenko <petro.karashchenko@caviumnetworks.com>
@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Sep 28, 2017

FWIW golang/go#13548 was closed through golang/go@1eacf78, and scheduled for releasing in Go 1.10

@AkihiroSuda

This comment has been minimized.

Member

AkihiroSuda commented Sep 28, 2017

@thaJeztah @stevvooe

TAR with sparse files are still unrecommended in OCI
opencontainers/image-spec#733

Should we implement support for sparse files?

maxaf added a commit to maxaf/demesne that referenced this issue Dec 5, 2017

maxaf added a commit to maxaf/demesne that referenced this issue Dec 5, 2017

@pspacek

This comment has been minimized.

pspacek commented Dec 19, 2017

Please note that lastlog is not the only thing which uses sparse files.

I just ran into this problem while trying to build container with LMDB instance inside. LMDB is extensively used by OpenLDAP and other projects and it creates sparse files by design.

@sergey-safarov

This comment has been minimized.

sergey-safarov commented Jan 4, 2018

reproduced on 17.12.0-ce

@thaJeztah thaJeztah added this to backlog in maintainers-session Jan 4, 2018

@thaJeztah thaJeztah moved this from backlog to Needs review in maintainers-session Jan 4, 2018

@tiborvass

This comment has been minimized.

Collaborator

tiborvass commented Jan 18, 2018

Let's get #35739 in first and see if we could use the new APIs in archive/tar, such as tar.DetectSparseHoles.

jonpugh added a commit to provision4/provision that referenced this issue Apr 23, 2018

Instead of using usermod, delete the users and create anew, so we can…
… use --no-log-init option. Without this option, users with very large UID numbers were experiencing docker build hangs, and filling up drives. See moby/moby#5419
@jtreminio

This comment has been minimized.

jtreminio commented Jul 3, 2018

Logs were eating up several tens of GB of space on my laptop.

Do the following to find the culprits and delete them, if you ran useradd without the -l flag:

find /var/lib/docker -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

@thaJeztah thaJeztah removed this from Needs review in maintainers-session Aug 2, 2018

eolivelli pushed a commit to eolivelli/bookkeeper that referenced this issue Sep 11, 2018

Enrico Olivelli
Use useradd -l option for docker scripts
This is a workaround for the Docker issue moby/moby#5419

If you run the scripts with an very large uid the script hangs and it fills up the disk which contains /var/lib/docker

eolivelli pushed a commit to apache/bookkeeper that referenced this issue Sep 11, 2018

Enrico Olivelli
Use useradd -l option for docker scripts
This is a workaround for the Docker issue moby/moby#5419

If you run the scripts with an very large uid the script hangs and it fills up the disk which contains /var/lib/docker

Author: Enrico Olivelli <eolivelli@apache.org>

Reviewers: Sijie Guo <sijie@apache.org>

This closes #1673 from eolivelli/fix/docker-large-uid

eolivelli pushed a commit to apache/bookkeeper that referenced this issue Sep 11, 2018

Enrico Olivelli
Use useradd -l option for docker scripts
This is a workaround for the Docker issue moby/moby#5419

If you run the scripts with an very large uid the script hangs and it fills up the disk which contains /var/lib/docker

Author: Enrico Olivelli <eolivelli@apache.org>

Reviewers: Sijie Guo <sijie@apache.org>

This closes #1673 from eolivelli/fix/docker-large-uid

(cherry picked from commit c420b39)
Signed-off-by: Enrico Olivelli <eolivelli@apache.org>

eolivelli pushed a commit to apache/bookkeeper that referenced this issue Sep 11, 2018

Enrico Olivelli
Use useradd -l option for docker scripts
This is a workaround for the Docker issue moby/moby#5419

If you run the scripts with an very large uid the script hangs and it fills up the disk which contains /var/lib/docker

Author: Enrico Olivelli <eolivelli@apache.org>

Reviewers: Sijie Guo <sijie@apache.org>

This closes #1673 from eolivelli/fix/docker-large-uid

(cherry picked from commit c420b39)
Signed-off-by: Enrico Olivelli <eolivelli@apache.org>
@aduzsardi

This comment has been minimized.

aduzsardi commented Nov 9, 2018

  • It doesn't just fill up the disk , it is also memory intensive when it happens and if your system has a swap partition it will start using it

  • by the way this is also reproduced using ENV instead of ARG with latest docker version.

  • usecases: we are using active directory accounts to login on linux workstations , and the ID's generated from samba/winbind/sssd are some very large integers , so running something like --build-arg=APP_USER=$UID will very quickly fill up the disk,ram,swap and also high cpu usage was reported

@sdwolfz

This comment has been minimized.

sdwolfz commented Nov 9, 2018

@aduzsardi AD was the source of my long uid/gid at that time as well, but it does not matter where they comes from (ENV or ARG), as long as you have large uid/gid your system will be destroyed.

So far useradd -l is our only salvation.

@aduzsardi

This comment has been minimized.

aduzsardi commented Nov 9, 2018

useradd -l is ok , but what about usermod , groupmod and probably others like somebody mentioned above ldap-utils , samba-utils ... ? :)

@sdwolfz

This comment has been minimized.

sdwolfz commented Nov 9, 2018

Unfortunately I can not help you with those as I have not used them inside docker yet. But if I find any issue/workaround I will post them here (until the problem is properly fixed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment