No such file or directory for /var/lib/docker/overlay2

[cptaffe]

Expected behavior

Docker image builds without error

Actual behavior

Docker image encounters error no such file or directory when building images. Restarting Docker does not resolve.

$ docker-compose up
Building map
Step 1/2 : FROM golang:onbuild
onbuild: Pulling from library/golang
693502eb7dfb: Already exists
081cd4bfd521: Extracting [==================================================>] 18.54 MB/18.54 MB
5d2dc01312f3: Download complete
fb1a82f32527: Download complete
a977e8d015dd: Download complete
08c0c39ec85a: Download complete
489f1299f3aa: Download complete
b87be4361d1b: Download complete
ERROR: Service 'map' failed to build: failed to register layer: error creating overlay mount to /var/lib/docker/overlay2/e0ae7864b7c2ba8e924255bd704cd9083bad4471a53bd6dff19a77696718c76a/merged: no such file or directory

$ docker build . -t postgres-thrillshare
Sending build context to Docker daemon  21.9 GB
Step 1/5 : FROM postgres:9.6
 ---> 4e18b2c30f8d
Step 2/5 : MAINTAINER Connor Taffe <connor.taffe@apptegy.com>
 ---> Using cache
 ---> 056821cd8af6
Step 3/5 : RUN openssl req -nodes -new -x509 -newkey rsa:4096 -keyout $PGDATA/server.key       -subj "/C=US/ST=Arkansas/L=Little Rock/O=Apptegy/OU=Development/CN=admin.apptegy.dev"       -out $PGDATA/server.crt     && chmod og-rwx $PGDATA/server.key
 ---> Using cache
 ---> a4f25738dc10
Step 4/5 : ENV POSTGRES_USER foo POSTGRES_DB bar POSTGRES_PASSWORD baz
error creating overlay mount to /var/lib/docker/overlay2/4a815bef0022da237ceafb1297b4958d3edbea74de1007ea2b1b864076b6cd5e-init/merged: no such file or directory

Information

ID: 16AAC546-A181-43B5-AD4E-F69C257AAFE7

Docker for Mac: version: 17.03.0-ce-mac2 (32de84251)
macOS: version 10.12.3 (build: 16D32)
logs: /tmp/16AAC546-A181-43B5-AD4E-F69C257AAFE7/20170306-105112.tar.gz
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

Contents of docker-compose.yml:

version: '2'
services:
  db:
    build: ./db
    networks:
      - back
    volumes:
      - db-data:/var/lib/postgresql/data
  thrillshare:
    build: ./thrillshare
    ports:
      - "443:443"
    networks:
      - back
  map:
    build: ./map
    networks:
      - back
  redis:
    image: redis
    networks:
      - back
networks:
  back:
    driver: bridge
volumes:
  db-data: {}

Contents of ./map/Dockerfile:

FROM golang:onbuild
MAINTAINER Connor Taffe <connor.taffe@apptegy.com>

Contents of Dockerfile for postgres-thrillshare:

FROM postgres:9.6
MAINTAINER Connor Taffe <connor.taffe@apptegy.com>

# Generate ssl certs
RUN openssl req -nodes -new -x509 -newkey rsa:4096 -keyout $PGDATA/server.key \
      -subj "/C=US/ST=Arkansas/L=Little Rock/O=Apptegy/OU=Development/CN=admin.apptegy.dev" \
      -out $PGDATA/server.crt \
    && chmod og-rwx $PGDATA/server.key

ENV POSTGRES_USER=foo \
  POSTGRES_DB=bar \
  POSTGRES_PASSWORD=baz

COPY init.sql /docker-entrypoint-initdb.d/

Steps to reproduce the behavior

1. Try to run docker build or docker-compose up.

May have something to do with running docker system prune.

[cptaffe]

Steps to resolve

Warning: this will delete any volumes on your machine, see this comment.
You can also reset Docker from preferences, see this comment.

1. Screen into the Docker VM
2. Remove /var/lib/docker
3. Restart Docker via the widget

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
$ umount /var/lib/docker/overlay2
$ rm -rf /var/lib/docker

[MagnusS]

Thanks for the report @cptaffe. I've looked at the logs and unfortunately I can't really see anything that should have caused this. It's good to hear that the issue is resolved. I'm closing the issue, but please reopen if you experience it again. If you find a way to reliably reproduce it that would also be very helpful for debugging.

[chalmagean]

I get the following error when trying to remove /var/lib/docker

/var/lib/docker # rm -rf /var/lib/docker
rm: can't remove '/var/lib/docker/overlay2': Resource busy
rm: can't remove '/var/lib/docker': Directory not empty

[cptaffe]

I've updated my comment above. I believe you have just need to unmount the fs

$ umount /var/lib/docker/overlay2

[ayhoung]

was wondering what exactly is causing this problem? ran into this issue as well

[jbarefoot]

I just hit the same error, after running the same docker compose build command several times in a row.
Docker version: 17.03.1-ce-mac5 (16048)
Mac version: 10.10.5 (14F2109)

[MrTB]

In my case I must restart the docker (MacOS) after did the "Steps to resolve". Maybe the information helps other.

[auvalade]

Just had the issue, in my case I resolved it by rebuilding the containers with the no-cache option.
I also restarted Docker for Mac before that but I don't know if it had any consequence in the overall solution.

docker-compose build --no-cache
docker-compose up

To be honest I don't really know what caused the issue either.

[bhardy]

I just ran into this issue and our handy DevOps engineer recommended that I follow the following steps:

1. click the docker icon in my menubar
2. click preferences
3. click the reset tab
4. click remove all data
5. run docker-compose up -d

This fixed my issue. It may be overkill, but it's easy. Use at your own risk.

Note: it'll take a long time if you have a lot of images.

[sMorm]

Ran into this issue as well, seems like it was caused by docker system prune. removing /var/lib/docker in the vm fixed it, thanks @cptaffe

[tagiris123]

How can i solve this one madame/sir? i tried to pull my gitlab-ce, but always failed . i tried it 8-10 times.

[cinqs]

In my situation, I changed my data location of my docker engine to some folder /some/folder, and today I just changed it back to /var/lib/docker. then did a docker system prune to remove all the volumes, overlays, images etc. after a reload of docker-daemon, and a restart of docker-engine. this problem happened.

@tagiris123 , I suggest you to stop the docker engine first, by systemctl stop docker for example, and manually remove the folder using rm -rf /var/lib/docker. then you should start the docker again with systemctl start docker. hope this helps. By the way, nice pic.

[tagiris123]

Thank you for answering my question, @cinqs
But instead of using docker pull gitlab/gitlab-ce, i use docker pull sameersbn/gitlab:latest . And it will run now. Can i ask @cinqs if this container ID is usable in apache2 also ?

[karuppiah7890]

@cptaffe Please include @MrTB 's comment too in your answer. I think it works only after this ? But I am on Linux though and this repo is for Mac's Docker, but I faced the same issue

In my case I must restart the docker (MacOS) after did the "Steps to resolve". Maybe the information helps other.

[bencoder]

I am getting this regularly, every couple of weeks - is there any investigation into it?

OSX: 10.12.6 (16G29)
Docker version 17.06.1-ce, build 874a737 (recently upgraded)

[mkrogh]

Had this problem toady as well after running docker system prune. Followed @cptaffe 's steps to resolve it. But it would be nice if this was resolved instead.

[artyfarty]

Well, advice to drop all data volumes is not a viable solution at all.

[ryanwalls]

@MagnusS I vote this should be reopened. I hit this today on latest Docker for Mac beta.

[hack2learn]

Only way to fix it for me is to reset Docker on my Mac via Widget.

[omendezmorales]

Hi,
I have a similar problem, and hope someone can help me. I've tried to reproduce the following post on cross/compiling fort the Onion2+:
https://rawsite.onioniot.com/2bt-cross-compiling-c-programs-part-1/

The problem is that after downloading the image, I get the following error message:
failed to register layer: open /var/lib/docker/aufs/layers/cfad77b8e9fcb568a0d806fce422590eded5babb91d5d5de8e5c6bfa310bf78e: no such file or directory

Could anybody point me to the possible solution? Thanks

I'm using Ubuntu 16.04.3 LTS, and docker 17.05.0-ce

Thanks!

[tsturzl]

I felt I was being pretty charitable in my interpretation of what you were saying. My intention certainly was not to be rude or condescending. A docker system prune -all will not affect your volumes, including named volumes; if I understand correctly this should be different from wiping docker from the Docker Desktop For Mac application menu. Deleting /var/lib/docker would however, you're correct, but this is probably unnecessary, as the issue is likely caused by an issue with a corrupt layer. I don't believe the fix for this requires you to lose your volumes in that case. I'm sorry you're experience has been a poor one. I am not a developer or contributor for this project and am simply weighing in on this as I'm currently working on something affected by this.

[alkanna]

This is still a problem for me with docker for mac years after...

[githubsands]

Still having issues with this on darwin

[wadewilliams]

My fix for this is to get rid of macos and go full linux. Everything works fine there. Deprecate the repo.

[shengwu1026]

Still having this problem and all solutions don't work for me.

[sgeffken]

We faced the same issue twice in a couple of days. Once on Core-OS, once on a yocto embedded linux.
In my case deleting /var/lib/docker helped in the embedded linux. Afterwards reloading images worked fine. On Core-OS reloading images broke docker again.
Any news on this? Even though this issues is under docker-for-mac, an open issue (just technically closed) from 2017 seems kind of bad imho.

Current suspicions point on USB-devices mounted into running docker containers. But we do not have any proofs for that, yet.

[soriyath]

I stiil have this issue.

[stevieray8450]

I'm having this issue too. When trying to screen in to the VM, I see failures due to com.docker.driver.amd64-linux not even existing...

[abserari]

the exact error that i am getting is
"error creating overlay mount to /var/lib/docker/overlay2/84368e229076b0595da9196982d56b77aa95faf1d1f47f3accf94641e2ee45ee-init/merged: no such file or directory"

This can also happen if the docker space is corrupted .
so try removing all stopped containers and build cache with command docker system prune --all

this is good for me.
I just run docker system prune --all and choose yes.
Everything gonna is OK!

[gbooth27]

just ran into this issue as well

[Jiia]

Had this issue as well 2.1.0.5 (40693)

[wadewilliams]

In my experience over the past 2 years of hitting this about once every week or two, I can only shed this much light:

It seems most likely to occur when either:

1. Importing a large database (10+ gigabytes) that is backed up with a named volume
2. The codebase has a ton of dependencies that are resolved by a job run inside the container and you've mounted your local code into the container

In both cases the docker engine occasionally becomes unresponsive, if you restart docker-for-mac and re-run the container where the error occured, then you'll hit the

ERROR: readlink /var/lib/docker/overlay2: invalid argument

It seems that deleting the image for this container is the least intrusive way to get things working again; this isn't helpful if you're trying to import 15gb worth of data and docker crashes on the 14th gb, you'll need to start over from scratch. But if you're half way through an npm/pip/composer install, it will pick back up where you left off.

Diagnostic report 054F9A95-A26C-49C1-83F9-E0C231875799/20200131212107

Looking forward to the time that I stop hitting this so frequently

[artyfarty]

I got tired of this and used docker-machine with vmware driver. It's a bit tricky to set up to work as "natural" as docker for mac (used wrapper for docker compose, some envvars-to-compose-file magic and shared folders), but IO is so much faster that it's worth it.
It you folks have some spare time and a licence (or demo), try it.

And there is also an NFS option, which also may work better than volume mounts for many cases. This is what we've used previously, also with a wrapper script etc.

[coryrc]

I am able to repro this problem on Linux across multiple computers with a specific image and was able to solve by using aufs storage-backend instead. Docker 19.03.5

[FantomX1]

For me, none of the mentioned solutions seemed to work (didn't try yet aufs approach yet though) in Docker Toolbox even after reinstalling it and removing all the data. However, the working workaround surprisingly was to restrict the number of 'RUN' docker building commands/layers, since if the number surpassed 60 layers/commands, it always ended up with that error, no matter what was the contents of the command, even simple command such as 'RUN ls -la' ended up with that error, if the total number of such/any commands was higher than about 60, strange. 'Merged' subfolder was always missing, though even when I automatically generated all the merged subfolders, always was created on the fly a new layer with a new hash, which was missing that subfolder.

[Sherlock1226]

hi man,
I run docker in my win10,and run dockerfile in WSL,happened an error ,"/var/lib/docker/overlay2" ," caused \"not a directory\"

[maresb]

@FantomX1 I think you might be onto something. Are we hitting a limitation of overlay with the number of layers? Here is the latest which has been driving me crazy, and look at my layer count!

Step 59/60 : ENV CONDA_PREFIX $CONDA_DIR
error creating overlay mount to /media/mares/Data/docker/overlay2/352ede2b08c2c26eac1803e225bd1b8bba07dd7cbbbc1c5756aba34d1d595e32-init/merged: no such file or directory

[maresb]

@FantomX1, if you happen to have defined a custom data root, then I recommend trying to add or remove a character from the directory name to see if the problem goes away.

[amitCandidly]

i followed every possible solution in this thread but still facing the issue

COPY failed: stat /var/lib/docker/overlay2/4b5778ef1684bb405c528a650a84d1c763984ecefabbcfac27ea2032313fff26/merged/app/dist: no such file or directory

[FantomX1]

@maresb thx will try to give it a check, but I don't seem to have defined it the custom root,
@amitCandidly maybe it doesn't work for you, although when you notice, the error message is considerably different from yours, you have a completely different folder missing "dist", not the here mentioned one "merged" as is in the question description, and it may make a complete difference in my opinion.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked