Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker: Error response from daemon: oci runtime error: container_linux.go:262: starting container process caused "open /proc/self/fd: no such file or directory". #297

Closed
suiwenfeng opened this issue Jul 5, 2017 · 32 comments
Labels

Comments

@suiwenfeng
Copy link

@suiwenfeng suiwenfeng commented Jul 5, 2017

Description

Steps to reproduce the issue:

  1. docker build
FROM nginx
MAINTAINER suiwenfeng <suiwenfeng@fedoraproject.org>

# add config to 
# copy ./default.conf /etc/nginx/conf.d

VOLUME [".","/usr/share/nginx/html"]

CMD ["nginx", "-g", "daemon off;"]
  1. docker rm -f {containerid}
  2. docker volume rm $(docker volume ls -qf dangling=true)

Describe the results you received:

docker: Error response from daemon: oci runtime error: container_linux.go:262: starting container process caused "open /proc/self/fd: no such file or directory".

throw the same error for each docker run , and docker run failed.

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Docker version 17.06.0-ce, build 02c1d87

Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 7
Server Version: 17.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.31-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: moby
ID: AX4D:5GVQ:SDCP:PFA2:BJFW:Y3RJ:YYCB:QWV7:UW4B:U5BF:5UJB:CZHF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 19
 Goroutines: 34
 System Time: 2017-07-05T08:02:41.906145547Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

@suiwenfeng

This comment has been minimized.

Copy link
Author

@suiwenfeng suiwenfeng commented Jul 5, 2017

i have tried reset docker deamon, it not work.

@suiwenfeng

This comment has been minimized.

Copy link
Author

@suiwenfeng suiwenfeng commented Jul 5, 2017

not working even after restart the computer.

@suiwenfeng

This comment has been minimized.

Copy link
Author

@suiwenfeng suiwenfeng commented Jul 5, 2017

looks some symlink removed from /dev, and the mobylinux crashed? so i have to reinstall docker?

@lpegoraro

This comment has been minimized.

Copy link

@lpegoraro lpegoraro commented Jul 15, 2017

same issue, running same version and on Fedora 25 here too.

@dnephin

This comment has been minimized.

Copy link
Contributor

@dnephin dnephin commented Jul 19, 2017

https://github.com/docker/for-linux/issues is probably a better place to report this issue. I don't believe this is related to the cli.

@supernelis

This comment has been minimized.

Copy link

@supernelis supernelis commented Oct 9, 2017

Isn't this because you try to define the volume twice? This volume is already defined in the nginx docker file.

If I mount the volume on run it works "docker run -d -p 80:80 -v $PWD:/usr/share/nginx/html nginx"

@bitliubei

This comment has been minimized.

Copy link

@bitliubei bitliubei commented Nov 29, 2017

same error, same arm64, same docker version

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Dec 26, 2017

VOLUME [".","/usr/share/nginx/html"]

@suiwenfeng What exactly are you trying to do with that line? Using the JSON notation for VOLUME in this case attempts to define two volumes; one at "." (current working-directory), and one at "/usr/share/nginx/html" (which is already defined as a volume in the base image)

The first (".") is what's causing the issue here, because the working-dir for the nginx image is /, which effectively would try to create a volume for the whole container (which isn't allowed / won't work).
You can find the current working-directory for the image using the following;

$ docker run --rm nginx pwd
/

Interesting bit here, is that there is some validation in the daemon, but it's missing validation for certain cases. For example, these Dockerfiles all build without producing an error during build;

FROM nginx
VOLUME ["/"]
FROM nginx
VOLUME /
FROM nginx
VOLUME ["."]
FROM nginx
VOLUME .

But starting a container from any of the above (as expected) won't work;

docker: Error response from daemon: OCI runtime create failed: container_linux.go:296: starting container process caused "process_linux.go:398: container init caused \"open /dev/ptmx: no such file or directory\"": unknown.

When defining a volume at runtime, the following correctly produces a validation error;

$ docker run -it --rm -v myvolume:/ nginx 
docker: Error response from daemon: invalid volume specification: 'myvolume:/': invalid mount config for type "volume": invalid specification: destination can't be '/'.
See 'docker run --help'.

But only specifying the "container path" does not perform the correct validation, thus results in the same "cryptic" error;

$ docker run -it --rm -v / nginx
docker: Error response from daemon: OCI runtime create failed: container_linux.go:296: starting container process caused "process_linux.go:398: container init caused \"open /dev/ptmx: no such file or directory\"": unknown.
@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Dec 26, 2017

I opened moby/moby#35879 for tracking this in the Moby repository (where this validation should be performed). Given that this is not a bug in the CLI, and the bug is missing validation for these cases (the Dockerfile is actually incorrect), I'm closing the issue here, but feel free to continue the conversation

@thaJeztah thaJeztah closed this Dec 26, 2017
@namedgraph

This comment has been minimized.

Copy link

@namedgraph namedgraph commented Jan 21, 2018

Same problem on docker-ce within jenkins container, running on Windows 10 host (inception):

docker run -d --name test.nginx -p 90:80 -p 554:443 --network graphityplatform_default --network-alias test.nginx -v /var/jenkins_home/workspace/docker-platform_stage.localhost_/docker/nginx-frontend/nginx.test.conf:/etc/nginx/nginx.conf nginx
docker: Error response from daemon: OCI runtime create failed: container_linux.go:296: starting container process caused "process_linux.go:398: container init caused \"rootfs_linux.go:58: mounting \\\"/var/jenkins_home/workspace/docker-platform_stage.localhost_/docker/nginx-frontend/nginx.test.conf\\\" to rootfs \\\"/var/lib/docker/overlay2/81cb11389790f97d5026133a05e4f555d499963dd614e082072276ca6a362bc6/merged\\\" at \\\"/var/lib/docker/overlay2/81cb11389790f97d5026133a05e4f555d499963dd614e082072276ca6a362bc6/merged/etc/nginx/nginx.conf\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
def81c6ffdf5f699517b2908d6a96ca40551646c4a965cef2be561c9cffd6433
@akivanctp

This comment has been minimized.

Copy link

@akivanctp akivanctp commented Feb 28, 2018

Any solutions on this error ?

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Feb 28, 2018

@pumba-lt your error looks different; from the error (copied the important parts below):

mounting "/var/jenkins_home/workspace/docker-platform_stage.localhost_/docker/nginx-frontend/nginx.test.conf" to rootfs 
at "/var/lib/docker/overlay2/81cb11389790f97d5026133a05e4f555d499963dd614e082072276ca6a362bc6/merged/etc/nginx/nginx.conf"
caused "not a directory """: unknown: 
Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.

That error indicates that likely the nginx.test.conf file did not exist on the daemon host. If you were attempting to mount a file using the -v <host-path>:<container-path> option, and <host-path> doesn't exist on the daemon host, docker will assume you want to bind-mount a directory, and create the directory if it doesn't exist. It then bind-mounts the directory in the container at <container-path>, which will fail if <container-path> is a file (because you cannot mount a directory on top of a file). That's the last part of the error message:

Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
  • check if the file /var/jenkins_home/workspace/docker-platform_stage.localhost_/docker/nginx-frontend/nginx.test.conf exists on the daemon host
  • if it didn't exist, docker now created a directory at the path above; remove that directory and replace it with the file you're trying to mount
  • you can also use the --mount flag instead of -v / --volume: the --mount flag will not automatically create a directory if the host-path is missing, but instead produce an error, so that you're aware the path (file/directory) on the host is missing.

Bind-mounting always happens from the host where the daemon runs, so even though the file may be present on your "client" machine; if that machine is a local machine, and not the machine where the daemon runs, bind-mounting won't work.

Also note that if you are running docker-in-docker, the "daemon host" may be the container in which the daemon is running.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Feb 28, 2018

@akivanctp the original error reported is an invalid volume path (/ cannot be used as a volume). For other situations, see my comment above: TL;DR make sure that, when using bind-mounts, the path that you're trying to mount exists.

@atiasadir

This comment has been minimized.

Copy link

@atiasadir atiasadir commented Nov 12, 2018

any solution for this issue ?
oci runtime error: container_linux.go:265: starting container process caused "exec:

@hemangjoshi37a

This comment has been minimized.

Copy link

@hemangjoshi37a hemangjoshi37a commented Dec 27, 2018

Any solution??

@jakubzloczewski

This comment has been minimized.

Copy link

@jakubzloczewski jakubzloczewski commented Dec 28, 2018

same with "Version 2.0.0.0-mac78 (28905)"
I've tried reset to factory settings but still got:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"-d\": executable file not found in $PATH": unknown.
@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Dec 28, 2018

@jakubzloczewski double-check the exact command you're running to start your container; from that output, it looks like you passed the -d option after the name of the image you're trying to run. Because of that, -d is used as the command to run inside the container, thus will fail;

docker run busybox -d
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"-d\": executable file not found in $PATH": unknown.

The -d (--detach) option is an option for docker run, so must be passed before the name of the image you're running;

docker run -d busybox
@Liuchengda

This comment has been minimized.

Copy link

@Liuchengda Liuchengda commented Jan 10, 2019

I have a similar issue, when I tried to install elasticsearch on my personal laptop, I execute

docker run --name=amp-es -d -v ~/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml -p 9200:9200 -p 9300:9300 --network=amp-network elasticsearch:2.4

then got the same error:

Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/Users/cliu426/elasticsearch.yml\\\" to rootfs \\\"/var/lib/docker/overlay2/53adcd81e2c00ea1f7b40ee67b5ae62ae6d80afbb939b57ef48adf37d003f7fd/merged\\\" at \\\"/var/lib/docker/overlay2/53adcd81e2c00ea1f7b40ee67b5ae62ae6d80afbb939b57ef48adf37d003f7fd/merged/usr/share/elasticsearch/config/elasticsearch.yml\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
Error: failed to start containers: amp-es
@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jan 10, 2019

@Liuchengda Not the same error (cleaned up the error for readability);

mounting "/Users/cliu426/elasticsearch.yml" to rootfs "/var/lib/docker/overlay2/53adcd81e2c00ea1f7b40ee67b5ae62ae6d80afbb939b57ef48adf37d003f7fd/merged"
at "/var/lib/docker/overlay2/53adcd81e2c00ea1f7b40ee67b5ae62ae6d80afbb939b57ef48adf37d003f7fd/merged/usr/share/elasticsearch/config/elasticsearch.yml"
caused "not a directory": unknown:

Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

That last line is the actual error; you're trying to mount a local file or directory from your host (/Users/cliu426/elasticsearch.yml) into the container (at /usr/share/elasticsearch/config/elasticsearch.yml). However, the path on your host cannot be found, or is a directory;

  • Check if the file /Users/cliu426/elasticsearch.yml exists
    • When using the -v option; docker creates a directory at the given location if the file/path does not exist
  • Note that bind-mounting is done from the host that the daemon runs on, and thus won't work if the daemon is running remotely ("remotely" could be a Virtual Machine)
@ldynia

This comment has been minimized.

Copy link

@ldynia ldynia commented Jan 18, 2019

For me the error occurred when I was trying to execute command to run my startup script command: bash startup.sh. Problem was that my image build was made on alpine linux which uses ash as shell instead of bash. Just double check that you are referring to correct shell interpreter.

@n0str

This comment has been minimized.

Copy link

@n0str n0str commented May 15, 2019

It is also may be caused by modified credentials (on Windows). When you have changed your system password, Docker cannot automatically determine it.
You should go to Settings\Shared Drives and Reset credentials.

@iwpnd

This comment has been minimized.

Copy link

@iwpnd iwpnd commented May 15, 2019

It is also may be caused by modified credentials (on Windows). When you have changed your system password, Docker cannot automatically determine it.
You should go to Settings\Shared Drives and Reset credentials.

This actually fixed it for me. Went nuts for a whole afternoon. Thank you!

@meirkr

This comment has been minimized.

Copy link

@meirkr meirkr commented May 21, 2019

It is also may be caused by modified credentials (on Windows). When you have changed your system password, Docker cannot automatically determine it.
You should go to Settings\Shared Drives and Reset credentials.

This actually fixed it for me. Went nuts for a whole afternoon. Thank you!

Same what happened for me: "This actually fixed it for me. Went nuts for a whole afternoon. Thank you!"

@ThaSami

This comment has been minimized.

Copy link

@ThaSami ThaSami commented Oct 27, 2019

same error here when running Hello-world docker version.. Docker on Fedora 31

[ThaSami@localhost ~]$ docker run hello-world
docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused "open /sys/fs/cgroup/docker/cpuset.cpus.effective: no such file or directory"": unknown.
ERRO[0009] error waiting for container: context canceled

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Oct 28, 2019

@ThaSami current version of Fedora 31 switched to using cgroupsV2 by default, which is not yet supported by the container runtimes (and kubernetes); work is in progress on this, but not yet complete, and not yet production ready. To disable v2 cgroups, run:

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

And restart your machine.

@tomwj

This comment has been minimized.

Copy link

@tomwj tomwj commented Nov 1, 2019

I had the same issue on a host running Ubuntu and needed to use:

sudo update-grub "systemd.unified_cgroup_hierarchy=0"

For anyone else that is on Ubuntu so doesn't have grubby

@florianajir

This comment has been minimized.

Copy link

@florianajir florianajir commented Nov 20, 2019

I have this error on windows and I never changed my credentials

@dkypuros

This comment has been minimized.

Copy link

@dkypuros dkypuros commented Nov 22, 2019

@ThaSami current version of Fedora 31 switched to using cgroupsV2 by default, which is not yet supported by the container runtimes (and kubernetes); work is in progress on this, but not yet complete, and not yet production ready. To disable v2 cgroups, run:

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

And restart your machine.

Thanks for your fix. I'm on Fedora 31, and docker works with latest docker documentation for fedora: https://docs.docker.com/install/linux/docker-ce/fedora/#install-docker

@mokaymakci

This comment has been minimized.

Copy link

@mokaymakci mokaymakci commented Nov 23, 2019

@ThaSami current version of Fedora 31 switched to using cgroupsV2 by default, which is not yet supported by the container runtimes (and kubernetes); work is in progress on this, but not yet complete, and not yet production ready. To disable v2 cgroups, run:

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

And restart your machine.

The solution worked like a charm for Fedora 31.

@Montana

This comment has been minimized.

Copy link

@Montana Montana commented Dec 16, 2019

Getting this on Mac OSX Catalina still.

@sergio00

This comment has been minimized.

Copy link

@sergio00 sergio00 commented Jan 23, 2020

Getting this on Mac OSX Catalina still.

Any advice or suggest?

@SalathielGenese

This comment has been minimized.

Copy link

@SalathielGenese SalathielGenese commented Feb 5, 2020

Fedora 31

$ sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
grub2-editenv: error: invalid environment block.
grub2-editenv: error: invalid environment block.
grub2-editenv: error: invalid environment block.
grub2-editenv: error: invalid environment block.
grub2-editenv: error: invalid environment block.
grub2-editenv: error: invalid environment block.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.