docker can't start container

[cc272309126]

Hi,
I have found that docker start a container will appear "Cannot start container 4a95e162ebab: open /sys/class/net/veth1afc6a6/brport/hairpin_mode: read-only file system" and I test with the master branch, I think it may related to libcontainer.

docker info
Containers: 116
Images: 436
Storage Driver: devicemapper
Pool Name: docker-253:0-2229360-pool
Pool Blocksize: 65.54 kB
Data file: /var/lib/docker/devicemapper/devicemapper/data
Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 28.61 GB
Data Space Total: 107.4 GB
Metadata Space Used: 39.26 MB
Metadata Space Total: 2.147 GB
Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 2.6.32-431.el6.x86_64
Operating System:
CPUs: 16
Total Memory: 31.34 GiB
Debug mode (server): true
Debug mode (client): false
Fds: 15
Goroutines: 14
EventsListeners: 0
Init Path: /usr/bin/docker

[cc272309126]

I get it‘s answer。https://github.com/docker/libcontainer/blob/master/netlink/netlink_linux.go#L1203
this function will set hairpin_mode for docker container。but on the rhel6.5 "/sys/class/net/vethXX/brport/hairpin_mode" is a readonly file. so the bug happend. I test it on rhel7 and rhel6.4 /sys/class/net/vethXX/brport/hairpin_mode all is a write-able file. it is relate kernel?

[crosbymichael]

Thanks for the report. I'll look into this.

ping @phemmer did you have any idea why this would be readonly on some systems?

[phemmer]

The message read-only file system implies that all of /sys is read-only for some reason. If it were just this one file the error would be "Permission denied" instead. This param probably just happens to be the first /sys param that docker touches when starting an instance.

I checked on one of my RHEL 6.5 system and the file is read-write as expected.
I've also gone though the kernel source for 2.6.32 (the version in RHEL6.5), and there is nothing in the sysfs code for this parameter that would cause it to be read only.
(edit: right, kernel 2.6.32 is probably not being used in this case since I don't think docker works with anything less than 3.8. I checked 3.13 and it will never make the param read only either)

If I recall correctly, there was a bug in an older version of docker that would cause /sys to become remounted read-only. Was docker just upgraded on this host? Does it clear up after a reboot?

[phemmer]

It looks like the bug with /sys becoming mounted read-only was fixed somewhere between 1.1.0 and 1.2.0. I can reproduce it on my system with 1.1.0 (1.0.0 as well), but nothing later. It happens when you run a container with --net=host.

[cc272309126]

@phemmer , you mean if I had used the docker 1.1.0 to start the docker container. the /sys would be read-only. and if I upgrade my doceker to 1.2.0 or higher, the /sys would still be read-only. is that right? if I reboot or remount it, it will be fix it?

[crosbymichael]

@cc272309126 what docker version are you using?

[phemmer]

@cc272309126 Yes. If you were previously running a version older than 1.2.0, your /sys could have become mounted read-only then. Even if you upgrade, it'll still remain mounted read-only until you either mount -o remount,rw /sys, or reboot.

[cc272309126]

@crosbymichael I use the 1.3.1-dev.

[crosbymichael]

Ok thanks. @phemmer it is related to the code changes and not anything with a past version. We do not fully mount /sys as readonly and at the time that we setup the network interfaces /sys is not RO at all

[phemmer]

@crosbymichael I don't follow.
If you're talking about /sys inside the container, the code uses /sys on the host, not container.
If you're talking about /sys on the host, docker doesn't mount/remount /sys on the host, so it should be read/write.

[cc272309126]

@phemmer if I want to upgrade the docker from 1.2.0 to 1.3.0, how can I avoid this issue without reboot? I found that remount just work for start one container, if I start another container it will ask me remount again.

[phemmer]

You're saying that with >= 1.3.0 it's remounting /sys as read-only? Does it do it every time? Are you using --net=host?

Maybe that's what @crosbymichael was talking about in his last comment, some unrelated code in 1.3.0 which is remounting /sys as read only (because the hairpin code certainly won't result in this).

[crosbymichael]

No, we do not mount /sys RO at the time that we write to this file. There is another, non docker issue that is causing this.

[cc272309126]

@phemmer @crosbymichael, I have used --net=host with docker which version is under 1.2.0. and now I upgrade the docker to 1.3.1-dev.and this happend. and if I just upgrade to 1.3.0, this issue will not happend. I am sorry about that I don't describe it clearly.

[cc272309126]

I found that on the docker 1.3.0, the code doesn't implemente the function func SetHairpinMode(iface *net.Interface, enabled bool), but on the master it add the function.

[crosbymichael]

Humm, maybe it has to do with your 2.6 kernel. I guess you are running RHEL or centos ? I'll try to reproduce on that system.

[cc272309126]

yes, I run on rhel 6.5. the kernel is 2.6.32-431.

[thenetimp]

i am having the same issue CentOS 6.5 Kernel 2.6.32-504.1.3.el6.x86_64

[crosbymichael]

ok

[crosbymichael]

I'll work on a fix for this today

[Nottt]

I'm still having issues with this. I have a software that need access to the network devices for network monitoring purposes.

With -v /sys/class/net:/sys/class/net \ the folder exposes docker0 which is still read-only...

With -v /sys/class/net/lo:/sys/class/net/lo \ this happens:

Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/sys/class/net/enp2s0\\\" to rootfs \\\"/var/lib/docker/overlay2/987ef1d2f79c42f8874ccb2410e506da42e824eeca8aae59b3e98e1f0cd02b83/merged\\\" at \\\"/var/lib/docker/overlay2/987ef1d2f79c42f8874ccb2410e506da42e824eeca8aae59b3e98e1f0cd02b83/merged/sys/class/enp2s0\\\" caused \\\"mkdir /var/lib/docker/overlay2/987ef1d2f79c42f8874ccb2410e506da42e824eeca8aae59b3e98e1f0cd02b83/merged/sys/class/enp2s0: read-only file system\\\"\"": unknown

Don't ask my why but the application tries to run chown:chown on that folder

I followed the symlink in my host but I can't start docker with - v /sys/devices/pci0000:00/0000:00:1c.1/0000:02:00.0/net/enp2s0 either

Error response from daemon: invalid volume specification: '/sys/devices/pci0000:00/0000:00:1c.1/0000:02:00.0/net/enp2s0:/sys/class/enp2s0

Sorry for reviving this, but google isn't helping much 😕