cannot mount /dev read-only, operation not permitted

[hazcod]

Description

docker-compose up with user namespace remapping results in the error: ERROR: for db Cannot start service db: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused "rootfs_linux.go:125: remounting \"/dev\" as readonly caused \"operation not permitted\"".

Steps to reproduce the issue:

1. Use daemon.json;

{
        "storage-driver": "overlay2",
        "dns": ["8.8.8.8","8.8.4.4"],
        "live-restore": true,
        "selinux-enabled": true,
        "userns-remap": "container:container"
}

2. Use /etc/subuid, /etc/subgid

...
container:1007:65536
...

...
container:1008:65536
...

3. container user:

sudo -u container id
uid=1007(container) gid=1008(container) groups=1008(container),998(docker)

4. Composefile:

version: '3'

networks:
  frontend:
    external: false

  backend:
    external: false

volumes:
    db-data:

services:
    db:
        image: "db:${stage}"
        volumes:
            - db-data:/app/data
        expose:
            - 5432
        tty: false
        restart: "on-failure:5"
        read_only: true
        tmpfs:
            - /tmp/
        security_opt:   
            - "no-new-privileges"
        networks:
            - backend
        deploy:
            resources:
                limits:
                    memory: 2GB
                    #memory-swap: 0
        env_file:
            - secrets/xxx.secret

5. system:

 Linux xxx 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux

Describe the results you received:
The error.

Describe the results you expected:
Successfull container start.

Additional information you deem important (e.g. issue happens only occasionally):
sudo docker run -ti --rm --read-only=true db:xxx
docker: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused "rootfs_linux.go:125: remounting \"/dev\" as readonly caused \"operation not permitted\""".

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:48 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 3
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
 userns
Kernel Version: 4.9.0-3-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859GiB
Name: xxx
ID: YYLA:WT7V:T7XF:ZVTE:C246:JGJD:VXCC:CAAJ:M7U3:7HEK:J7SZ:FXGJ
Docker Root Dir: /var/lib/docker/1007.1008
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on debian VPS.

[hazcod]

Is it possible that --read-only is not possible with user namespace mapping?

[thaJeztah]

ping @estesp is this a known limitation?

[estesp]

It could be quite dependent on distro kernel version/patches. On my Ubuntu 16.04.3 LTS system I can run alpine:latest as --read-only on v17.09.0-ce with userns-remap enabled.

I wonder if in the debian VPS instance there is some other mount options/protections on /dev/ that is causing the problem? Would probably need an strace from the runc invocation to really understand what the kernel is rejected about the remount as ro.

[hazcod]

@estesp any idea how I might debug this further? I can't just strace docker daemon I suppose?

[estesp]

@hazcod not easily.. but I did create this a few years ago: https://github.com/estesp/utils/tree/master/strace-runc

I'm assuming it should still work properly with current versions of the engine, you can follow those instructions. Note that this script messes horribly with containers working properly in general, but great for capturing strace on start. :) Once you have an strace of the failure, you will want to reset your installation otherwise you will have all kinds of weird behavior :)

[csnover]

I am able to reproduce this issue in a Debian 8 VM. Here is an attached strace using strace-runc. I had to change the strace-runc slightly since the failure occurs on a "create" command, not a "start" command.

strace-af0728429b3768607cb4846303a7ca5305e5ac66d7e64ec5b7b4cfcd27859f2e.log

[GatoNegr0]

Probably try "--tmpfs /dev" within the docker run command