docker-containerd started occupying port 10010

[kosciak9]

Description

We are using docker with docker-compose for our services (GitLab, Wiki, etc).
Every service is using port from 10010 to 10050 (we jump by ten for every service) and it is sitting behind Nginx for reverse proxy. It worked nice, but yesterday I have added new service at port 10050 and after reboot I'm unable to start GitLab on 10010, as it keeps erroring on Error starting userland proxy: listen tcp 0.0.0.0:10010: bind: address already in use. Process using port 10010 is docker-containerd. I have moved GitLab to 10060 and it works currently. Why did docker-containerd started ocupying port 10010? Is it configurable?

Steps to reproduce the issue:
I'm unsure. I had five services (10010 and 10011, 10020, 10030, 10040, 10050). Rebooted machine.

Describe the results you received:
Port 10010 is used even though no container is using it.

Describe the results you expected:
Port 10010 is free and ready to bind.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:11:02 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:09:05 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 16
 Running: 10
 Paused: 0
 Stopped: 6
Images: 76
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-130-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.42GiB
Name: luna
ID: 6H36:M6K6:T2XY:PCPQ:23KJ:LCLS:TE3K:GLV3:6SLF:OWQ6:3IM4:S4AF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

• physical server
• Ubuntu 16.04.5 LTS

I will happily provide any details needed.

[thaJeztah]

Is there anything in the logs indicating a possible problem? What does docker ps -a show? (no container mapping to that port?)

[thaJeztah]

Also; I see you're running Docker 18.06; was this an upgrade of an existing installation, or a fresh install? (What version of docker were you running before upgrading?)

[thaJeztah]

Actually; looks indeed like it's listening on that port by default; let me check with the containerd team;

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1617/sshd       
tcp        0      0 188.166.23.33:10010     0.0.0.0:*               LISTEN      3757/docker-contain
tcp6       0      0 :::22                   :::*                    LISTEN      1617/sshd       

[thaJeztah]

Ok; did some research and port 10010 the cri stream server port (containerd 1.1.x and up provides cri support to run kubernetes).

The cri plugin can be disabled by setting disabled_plugins = ["cri"] in the containerd configuration file;

root = "/var/lib/docker/containerd/daemon"
state = "/var/run/docker/containerd/daemon"
oom_score = -500
disabled_plugins = ["cri"]

[grpc]
  address = "/var/run/docker/containerd/docker-containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = "/var/run/docker/containerd/docker-containerd-debug.sock"
  uid = 0
  gid = 0
  level = "info"

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.linux]
    shim = "docker-containerd-shim"
    runtime = "docker-runc"
    runtime_root = "/var/lib/docker/runc"
    no_shim = false
    shim_debug = false

Here comes the tricky bit; I see that the configuration file (/var/run/docker/containerd/containerd.toml) is automatically generated, and overwritten when docker is restarted. Investigating now if this is automatically done by containerd, or if there's code in dockerd that triggers containerd to generate the file

[thaJeztah]

found it; it's generated here;

moby/libcontainerd/remote_daemon.go

Lines 186 to 203 in 42bd8e1

	func (r *remote) getContainerdConfig() (string, error) {
	path := filepath.Join(r.stateDir, configFile)
	f, err := os.OpenFile(path, os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0600)
	if err != nil {
	return "", errors.Wrapf(err, "failed to open containerd config file at %s", path)
	}
	defer f.Close()
	enc := toml.NewEncoder(f)
	if err = enc.Encode(r.Config); err != nil {
	return "", errors.Wrapf(err, "failed to encode general config")
	}
	if err = enc.Encode(r.pluginConfs); err != nil {
	return "", errors.Wrapf(err, "failed to encode plugin configs")
	}
	return path, nil
	}

[kosciak9]

I'm actually fine with port 10010 being used - just there was no documentation I could find and I was unsure what's going wrong. I'm grateful for fast reactions and clarification.

[thaJeztah]

Thanks for reporting! I also wasn't aware of this. We're discussing options to make this configurable (in situations where kubernetes won't be used, it should be possible to disable the plugin).

I'll keep this issue open to track that effort

Also, will discuss with the release team to consider adding a mention in the changelog

[AkihiroSuda]

As commented in containerd/containerd#2483 , I think we should disable CRI plugin in the default config.

[jeffijoe]

I'm using Docker for Mac and am using port 10010 for RethinkDB, any way I can fix this without having to change that port?

[thaJeztah]

a configuration option is being worked on in #37519, until that lands in a release (it's being considered for inclusion in a 18.06.1 patch release, but no date is set yet), there's no workaround on Docker for Mac, other than temporarily using a different port

[cpuguy83]

For the patch release, it seems like the CRI listener should just be disabled.

[vincentjorgensen]

This just cropped up for us on Ubuntu 16.04 on a server I just rebuilt. I do not see it on any server provisioned with docker-ce prior to today. Port 10010 is already in use. Is there a way I can quickly add "disabled_plugins = ["cri"]" to the toml file? Manually editting /var/run/docker/containerd/containerd.toml doesn't stick.

[thaJeztah]

Unfortunately, editing the file won't work (see above), as it's dynamically generated if containerd is started as child-process of dockerd; only option currently is to start containerd as a separate service and set the --containerd option to provide the path to the containerd socket

[vincentjorgensen]

For now, I'm downgrading to to 18.03 which doesn't have cri installed. That seems to address the issue. I look forward to being able to use 18.06 again as soon as the patch is in place. Thanks!

[thaJeztah]

Backport is queued if a patch release will be done; docker-archive#29

[saikiran939]

@vincentjorgensen is it possible to downgrade docker without loosing existing containers and their data ? if yes could you please share the steps ?

@thaJeztah could you please share steps for starting containerd as a service and steps to provide path to socket ?

I have existing containers that are not getting started because of this issue. any help is much appreciated.

[vincentjorgensen]

@saikiran939 Running containers? No, they need to be turned off. I install via ansible. Here is the relevant snippet:

- name: docker-ce ubuntu apt install
  apt:
    name: docker-ce=18.03.1~ce-0~ubuntu
    state: present

[tomwidmer]

This seems to be fixed in 18.06.1 by the way, so you can upgrade rather than downgrading...

[thaJeztah]

Correct; the fix for this shipped with Docker 18.06.1 and up

[ju2wheels]

Does anyone know if there are any CVE's related to CRI 10010 port being left open or what capabilities it would give someone on the same network with an out of the box 18.06.0 install? I understand that its better this not be opened at all but theres no explanation at all as to what capabilities this opens up to someone on the same network by default (if at all) so we can characterize the urgency with which one may want to upgrade.

[thaJeztah]

18.06.0 has reached EOL, and the runc version shipping with that version has critical vulnerabilities (some are patched in 18.06.2 and 18.06.3: https://docs.docker.com/engine/release-notes/#18063-ce) so separate from this issue, I highly recommend upgrading to a currently maintained version.