[1.13.0-rc1] Continuous "orphan" and "/proc/.../stat" logs. Maybe HEALTHCHECK related

[JamesJJ]

Description

journalctl -u docker.service shows continuous pairs of logs as shown below.

...logs appear to correspond to the only container that has HEALTHCHECK enabled:

[root@ip-10-10-47-139 centos]# docker ps
CONTAINER ID        IMAGE                                                                                                            COMMAND                  CREATED             STATUS                   PORTS                                                                 NAMES
db19a04a6703        xxxx:yyyy       "/bin/sh -c '/bin/..."   8 minutes ago       Up 8 minutes (healthy)   9080-9081/tcp, 9091-9095/tcp, 9097-9099/tcp, 0.0.0.0:9080->9096/tcp   webhelper_sgngx_1
...
...

Log frequency seems to match HEALTHCHECK interval:

[root@ip-10-10-47-139 ops-devops]# docker inspect webhelper_sgngx_1 | grep Healthcheck -A 6
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "curl --max-time 3 --connect-timeout 2 -sSf http://127.0.0.1:9080/health || exit 1"
                ],
                "Interval": 4000000000
            },

The logs:

Nov 12 14:32:31 ip-10-10-47-139 dockerd[5586]: time="2016-11-12T14:32:31.549149124Z" level=warning msg="containerd: unable to save db19a04a67036f5b3dcae2fc6aae6c03d564fc4cb0b1e41484bbdaeb8bc31014:2b8edc80fea35372bb233a591a8b831912058d559599fa3a38c9eb1909f81e4d starttime: open /proc/23611/stat: no such file or directory"
Nov 12 14:32:31 ip-10-10-47-139 dockerd[5586]: time="2016-11-12T14:32:31.549284839Z" level=info msg="containerd: db19a04a67036f5b3dcae2fc6aae6c03d564fc4cb0b1e41484bbdaeb8bc31014:2b8edc80fea35372bb233a591a8b831912058d559599fa3a38c9eb1909f81e4d (pid 23611) has become an orphan, killing it"

Nov 12 14:32:35 ip-10-10-47-139 dockerd[5586]: time="2016-11-12T14:32:35.587753348Z" level=warning msg="containerd: unable to save db19a04a67036f5b3dcae2fc6aae6c03d564fc4cb0b1e41484bbdaeb8bc31014:2b1c777221c13b7c9aaf37fa33408c115b0cabf533ffeb1557cf81a0684b852e starttime: open /proc/23635/stat: no such file or directory"
Nov 12 14:32:35 ip-10-10-47-139 dockerd[5586]: time="2016-11-12T14:32:35.587902057Z" level=info msg="containerd: db19a04a67036f5b3dcae2fc6aae6c03d564fc4cb0b1e41484bbdaeb8bc31014:2b1c777221c13b7c9aaf37fa33408c115b0cabf533ffeb1557cf81a0684b852e (pid 23635) has become an orphan, killing it"

Nov 12 14:32:39 ip-10-10-47-139 dockerd[5586]: time="2016-11-12T14:32:39.626162789Z" level=warning msg="containerd: unable to save db19a04a67036f5b3dcae2fc6aae6c03d564fc4cb0b1e41484bbdaeb8bc31014:517e2460c8ff9b2ea9814ffa09b25f6f72f75ccd49cd396cdd155dcebf58de68 starttime: open /proc/23656/stat: no such file or directory"
Nov 12 14:32:39 ip-10-10-47-139 dockerd[5586]: time="2016-11-12T14:32:39.62630815Z" level=info msg="containerd: db19a04a67036f5b3dcae2fc6aae6c03d564fc4cb0b1e41484bbdaeb8bc31014:517e2460c8ff9b2ea9814ffa09b25f6f72f75ccd49cd396cdd155dcebf58de68 (pid 23656) has become an orphan, killing it"

Describe the results you expected:

• No logs (unless e.g. the HEALTHCHECK result changes state)
• .... not running out of disk space due to continuous logs ... etc ;)
• I could decrease the log level, but these include WARNING level entries, which I'd rather not filter out totally

Output of docker version:

Client:
 Version:      1.13.0-rc1
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   75fd88b
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:             1.13.0-rc1
 API version:         1.25
 Minimum API version: 1.12
 Go version:          go1.7.3
 Git commit:          75fd88b
 Built:               
 OS/Arch:             linux/amd64
 Experimental:        false

Output of docker info:

Containers: 8
 Running: 8
 Paused: 0
 Stopped: 0
Images: 8
Server Version: 1.13.0-rc1
Storage Driver: devicemapper
 Pool Name: docker_lvm_vg-docker_lvm
 Pool Blocksize: 524.3 kB
 Base Device Size: 8.59 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 1.857 GB
 Data Space Total: 49.92 GB
 Data Space Available: 48.07 GB
 Metadata Space Used: 778.2 kB
 Metadata Space Total: 1.615 GB
 Metadata Space Available: 1.614 GB
 Thin Pool Minimum Free Space: 3.494 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8517738ba4b82aff5662c97ca4627e7e4d03b531
runc version: ac031b5bf1cc92239461125f4c1ffb760522bbf2
init version: N/A (expected: v0.13.0)
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.795 GiB
Name: ip-10-10-47-139
ID: BZMB:SWXA:DKST:B5XF:OJ4P:GIL5:XLPL:AGMO:WW5K:RPWU:G3JK:KNYF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

• Fully updated Centos 7
• AWS
• EBS Disks
• Docker from docker's 'testing' yum repo

[cpuguy83]

ping @mlaventure

[JamesJJ]

Ok to close, I think? ... appears resolved in 1.13.0-rc2.

[thaJeztah]

Thanks for testing rc2! Yes, it looks like https://github.com/docker/containerd/issues/351 was merged in docker in #28590