Docker containers on server3 do not get started after a server restart

[mitar]

No description provided.

[mitar]

It seems the issue is that we have restart policy set to unless-stopped, which is not supported on Docker < 1.9. We have currently Docker 1.8.3 on server3.

[mitar]

Fixed with tozd/salt@4594eca.

I redeployed containers on server3 with the new restart policy.

[clonm]

I still had to restart it manually

[clonm]

After power outage, docker ps hangs. Output of cat syslog | grep docker:

Sep 29 10:46:06 server3 kernel: [  220.837419] audit: type=1400 audit(1569779166.039:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=975 comm="apparmor_parser"
Sep 29 11:14:20 server3 dockerd[3322]: time="2019-09-29T11:14:20.645348694-07:00" level=info msg="libcontainerd: new containerd process, pid: 3329"
Sep 29 11:14:22 server3 kernel: [ 1916.701963] audit: type=1400 audit(1569780862.150:9): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=3342 comm="apparmor_parser"
Sep 29 11:15:34 server3 dockerd[3322]: time="2019-09-29T11:15:34.309096926-07:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Sep 29 11:15:34 server3 dockerd[3322]: time="2019-09-29T11:15:34.309491833-07:00" level=warning msg="Your kernel does not support swap memory limit."
Sep 29 11:15:34 server3 dockerd[3322]: time="2019-09-29T11:15:34.310262644-07:00" level=info msg="Loading containers: start."
Sep 29 11:15:35 server3 dockerd[3322]: ............................time="2019-09-29T11:15:35.240805555-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (96491bfe0c9cff2454d2565f7bfd99159fa2720fb417004cbc49f13f2e498a0e)."
Sep 29 11:15:35 server3 dockerd[3322]: time="2019-09-29T11:15:35.241460676-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers 96491bfe0c9cff2454d2565f7bfd99159fa2720fb417004cbc49f13f2e498a0e/shm: invalid argument"
Sep 29 11:17:10 server3 dockerd[3322]: time="2019-09-29T11:17:10.811812666-07:00" level=info msg="Processing signal 'terminated'"
Sep 29 11:17:20 server3 dockerd[3322]: time="2019-09-29T11:17:20.954444523-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (d4c0a7c04d8e50283ab3555eb534f974f1fd36bfa10c922de151adbc2f05db52)."
Sep 29 11:17:20 server3 dockerd[3322]: time="2019-09-29T11:17:20.955368704-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/d4c0a7c04d8e50283ab3555eb534f974f1fd36bfa10c922de151adbc2f05db52/shm: invalid argument"
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.299657097-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (68d7d76d552a8974b366730f97beaa752b26fdeeec5e197bf7392efa5c16f853)."
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.300564525-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/68d7d76d552a8974b366730f97beaa752b26fdeeec5e197bf7392efa5c16f853/shm: invalid argument"
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.489469578-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (2f3bad9727ab14654dea1247ff1b0a236dd83e423816805123712a05b5c4884b)."
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.490086369-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/2f3bad9727ab14654dea1247ff1b0a236dd83e423816805123712a05b5c4884b/shm: invalid argument"
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.657160860-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (c774112b6544330babdb58d5a445cd1fd103776defcbde338b3e973c145ee60b)."
Sep 29 11:17:21 server3 dockerd[3322]: time="2019-09-29T11:17:21.657908245-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/c774112b6544330babdb58d5a445cd1fd103776defcbde338b3e973c145ee60b/shm: invalid argument"
Sep 29 11:17:22 server3 dockerd[3322]: time="2019-09-29T11:17:22.402786336-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (df188824b0f7323a80b97b7bebeeca757fdbe3836f963029fae086d902d5da21)."
Sep 29 11:17:22 server3 dockerd[3322]: time="2019-09-29T11:17:22.403641552-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/df188824b0f7323a80b97b7bebeeca757fdbe3836f963029fae086d902d5da21/shm: invalid argument"
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.148499532-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (888f4ad8e6fe5747d1b051a2bbb3e941680c6160fba4eb4722c2901cc891a671)."
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.153025606-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/888f4ad8e6fe5747d1b051a2bbb3e941680c6160fba4eb4722c2901cc891a671/shm: invalid argument"
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.326681631-07:00" level=warning msg="libcontainerd: client is out of sync, restore was called on a fully synced container (635df72c69c2a94fba6fc41c08b48e08b46be8c515350d410089199d2d871404)."
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.327626754-07:00" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /srv/docker/containers/635df72c69c2a94fba6fc41c08b48e08b46be8c515350d410089199d2d871404/shm: invalid argument"
Sep 29 11:17:23 server3 dockerd[3322]: time="2019-09-29T11:17:23.594646547-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:23 server3 kernel: [ 2098.306289] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
Sep 29 11:17:24 server3 dockerd[3322]: time="2019-09-29T11:17:24.138296872-07:00" level=info msg="Removing stale sandbox b529a7fb5226c7126df9b7cfc4887eafb8bf9f11dcca6e4885ff4d3dc05a2617 (888f4ad8e6fe5747d1b051a2bbb3e941680c6160fba4eb4722c2901cc891a671)"
Sep 29 11:17:25 server3 dockerd[3322]: time="2019-09-29T11:17:25.228154698-07:00" level=info msg="Removing stale sandbox d659e1885345f922af8f2b2daba1fc08d6c2fe8f640a2cdd07f37befbef6700c (df188824b0f7323a80b97b7bebeeca757fdbe3836f963029fae086d902d5da21)"
Sep 29 11:17:25 server3 dockerd[3322]: time="2019-09-29T11:17:25.783207780-07:00" level=info msg="Removing stale sandbox 0d494be82acd7c90411b60f3d8977889b2284ca1a821004ded833bc3da13f73b (635df72c69c2a94fba6fc41c08b48e08b46be8c515350d410089199d2d871404)"
Sep 29 11:17:26 server3 dockerd[3322]: time="2019-09-29T11:17:26.170928661-07:00" level=info msg="Removing stale sandbox 1e5c49cadd7838950e953a9bb1d9fd6c2aad68aa96977775c3cfb5b4ce0046eb (2f3bad9727ab14654dea1247ff1b0a236dd83e423816805123712a05b5c4884b)"
Sep 29 11:17:26 server3 dockerd[3322]: time="2019-09-29T11:17:26.531080976-07:00" level=info msg="Removing stale sandbox 454dc89aa9e94352d3359d9fe3a62807883d38b1666fc23272bdbbb65e4ffe8b (d4c0a7c04d8e50283ab3555eb534f974f1fd36bfa10c922de151adbc2f05db52)"
Sep 29 11:17:26 server3 dockerd[3322]: time="2019-09-29T11:17:26.869025267-07:00" level=info msg="Removing stale sandbox 584b842cd64271c9b079943d638f64300c1a8dd72b13babd89edf2d944dea808 (96491bfe0c9cff2454d2565f7bfd99159fa2720fb417004cbc49f13f2e498a0e)"
Sep 29 11:17:27 server3 dockerd[3322]: time="2019-09-29T11:17:27.170912300-07:00" level=info msg="Removing stale sandbox 84146f3c236d6c8880a707db9e924d9c7e092c554250f9c4d7ec698ab9b84d1e (c774112b6544330babdb58d5a445cd1fd103776defcbde338b3e973c145ee60b)"
Sep 29 11:17:27 server3 dockerd[3322]: time="2019-09-29T11:17:27.647156587-07:00" level=info msg="Removing stale sandbox 8f84fe1dfcca7b8cbc94a2fe652b3fddf0cc9ea1b937588cb9feecf4b853a94c (68d7d76d552a8974b366730f97beaa752b26fdeeec5e197bf7392efa5c16f853)"
Sep 29 11:17:27 server3 dockerd[3322]: time="2019-09-29T11:17:27.988341341-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Sep 29 11:17:36 server3 dockerd[3322]: time="2019-09-29T11:17:36-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:36 server3 dockerd[3322]: time="2019-09-29T11:17:36-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:37 server3 dockerd[3322]: time="2019-09-29T11:17:37-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:42 server3 dockerd[3322]: time="2019-09-29T11:17:42-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:42 server3 dockerd[3322]: time="2019-09-29T11:17:42-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:48 server3 dockerd[3322]: time="2019-09-29T11:17:48-07:00" level=info msg="Firewalld running: false"
Sep 29 11:17:55 server3 dockerd[3322]: time="2019-09-29T11:17:55.972804177-07:00" level=info msg="Loading containers: done."
Sep 29 11:17:56 server3 dockerd[3322]: time="2019-09-29T11:17:56.031355165-07:00" level=info msg="Daemon has completed initialization"
Sep 29 11:17:56 server3 dockerd[3322]: time="2019-09-29T11:17:56.031415687-07:00" level=info msg="Docker daemon" commit=7392c3b graphdriver=overlay2 version=1.12.5
Sep 29 11:17:56 server3 dockerd[3322]: time="2019-09-29T11:17:56.045097929-07:00" level=info msg="API listen on /var/run/docker.sock"

Running service docker restart fixed it, after a delay.

[clonm]

This time it did eventually start, but something is wrong with rocketchat's dockergen setup so it's not starting automatically.