Sometimes a container get stuck in restarting mode

[foretspaisibles]

CONTAINER ID        IMAGE                           COMMAND                   CREATED             STATUS                              PORTS                    NAMES
78d87d7f4ac8        xxx/yyy  "command"   4 weeks ago         Restarting (-1) About an hour ago   0.0.0.0:2012->2012/tcp   yyy-2012

These containers still can be stopped and removed.

The condition ist not perfectly reproducible. These containers are run with a memory limit set --memory=1318m and the condition seems to occur only after that the container has been killed because of excessing this memory limit.

$ docker info
Containers: 3
Images: 17
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-202:1-264750-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: 
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 795.2 MB
 Data Space Total: 107.4 GB
 Data Space Available: 27.62 GB
 Metadata Space Used: 2.032 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.145 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.7-15.23.amzn1.x86_64
Operating System: Amazon Linux AMI 2015.09
CPUs: 2
Total Memory: 3.862 GiB
Name: ip-172-16-1-241
ID: MKMR:UVOD:VZ5M:AZWU:5CH7:YXWZ:UOI3:FY4R:YBZA:3MU4:AFG7:CH7U
Username: quintlyworker
Registry: https://index.docker.io/v1/

$ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5/1.9.1
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5/1.9.1
 Built:        
 OS/Arch:      linux/amd64

$ uname -a
Linux ip-172-16-1-241 4.1.7-15.23.amzn1.x86_64 #1 SMP Mon Sep 14 23:20:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[thaJeztah]

This looks like expected behavior; if there's a restart-policy set on the container, the docker daemon will attempt to restart the container if it exits, and if there's no "max-retries" set, it will keep trying to restart the container each time it exits.

[HackToday]

hi @michipili Could that contianer be removed with force option?

[foretspaisibles]

@thaJeztah There is no indication that the container is actually restarted and exited. As far as I can tell, the program in the container does not start. We see no application logs, no log output. In other cases where a restart policy causes a “restart-exit loop”, docker ps never shows durations like one hour as displayed in the output above Restarting (-1) About an hour ago but always within a few seconds, if the state even can be glimpsed at

@HackToday The affected containers could be stopped and removed without using any forcing flag.

[foretspaisibles]

/CC @fredpe

[jpcope]

I am also seeing this issue:

$ docker info
Containers: 5
Images: 35
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-253:1-262145-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.433 GB
 Data Space Total: 107.4 GB
 Data Space Available: 40.65 GB
 Metadata Space Used: 5.132 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.142 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /mnt/docker/devicemapper/devicemapper/data
 Metadata loop file: /mnt/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 8
Total Memory: 14.28 GiB
Name: jobs-shared.stage.1ddd719a.va.localdomain
ID: 4OMM:YI7E:6IAX:P7DU:3XQK:CFWZ:PQBO:6TYE:SNW7:UZX2:6WKW:JGAH

$ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:25:01 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:25:01 UTC 2015
 OS/Arch:      linux/amd64

$ uname -a
Linux dockers.localdomain 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Any advice? Is a workaround known at this time?

[jpcope]

For my case, the containers can vary in execution time, sometimes less than a 2-3 seconds.

[jpcope]

Looking at the docker logs redirected me to #17653 going to give the workaround a shot #17653 (comment)

[thaJeztah]

@jpcope the "restart" timer is reset after a container has been running for 10 seconds, if it exits before that time, the duration between restarts is incremented with each restart. the "restarting" status, basically is "wait for the next restart attempt", and that can grow rapidly. If you have containers that only need to run for a short time, and be started with a certain interval, something like a cron-job to start those containers may be a better solution than setting a restart-policy

[jpcope]

@thaJeztah Thank you :-)

When I found this in the manual I did just that. Thanks for following up as I was experiencing both this and the issue in #17653

[nagaran1]

First check the logs why the container failed. Because your restart policy might bring your container back to running status. Better to fix the issue, Then probably you can build a new image with/without fix. Later execute below command

docker system prune

https://forums.docker.com/t/docker-registry-in-restarting-1-status-forever/12717/3