Flaky test: TestCheckpoint

[thaJeztah]

This test was added recently in #38452, but looks to be flaky;

Seen failing on #38910, #38441, #38952, #38939

For example:

https://jenkins.dockerproject.org/job/Docker-PRs-experimental/44645/console

01:42:48 === RUN   TestCheckpoint
01:42:50 --- FAIL: TestCheckpoint (1.54s)
01:42:50     checkpoint_test.go:40: Error (criu/util.c:816): exited, status=3
01:42:50         Warn  (criu/net.c:2840): Unable to get socket network namespace
01:42:50         Warn  (criu/net.c:2840): Unable to get tun network namespace
01:42:50         Warn  (criu/sk-unix.c:229): unix: Unable to open a socket file: Bad address
01:42:50         Warn  (criu/net.c:2840): Unable to get socket network namespace
01:42:50         Warn  (criu/kerndat.c:881): Can't keep kdat cache on non-tempfs
01:42:50         Looks good.
01:42:50     checkpoint_test.go:51: Start a container
01:42:50     checkpoint_test.go:69: ++ type -P true
01:42:50         ++ type -P ip6tables-restore
01:42:50         + mount --bind /bin/true /sbin/ip6tables-restore
01:42:50         ++ type -P true
01:42:50         ++ type -P ip6tables-save
01:42:50         + mount --bind /bin/true /sbin/ip6tables-save
01:42:50     checkpoint_test.go:81: Do a checkpoint and leave the container running
01:42:50     checkpoint_test.go:24: Exec: [touch /tmp/test-file]
01:42:50     checkpoint_test.go:28: 
01:42:50     checkpoint_test.go:115: Do a checkpoint and stop the container
01:42:50     checkpoint_test.go:117: assertion failed: error is not nil: Error response from daemon: Cannot checkpoint container 632ee0139e1239e1fa4a6bbd6e37ad7c21500fe4d2d8ee8139d40e26cebd6e92: cannot checkpoint a stopped container: unknown
01:42:50     checkpoint_test.go:77: ++ type -P ip6tables-restore
01:42:50         + umount -c -i -l /sbin/ip6tables-restore
01:42:50         ++ type -P ip6tables-save
01:42:50         + umount -c -i -l /sbin/ip6tables-save
01:42:50     main_test.go:32: assertion failed: error is not nil: Error response from daemon: Container 632ee0139e1239e1fa4a6bbd6e37ad7c21500fe4d2d8ee8139d40e26cebd6e92 is not paused: failed to unpause container 632ee0139e1239e1fa4a6bbd6e37ad7c21500fe4d2d8ee8139d40e26cebd6e92

[thaJeztah]

ping @kolyshkin @avagin @rst0git

[thaJeztah]

Attaching the daemon and test logs:

docker.log
test.log

[rst0git]

returned error: Cannot checkpoint container 632ee0139e1239e1fa4a6bbd6e37ad7c21500fe4d2d8ee8139d40e26cebd6e92: cannot checkpoint a stopped container: unknown

[thaJeztah]

Yes, definitely something weird happening (perhaps an earlier test that doesn't clean up properly after itself?)

[kolyshkin]

This is some kind of a race between Exec (which is not yet deleted) and Checkpoint. Have to look deeper, but it seems Checkpoint sets exec task state to Paused, then (p *process) Delete() is called (asynchronously from some event loop) and it can't delete that process because of its state, here:
https://github.com/containerd/containerd/blob/f2a20ead833f8caf3ffc12be058d6ce668b4ebed/process.go#L204