Fix container pid race condition.

[Random-Liu]

If you try docker with containerd 1.3, and run docker run hello-world, it will stuck.

After #3366, we change pid after container/exec process created, this causes various issues:

1. The pid we get from task.Start might be -1 if the container is a short running container, because Start and State are not called in a transaction: https://github.com/containerd/containerd/blob/master/services/tasks/local.go#L230.
2. This is fine for Kubernetes integration, because we stopped relying on containerd event and pid matching. However, this will break Docker: https://github.com/moby/moby/blob/05469b5fa2b48cf20cd0137ca8c45645b63049ff/daemon/monitor.go#L50
3. In all higher level apis, pid is an uint, returning -1 will eventually make it 4294967295, which is not very user friendly:

        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 4294967295,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2019-11-27T22:21:24.999965567Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },

This PR partially reverts #3366, and use exited to figure out whether the process has already exited. I tested the change with Docker, and docker run hello-world works now.

Signed-off-by: Lantao Liu lantaol@google.com

[codecov-io]

Codecov Report

Merging #3857 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3857   +/-   ##
=======================================
  Coverage   42.34%   42.34%           
=======================================
  Files         130      130           
  Lines       14683    14683           
=======================================
  Hits         6217     6217           
  Misses       7540     7540           
  Partials      926      926

Flag	Coverage Δ
#linux	45.76% <ø> (ø)	⬆️
#windows	37.82% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3a31ce2...a6b6097. Read the comment docs.

[theopenlab-ci]

Build succeeded.

• containerd-build-arm64 : NODE_FAILURE in 0s (non-voting)

[AkihiroSuda]

Can we have a test?

[Random-Liu]

Can we have a test?

Done.

To make it easy to reproduce the race condition, I added a time.Sleep(5 * time.Second) before https://github.com/containerd/containerd/blob/v1.3.0/services/tasks/local.go#L225 in my experiment.

Here is the test result with containerd v1.3.0:

$ sudo PATH=$PATH GOPATH=$GOPATH TESTFLAGS="--test.run=TestShortRunningTaskPid -v" make integration
+ integration
INFO[0000] running tests against containerd              revision=942a06f75080ea055514d301dba83a8d71acc099.m runtime= version=v1.3.0-1-g942a06f7.m
INFO[0000] start to pull seed image                     
=== RUN   TestShortRunningTaskPid
=== PAUSE TestShortRunningTaskPid
=== CONT  TestShortRunningTaskPid
--- FAIL: TestShortRunningTaskPid (5.44s)
    container_test.go:1625: Pid -1
    container_test.go:1627: Unexpected task pid -1

Here is the test result with this patch:

$ sudo PATH=$PATH GOPATH=$GOPATH TESTFLAGS="--test.run=TestShortRunningTaskPid -v" make integration
+ integration
INFO[0000] running tests against containerd              revision=0f68b5b1ca30210d15391db4f889b8970f45ca9d.m runtime= version=v1.3.0-3-g0f68b5b1.m
INFO[0000] start to pull seed image                     
=== RUN   TestShortRunningTaskPid
=== PAUSE TestShortRunningTaskPid
=== CONT  TestShortRunningTaskPid
--- PASS: TestShortRunningTaskPid (5.80s)
    container_test.go:1625: Pid 227421
PASS
ok  	github.com/containerd/containerd	7.806s

BTW, I tried to add retry in the test to help reproducing the race condition, but it is hard. However, it can be easily reproduced by running docker run hello-world with containerd 1.3.0 at least on COS.

[theopenlab-ci]

Build succeeded.

• containerd-build-arm64 : SUCCESS in 5m 24s (non-voting)

[crosbymichael]

LGTM