Flaky test: TestStartReturnCorrectExitCode

[olljanat]

Seen on #31551

18:43:13 
18:43:13 ----------------------------------------------------------------------
18:43:13 FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
18:43:13 
18:43:13 docker_cli_start_test.go:199:
18:43:13     c.Assert(exitCode, checker.Equals, 12, check.Commentf("out: %s", out))
18:43:13 ... obtained int = 125
18:43:13 ... expected int = 12
18:43:13 ... out: time="2019-01-06T18:42:58Z" level=error msg="error waiting for container: EOF" 

Was failed on RS1 and found from Jenkins logs that it has failed at least couple of times on RS5 and at least on powerpc too

[olljanat]

Also on:

powerpc/2019/1/3/PR38473_12813.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
rs1/2019/1/9/PR38522_23622.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
rs1/2019/1/5/PR38500_23568.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
rs1/2019/1/5/PR143_23558.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
rs1/2019/1/4/PR38474_23549.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
rs1/2019/1/4/PR38480_23541.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
rs1/2019/1/6/PR31551_23575.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
rs1/2018/12/31/PR38392_23467.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
rs1/2018/12/31/PR38459_23462.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode
rs5/2019/1/3/PR38482_968.log:FAIL: docker_cli_start_test.go:189: DockerSuite.TestStartReturnCorrectExitCode

[andrewhsu]

I just saw this flaky test fail on PR #39638. From the console logs:

FAIL: docker_cli_start_test.go:190: DockerSuite.TestStartReturnCorrectExitCode

docker_cli_start_test.go:200:
    c.Assert(exitCode, checker.Equals, 12, check.Commentf("out: %s", out))
... obtained int = 125
... expected int = 12
... out: time="2019-08-01T07:18:09Z" level=error msg="error waiting for container: EOF" 

[andrewhsu]

Not the same error message, but I saw this flake out on PR #39655 powerpc check:

FAIL: docker_cli_start_test.go:190: DockerSuite.TestStartReturnCorrectExitCode

assertion failed: expected an error, got nil

[thaJeztah]

This test was added in b8464c1 (#25861)

[andrewhsu]

Seeing this issue on windowsRS1 in the master builds: https://ci.docker.com/public/blue/organizations/jenkins/moby/activity/?branch=master

e.g.

[2019-10-01T00:30:07.137Z]     --- FAIL: TestDockerSuite/TestStartReturnCorrectExitCode (8.05s)
[2019-10-01T00:30:07.137Z]         docker_cli_start_test.go:198: assertion failed: 125 (exitCode int) != 12 (int): out: time="2019-10-01T00:28:44Z" level=error msg="error waiting for container: EOF" 

[thaJeztah]

Wondering if it's also related to load on the test machine; I think it's been mainly failing on RS1 and Power/s390x, which have been slower/more resource constraint. For example, the same test on amd64 runs in 2.3 seconds; https://ci.docker.com/public/job/moby/job/PR-39992/3/testReport/amd64/integration-cli/Build___amd64___Run_tests___TestDockerSuite_TestStartReturnCorrectExitCode/

[thaJeztah]

Had a failure of this test on Linux this time; #41605, so not just affecting Windows it seems

=== RUN   TestDockerSuite/TestStartReturnCorrectExitCode
    --- FAIL: TestDockerSuite/TestStartReturnCorrectExitCode (1.04s)
        docker_cli_start_test.go:209: assertion failed: expected an error, got nil