Enhanced Dockerfile for cross compilation (test Jenkinsfile changes)

[thaJeztah]

Just a rebase of #43529, but Jenkinsfile changes are not taken into account for pull requests if the contributor doesn't have write access), so opening this to put everything to the test 😅

[crazy-max]

[2022-05-18T13:57:36.674Z] === Failed
[2022-05-18T13:57:36.674Z] === FAIL: amd64.integration-cli TestDockerDaemonSuite/TestHTTPSInfoRogueServerCert (0.55s)
[2022-05-18T13:57:36.674Z]     docker_cli_daemon_test.go:1407: Expected err: x509: certificate signed by unknown authority, got instead: exit status 1 and output: error during connect: Get "https://localhost:4272/v1.30/info": x509: certificate relies on legacy Common Name field, use SANs instead
[2022-05-18T13:57:36.674Z]     --- FAIL: TestDockerDaemonSuite/TestHTTPSInfoRogueServerCert (0.55s)
[2022-05-18T13:57:36.674Z] 
[2022-05-18T13:57:36.674Z] === FAIL: amd64.integration-cli TestDockerDaemonSuite (335.42s)

Looks like I need to fix the rogue certs for Jenkins like I've done on this branch for GHA: crazy-max@11a16d5

[crazy-max]

[2022-05-18T13:57:36.674Z] === Failed
[2022-05-18T13:57:36.674Z] === FAIL: amd64.integration-cli TestDockerDaemonSuite/TestHTTPSInfoRogueServerCert (0.55s)
[2022-05-18T13:57:36.674Z]     docker_cli_daemon_test.go:1407: Expected err: x509: certificate signed by unknown authority, got instead: exit status 1 and output: error during connect: Get "https://localhost:4272/v1.30/info": x509: certificate relies on legacy Common Name field, use SANs instead
[2022-05-18T13:57:36.674Z]     --- FAIL: TestDockerDaemonSuite/TestHTTPSInfoRogueServerCert (0.55s)
[2022-05-18T13:57:36.674Z] 
[2022-05-18T13:57:36.674Z] === FAIL: amd64.integration-cli TestDockerDaemonSuite (335.42s)

Looks like I need to fix the rogue certs for Jenkins like I've done on this branch for GHA: crazy-max@11a16d5

Of found the root cause of this issue. this is linked to the new Dockerfile that now builds the docker cli for the integration tests:

moby/Dockerfile

Lines 453 to 460 in 1bf5dbb

	FROM dockercli-base AS dockercli
	RUN --mount=from=dockercli-src,src=/usr/src/dockercli/components/cli,rw \
	--mount=type=cache,target=/root/.cache \
	--mount=type=cache,target=/go/pkg/mod <<EOT
	set -e
	go build -o /out/docker -v ./cmd/docker
	xx-verify /out/docker
	EOT

previously the Dockerfile was just downloading the docker cli from download.docker.com and this one was using an old Go version that didn't have this requirement for an alt name yet. I think it's fair to update the rogue certs anyway as in the future we are going to have the exact same issue anyway.

[crazy-max]

@thaJeztah Made some changes in #43529 so now docker cli is built against the right go version if you want to rebase to check that tests are ok now. let me know if you still want a way in the dockerfile to choose between building vs downloading.

[crazy-max]

@thaJeztah Made some changes in #43529 so now docker cli is built against the right go version if you want to rebase to check that tests are ok now. let me know if you still want a way in the dockerfile to choose between building vs downloading.

------
[2022-05-19T09:52:04.586Z]  > [internal] load metadata for docker.io/library/golang:1.8.3:
[2022-05-19T09:52:04.586Z] ------
[2022-05-19T09:52:04.586Z] failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = no match for platform in manifest sha256:32c769bf92205580d6579d5b93c3c705f787f6c648105f00bb88a35024c7f8e4: not found

Argh no arm64 arch for this golang image, I guess we need to go back to the previous logic with dow,loading from download.docker.com.

[thaJeztah]

Some failures - not sure if all related, but this one I hadn't seen before (or at least, not that I recall 🤔) - it's networking, so could still be just flaky 😂

=== RUN   TestQueryEndpointInfo
time="2022-05-19T18:09:59Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge/net1/ is not added to the store"
time="2022-05-19T18:09:59Z" level=warning msg="bridge store not initialized. kv object docker/network/v1.0/bridge-endpoint/ep1/ is not added to the store"
time="2022-05-19T18:09:59Z" level=warning msg="Failed to allocate and map port 23000-23000: Error starting userland proxy: listen tcp [::]:23000: bind: address already in use"
    bridge_test.go:665: Failed to program external connectivity: Error starting userland proxy: listen tcp [::]:23000: bind: address already in use
--- FAIL: TestQueryEndpointInfo (0.07s)

[crazy-max]

Some failures - not sure if all related, but this one I hadn't seen before (or at least, not that I recall 🤔) - it's networking, so could still be just flaky 😂

Should be fixed now if you can rebase, thx!

[crazy-max]

@thaJeztah Looks good on Jenkins, minus unrelated error while cleaning the workspace on windows nodes:

And known flaky test on GHA workflow for Windows #38521:

=== Failed
=== FAIL: github.com/docker/docker/integration-cli TestDockerSuite/TestStartReturnCorrectExitCode (5.92s)
    docker_cli_start_test.go:191: assertion failed: 
        Command:  D:\a\moby\moby\out\docker.exe start -a withRestart
        ExitCode: 0
        Stdout:   
        Stderr:   
        
        Failures:
        ExitCode was 0 expected 11
    --- FAIL: TestDockerSuite/TestStartReturnCorrectExitCode (5.92s)

[crazy-max]

Argh another racy one linked to #11966: https://github.com/moby/moby/runs/6591181007?check_suite_focus=true#step:15:4943

=== Failed
=== FAIL: github.com/docker/docker/integration-cli TestDockerSuite/TestRunContainerWithRmFlagCannotStartContainer (2.18s)
    docker_cli_run_test.go:2770: Expected not to have containers d0c6fef36fc5

And looks like Windows node on Jenkins has timed out: https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-43613/4/pipeline#step-284-log-1841

[crazy-max]

@thaJeztah TestNetworkDBIslands known as flaky #42459 failed: https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-43613/7/pipeline#step-327-log-110

=== FAIL: libnetwork/networkdb TestNetworkDBIslands (123.95s)

Can you restart the Jenkins pipeline?

[thaJeztah]

Restarted!

[crazy-max]

Looks like this one is still flaky #11966:

=== Failed
=== FAIL: github.com/docker/docker/integration-cli TestDockerSuite/TestRunContainerWithRmFlagCannotStartContainer (1.79s)
    docker_cli_run_test.go:2770: Expected not to have containers 0f26800e99b0
        
    --- FAIL: TestDockerSuite/TestRunContainerWithRmFlagCannotStartContainer (1.79s)

https://github.com/moby/moby/runs/6669613995?check_suite_focus=true#step:15:4943

Will wait for #43672 to be merged and keep you in touch when #43529 is updated.

[crazy-max]

@thaJeztah Ok #43529 rebased with latest changes from master.

[thaJeztah]

done 👍

[crazy-max]

Same one bites the dust: https://github.com/moby/moby/runs/6689351547?check_suite_focus=true#step:15:4948

=== FAIL: github.com/docker/docker/integration-cli TestDockerSuite/TestRunContainerWithRmFlagCannotStartContainer (1.66s)
    docker_cli_run_test.go:2770: Expected not to have containers 6a5c139fc3dc
        
    --- FAIL: TestDockerSuite/TestRunContainerWithRmFlagCannotStartContainer (1.66s)

[AkihiroSuda]

Needs rebase

[crazy-max]

This one is interesting: https://ci-next.docker.com/public/blue/organizations/jenkins/moby/detail/PR-43613/11/pipeline/252#step-253-log-76

[2022-08-04T23:02:23.570Z] docker/errors.py:31: in create_api_error_from_http_exception
[2022-08-04T23:02:23.570Z]     raise cls(e, response=response, explanation=explanation)
[2022-08-04T23:02:23.570Z] E   docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.43/images/create?tag=latest&fromImage=hello-world: Internal Server Error ("Head "https://registry-1.docker.io/v2/library/hello-world/manifests/latest": Get "https://auth.docker.io/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=registry.docker.io": EOF")

@thaJeztah Maybe linked to what you said at maintainer meeting about the arm32v7/hello-world image.

I think I will bring some bits of crazy-max@7751ec5 in the PR just to check if this is a regression.

[thaJeztah]

I kicked Jenkins again

[crazy-max]

:)