[Checklist] Move integration-cli tests to API tests

[dnephin]

Related to #32866

This issue is a checklist for prioritizing specific tests that need to be removed from the integration-cli suite, and moved to the new API integration suite.

• DockerSuite.TestRmiContainerImageNotFound (@dnephin wip)
• DockerSuite.TestRmiImageIDForceWithRunningContainersAndMultipleTags (@dnephin wip)
• DockerSuite.TestRunAttachDetachFromInvalidFlag
• DockerSuite.TestRunWithNanoCPUs (force inspect test format #34707 @vieux)
• DockerSuite.TestUpdateWithNanoCPUs (force inspect test format #34707 @vieux)
• DockerSwarmSuite.TestSwarmServicePsMultipleServiceIDs (Update service ps to be an API test #33684 @dnephin)

Deprioritized (test was fixed in place)

• DockerSwarmSuite.TestServiceLogs
• DockerSwarmSuite.TestSwarmNetworkPluginV2

If any test also requires additional testing of the CLI please add them to the docker/cli list as well: docker/cli#201

[dnephin]

@andrewhsu @tiborvass If you discover any test that fail in the integration test suite, please add them here (or comment and I can add them).

[andrewhsu]

@andrewhsu

[andrewhsu]

For reference, the problem with these tests are that they always fail with the latest code from docker/cli. Just a problem with the test, not a problem with the runtime code as demonstrated in docker/docker-ce#140

[andrewhsu]

Current situation for running the test-integration tests from moby/moby 184cea5 with a cli compiled from docker/cli e636a53 (which is the latest git commits from the respective master branches as of this morning) have more failures. Analyzing the list of failures with a helper script that checks for the 8 expected failures:

$ ./check_failures.sh https://jenkins.dockerproject.org/job/docker-integration-tests/108/consoleText
FOR: https://jenkins.dockerproject.org/job/docker-integration-tests/108/consoleText

Passed (thought would fail):
DockerSwarmSuite.TestServiceLogs
DockerSwarmSuite.TestSwarmNetworkPluginV2
DockerSwarmSuite.TestSwarmServicePsMultipleServiceIDs

Failed (should fail)
DockerSuite.TestRmiContainerImageNotFound
DockerSuite.TestRmiImageIDForceWithRunningContainersAndMultipleTags
DockerSuite.TestRunAttachDetachFromInvalidFlag
DockerSuite.TestRunWithNanoCPUs
DockerSuite.TestUpdateWithNanoCPUs

Failed (thought would pass):
DockerDaemonSuite.TestVolumePlugin
DockerNetworkSuite.TestConntrackFlowsLeak
DockerRegistrySuite.TestCreateByDigest
DockerRegistrySuite.TestRunByDigest
DockerRegistrySuite.TestRunImplicitPullWithNoTag
DockerSuite.TestRenameInvalidName
DockerTrustSuite.TestTrustedCreate
DockerTrustSuite.TestTrustedCreateFromBadTrustServer
DockerTrustSuite.TestTrustedIsolatedCreate
DockerTrustSuite.TestTrustedRun
DockerTrustSuite.TestTrustedRunFromBadTrustServer

Some of the errors in the Failed (thought would pass) sections may be legit bugs. Possibly DockerNetworkSuite.TestConntrackFlowsLeak. We're doing some analysis on them now.

[seemethere]

Confirmed TestConntrackFlowsLeak, issue #34693

Edit: issue has been moved to docker/cli#488

[dnephin]

It looks like most of these failures are fixed by docker/cli#488

The only exception is TestRenameInvalidName, which needs to be deleted, it's testing the CLI output strings which is just a problem with the test. We already test that in `docker/cli. I will open a PR to remove it.

[seemethere]

Can confirm that docker/cli#488 remedies all issues aside from known failures and TestRenameInvalidName

[dnephin]

#34695 fixes TestRenameInvalidName

[andrewhsu]

I think the 5 DockerSuite tests are the only ones that still fail on a regular basis with the code from master branch of docker/cli and moby/moby.

[seemethere]

Can confirm, from this morning (September 1, 2017) builds:

❯ bash ~/scripts/check_failures.sh https://jenkins.dockerproject.org/job/docker-integration-tests/110/consoleText
FOR: https://jenkins.dockerproject.org/job/docker-integration-tests/110/consoleText

Passed (thought would fail):
DockerSwarmSuite.TestServiceLogs
DockerSwarmSuite.TestSwarmNetworkPluginV2
DockerSwarmSuite.TestSwarmServicePsMultipleServiceIDs

Failed (should fail)
DockerSuite.TestRmiContainerImageNotFound
DockerSuite.TestRmiImageIDForceWithRunningContainersAndMultipleTags
DockerSuite.TestRunAttachDetachFromInvalidFlag
DockerSuite.TestRunWithNanoCPUs
DockerSuite.TestUpdateWithNanoCPUs

Failed (thought would pass):
DockerSuite.TestRenameInvalidName

[thaJeztah]

ping @cyli @n4ss could you help with porting the DockerTrustSuite tests? Looks like they still have to be disabled in the Docker CE repository CI, when running with a new Docker CLI.

[seemethere]

@thaJeztah @cyli @n4ss This is a major pain point for us when cutting new branches / doing nightly testing for Docker CE.

ping @andrewhsu

[n4ss]

@thaJeztah @seemethere yep sure, I'll port them!

[thaJeztah]

@n4ss actually ... I suspect those tests need the CLI code, so should probably be moved to the docker/cli repository (I wasn't really thinking when adding that comment); here's the list of tests that need to be looked into;

DockerTrust tests (see docker/docker-ce#397, and #34623 (comment))

• DockerTrustSuite.TestTrustedBuildTagFromReleasesRole
• DockerTrustSuite.TestTrustedBuildTagIgnoresOtherDelegationRoles
• DockerTrustSuite.TestTrustedPullReadsFromReleasesRole
• DockerTrustSuite.TestTrustedPullIgnoresOtherDelegationRoles
• DockerTrustSuite.TestTrustedPush
• DockerTrustSuite.TestTrustedPushWithIncorrectPassphraseForNonRoot

[vdemeester]

@thaJeztah yes, the DockerTrustSuite should be ported to docker/cli and removed from moby/moby as they are "cli-only". @n4ss if you need help porting them, feel free to ping me 😉