Replace integration-cli suite with API test suite

[vieux]

Many tests in integration-cli rely on the Docker CLI. The CLI has moved into into a new repo, and moby should not depend on docker, so the integration test suite needs to be transitioned to a new API test suite.

Testing guidelines doc: ./TESTING.md

Execution plan:

1. Identify and remove tests that are CLI-only. Removed tests should be documented in [Testing] Tracking issue for missing tests docker/cli#201 so that we can ensure the CLI has necessary coverage. There are many tests that test both CLI and API, but once the CLI portion is removed, the API tests are duplicated, so some can be removed
2. Identify integration tests that should be unit tests. Top candidates include:
   • tests for error conditions
   • tests for specific flags/query params
     These tests should be replaced with unit tests
3. Convert remaining integration-cli tests to integration/
4. Move API integration tests from integration-cli/docker_api_*_test to integration/

See #34623 for a checklist of prioritized tests

[cpuguy83]

We should probably update the contributing guidelines to indicate that users should not be using integration-cli. Or at the least only make API tests.

To go along with this either docker/cli needs to setup integration testing or some separate integration repo for "docker-ce".

[vdemeester]

@cpuguy83 yes and we need to setup a ci check with those for the time being. I'll start working on that next week 😇

[vdemeester]

Note that there is discussion happening in https://forums.mobyproject.org/t/evolution-of-testing-moby/38

[dnephin]

Opened #34407 to document the proposal from the forum thread.

[dnephin]

Current status: #34000 and #34407 have merged so we have a new suite and a document for testing guidelines. A validation check has been added to prevent any new cli tests from being added (#34552).

Ongoing: removing cli-only tests: #34560 and #34548

Next steps:

• continue removing any cli-only tests
• move more tests to the new suite to provide more examples

I'll update the description of this issue as well.

[michael2012z]

Hello, everyone,

What is the latest status of this task? I saw there had been quite a few PR's working on migrating legacy CLI test cases into API cases. But the CLI cases are still the majority of all integration test. Is this job still calling for contribution? Just want to know if I can help in this issue.