Add pagination for known tests API

[anmarchenko]

What

Fix known-tests pagination in the CI Visibility client by reading backend pagination metadata from data.attributes.page_info, sending follow-up cursors as data.attributes.page_info.page_state, and merging tests across pages.

Why

Shepherd issue https://github.com/DataDog/shepherd/issues/46 identified that known-tests clients must use the data.attributes.page_info contract and let the backend choose page size. ddtest already avoided top-level page_info requests, but it did not follow has_next cursors, so large known-tests responses could stop after page 1.

E2E testing

Run ddtest against a CI Visibility backend or mockdog endpoint that returns known tests with data.attributes.page_info.has_next=true and cursor="page-2"; verify ddtest sends a second known-tests request with data.attributes.page_info.page_state="page-2", omits page_size, and plans with tests from both pages.

Validated with go test ./civisibility/utils/net, make test, and make lint.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: daa97d41ff

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[anmarchenko]

E2E Test Report: SUCCESS

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

• Method: Local Shepherd E2E with sidekiq, ddtest plan, and mockdog-backed CI Visibility responses
• PR: Add pagination for known tests API #71
• Revision tested: 718f75af56338dee7300941b521f96bfe99b1539
• ddtest dependency: anmarchenko/fix-known-tests-pagination

Results

Check	Status	Evidence
Single-page known-tests response	Passed	ddtest made 1 known_tests request and cached 1 module / 1 suite / 1 test.
Multi-page known-tests response	Passed	ddtest made 2 requests, sent follow-up cursor as data.attributes.page_info.page_state = "page-2", omitted page_size, and cached merged known tests from both pages.
First page succeeds, second page errors	Passed	mockdog returned 500 for page-2; ddtest retried page 2, treated the whole known-tests flow as errored, planned with 0 modules / 0 suites / 0 tests, and did not write cache/http/known_tests.json.
Cached multi-page response shape	Passed	cache/http/known_tests.json has the expected JSON:API shape under data.attributes.tests.
Ruby cache parser compatibility	Passed	Datadog::CI::TestTracing::KnownTests::Response.from_json(...) parsed the cached multi-page response into exactly 2 known test ids, one from each page.
mockdog validation	Passed	make check and make build passed after adding mockdog pagination/error support for this scenario.

Cache Verification

The multi-page cached known_tests.json contains tests from both pages:

PageOneSuite at test/page_one_test.rb -> test_page_one
PageTwoSuite at test/page_two_test.rb -> test_page_two

Ruby parsed it through the real datadog-ci-rb cache path as:

{available: true, tests_count: 2, tests: [
  "PageOneSuite at test/page_one_test.rb.test_page_one.",
  "PageTwoSuite at test/page_two_test.rb.test_page_two."
]}

Issues Found

No ddtest regressions found in this pass. Shepherd/mockdog needed additional scenario support to exercise known-tests pagination and page-specific failures; that support was added locally and validated with mockdog checks.

Datadog UI Verification

Not applicable for this request: the behavior under test is ddtest's known-tests client pagination, local cache output, and Ruby cache compatibility using mockdog as the CI Visibility backend.

Test Methodology

1. Added mockdog support for backend-driven known-tests pagination and page-specific known-tests failures.
2. Ran ddtest plan through Shepherd/crook against sidekiq using three mockdog scenarios: single page, multiple pages, and second-page failure.
3. Inspected mockdog request recordings to verify cursor placement and absence of page_size.
4. Inspected cache/http/known_tests.json for merged multi-page contents.
5. Parsed the cached response using datadog-ci-rb's actual known-tests response/cache reader code.

---

This E2E test was performed by Shepherd - autonomous QA agent for Datadog Test Optimization.