test: roadmap for broader CLI/API coverage beyond current contract baseline

[axisrow]

Context

direct-cli already has full contract coverage against the declared Yandex Direct API surface (28/28 via WSDL/spec parity), but real integration confidence is lower.

Current integration coverage is split across:

• live read-only tests against production API
• sandbox write tests with VCR replay

Coverage is uneven: some groups are well covered, some are sandbox-limited, and some remain contract-only.

Current State

• Contract: 28/28
• Live read: 11/28
• Sandbox write: 17/28
• Any integration tier: 21/28 (75%)
• Contract-only groups: advideos, agencyclients, businesses, changes, keywordsresearch, leads, turbopages
• Explicit read gaps still noted in repo: sitelinks get, feeds get
• Current suite health blocker: TestWriteBidModifiers.test_toggle_existing replay is not green because cassette/state is stale and related API behavior is under investigation (#31, #33)

Roadmap

1. Stabilize the current integration baseline

• Resolve the bidmodifiers toggle branch of work first.
• Decide whether toggle is a valid API operation or whether the CLI must revert to another payload/method shape.
• Close or update #31 and #33 before expanding coverage on top of a broken baseline.

2. Close low-cost live-read gaps

• Add coverage for sitelinks get and feeds get using explicit fixture IDs or a documented prerequisite flow.
• Convert similar "requires explicit ID" gaps into deterministic integration cases instead of leaving them uncovered.

3. Expand integration coverage for contract-only groups

• Evaluate each uncovered group for the cheapest reliable tier:
  • live read
  • sandbox write
  • dry-run/schema parity only
  • manual/non-sandbox tier
• Prioritize in this order:
  • keywordsresearch
  • changes
  • agencyclients
  • businesses
  • leads
  • advideos
  • turbopages

4. Strengthen method/payload correctness checks where contract tests are too shallow

• Add assertions that dry-run or request builder output uses the exact API method name expected by WSDL/spec.
• Use keywordsresearch as the first concrete fix path because it already has an open mismatch issue (#35).

5. Make the coverage matrix explicit in repo docs and test headers

• Track every CLI group as one of:
  • contract-only
  • live-read covered
  • sandbox-write covered
  • sandbox-limited
  • blocked by API ambiguity
• Keep this matrix near the tests so future issue closure is based on visible state, not inference.

Acceptance Criteria

• every CLI group has an explicit coverage classification
• every contract-only group has a chosen next testing tier
• explicit read gaps (sitelinks, feeds) have a chosen strategy
• bidmodifiers toggle path is resolved as valid, invalid, or intentionally unsupported
• method-name parity checks exist for commands where CLI aliases can hide wrong wire method names
• repo documentation/test headers clearly describe the coverage matrix and remaining intentional gaps

Related

• #28 write/integration sandbox roadmap
• #31 stale cassette for bidmodifiers toggle
• #33 possible invalid API toggle method
• #35 keywordsresearch method-name mismatch

[axisrow]

Coverage audit update: declared parity vs live model gaps

Follow-up issue for the machine-readable/CI artifact work: #54.

Current audit result

The current declared coverage model still reports strict parity for its own WSDL surface:

• declared WSDL services: 27
• declared WSDL methods: 101
• strict_parity_ok: true

The live-discovered WSDL surface used for model-gap tracking is larger:

• live-discovered WSDL services: 29
• live-discovered WSDL methods: 112
• missing services from declared model / CLI: 2
• missing methods from declared model / CLI: 11
• live_model_parity_ok: false

Missing services

API service	Missing methods
dynamicfeedadtargets	add, delete, get, resume, setBids, suspend
strategies	add, archive, get, unarchive, update

Interpretation

strict_parity_ok should continue to mean “the CLI matches the declared canonical coverage model.” It should not be read as “the declared model includes every live Yandex Direct API service.”

The new model_gaps section in the coverage artifact is intended to make that distinction explicit and keep this issue’s coverage matrix actionable.

[axisrow]

Created the 0.3.0 release-gate roadmap for 100% command/API coverage: #55. This issue remains the coverage-matrix/broader-confidence tracker under that release gate; #55 defines the exact release criteria and links the blocker issues.

[axisrow]

Added issue-level API status audit requirement

The bidmodifiers.toggle case showed that broader coverage planning must consult closed issue evidence and official Yandex status before treating a command as missing coverage.

New rule for this roadmap:

• Every ambiguous command/API item must be classified from issue history + live WSDL + official docs/changelog before it becomes a blocker.
• Deprecated, removed, or not-in-WSDL operations are documented exclusions, not missing 0.3.0 coverage.
• Closed issues can be evidence. Example: test: re-record VCR cassette for bidmodifiers toggle after CLI fix #31 contained the sandbox error 55 evidence for bug: bidmodifiers toggle method returns error 55 on both sandbox AND production #33.

Current concrete classification:

Issue	API surface	Status	Action
#33	bidmodifiers.toggle	deprecated + not in WSDL	closed as not planned; do not count as missing coverage
#31	bidmodifiers.toggle cassette	superseded evidence	keep closed as evidence
#35	keywordsresearch camelCase methods	supported	keep as real fix

The local audit register is tests/API_ISSUE_AUDIT.md.

[axisrow]

Status: all acceptance criteria met

All 6 acceptance criteria have been fulfilled by subsequent work:

1. Every CLI group has an explicit coverage classification — smoke_matrix.py classifies all 122 CLI subcommands into SAFE (38), WRITE_SANDBOX (77), DANGEROUS (7).
2. Every contract-only group has a chosen next testing tier — smoke matrix assigns a tier to every command; sandbox Category A/B limitations documented.
3. Explicit read gaps (sitelinks, feeds) resolved — feeds fixed in fix(feeds): use valid FieldNames for get (SourceType, not Source) #84; sitelinks documented as sandbox Category A (permanently unavailable in sandbox).
4. Bidmodifiers toggle resolved — removed as deprecated (bug: bidmodifiers toggle method returns error 55 on both sandbox AND production #33 closed, Yandex deprecated toggle on 2025-11-13).
5. Method-name parity checks exist — METHOD_NAME_OVERRIDES in wsdl_coverage.py + test_api_coverage.py validates WSDL method parity.
6. Coverage matrix documented — API_COVERAGE.md, API_ISSUE_AUDIT.md, MANUAL_COVERAGE.md provide comprehensive coverage status.