fix(#655): add vector_database + kv_cache subtrees to DoD fixture

[FileSystemGuy]

Closes #655.

Summary

The Definition-of-Done end-to-end test (mlpstorage_py/tests/test_definition_of_done.py) exercises the full validator pipeline as a subprocess against a fixture built by conftest.build_submission. That fixture only ever creates training + checkpointing subtrees, so MODE_TO_CHECKERS never routed to VdbCheck (17 real §5 @rule bindings) or KVCacheCheck (§6 stub, will gain real bindings via PR #602). Their unit tests in test_vdb_checks.py (41 tests, all passing) exercise them in isolation, but the loader → checker hand-off, the log-line format, and the origin-path binding were unverified end-to-end.

This PR adds three kwargs on build_submission:

• include_vdb=True — creates results/<sys>/vector_database/milvus/DISKANN/{datagen,run}/<ts>/ per loader.py:207-238. Five run timestamps because 5.3.1 vdbRunCount requires exactly 5. Each run dir carries a §5-conformant summary.json + metadata.json + a CAP-03 fs_separation.json sidecar so 5.4.2 vdbFilesystemCheck passes.
• include_kv_cache=True — creates results/<sys>/kv_cache/llama3.1-8b/{datagen,run}/<ts>/ per loader.py:191-205. Because KVCacheCheck is a stub, the good-fixture assertion only needs to prove the loader routes through MODE_TO_CHECKERS without tripping [2.1.10 workloadCategories]. The subtree is scaffolded so a follow-up (once PR KV cache Rules for closed and open submission #602 gives §6 real bindings) can add a kv_cache_missing_field knob mirroring vdb_missing_metric_field.
• vdb_missing_metric_field=<name> — pops the named field from the vdb run summary.json to engineer 5.3.4 vdbMetricsReported end-to-end (bad-fixture case).

Also fixed: stale docstring

vdb_checks.py:150-155 — the _iter_run_files docstring described "Phase 4 land time" behavior:

Loader.load() only fills run_files / datagen_files for mode == "training"; the else branch fills checkpoint_files for everything else. For vector_database leaves this means run_files is None.

That was already outdated when #655 was filed — #612 added explicit run_files / datagen_files population for both vector_database (loader.py:224-225) and kv_cache (loader.py:198-199). Updated the docstring to reflect the current loader behavior; the empty-iterable guard remains as a defensive walk for test-time SubmissionLogs fakes.

Follow-up not in this PR

• Once PR KV cache Rules for closed and open submission #602 gives §6 real @rule bindings, add a kv_cache_missing_field kwarg and a TestDefinitionOfDoneKvcache bad-fixture case mirroring the vdb structure.
• The vanilla-fixture "no §5 rules tripped" assertion only checks that no 5.x.y prefix appears at ERROR level; it does not extend TestDefinitionOfDoneGood.test_good_fixture_does_not_trip_bad_fixture_rule_ids to add 5.3.4 to _EXPECTED_BAD_FIXTURE_RULE_IDS. Doing so would require a bad-fixture engineering of both §5.3.4 and one §6 rule (once those exist), so both live-fixture cases assert the same subset. Bundled into the same §6 follow-up.

Test plan

TDD RED-first (per project convention):

• `9224103 test(DoD end-to-end fixture omits vector_database and kv_cache subtrees — §5 checks and future §6 checks never fire #655): RED — DoD fixture lacks vector_database and kv_cache subtrees`
• `97ac94a fix(DoD end-to-end fixture omits vector_database and kv_cache subtrees — §5 checks and future §6 checks never fire #655): add vector_database and kv_cache subtrees to build_submission`

All 5 new tests failed on RED with TypeError: unknown override; all pass on GREEN after two rounds (initial GREEN tripped 5.3.1 vdbRunCount + 5.4.2 vdbFilesystemCheck against my one-timestamp fixture; fixed by extending to 5 timestamps + adding fs_separation.json sidecars).

Parallel 4-suite sweep:

• `tests/` — 2611 passed
• `mlpstorage_py/tests/` — 821 passed (+9 vs pre-fix(#653): allow N-part rule IDs in rules_coverage + violation-log parsers #656: 4 from fix(#653): allow N-part rule IDs in rules_coverage + violation-log parsers #656 + 5 new DoD end-to-end fixture omits vector_database and kv_cache subtrees — §5 checks and future §6 checks never fire #655 tests)
• `vdb_benchmark/tests/` — 155 passed
• `kv_cache_benchmark/tests/` — 238 passed
• Total: 3825 passed, 0 failed

Related

• fix(#653): allow N-part rule IDs in rules_coverage + violation-log parsers #656 — merged in the same review window; widened the log-line extractor regex to accept §5/§6 and 4-part IDs, which this PR's _extract_error_rule_ids usage depends on.
• KV cache Rules for closed and open submission #602 — will need a kv_cache_missing_field follow-up here once §6 gets real @rule bindings.
• Submission checker cannot validate kvcache or vector_database submissions (three coupled bugs) #612 — closed the loader-side half of the escape; the docstring cleanup here reflects that fix.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅