Problem
Parser.parse() in src/cloudai/parser.py globs and fully parses every .toml in --tests-dir and every hook toml, even when a scenario only references a few of them. Filtering to the tests actually used happens after everything is already loaded and validated.
This is hard to fix in isolation. A scenario references a test by its internal name field via test_name, not by file path. The only way to know which file has which name is to open and parse it. There is no name to file index, so lazy loading has nothing to work from without a change to how tests are referenced.
Proposal
Two small, sequential PRs.
PR 1: relative path test references. Add an optional path based reference on TestRunModel, alongside the existing test_name lookup. Fully backward compatible. Existing scenarios keep working unchanged. A [[Tests]] entry could point directly at a toml relative to the scenario file instead of relying on directory wide name resolution.
PR 2: lazy load referenced tomls. Depends on PR 1. When every [[Tests]] entry in a scenario uses a path reference, skip the eager glob and parse of the whole --tests-dir and hooks dir, and load only the files actually referenced. If a scenario mixes path and name references, fall back to today's eager behavior for the name based ones. Full mixed mode laziness can be a later follow up.
Open questions
What should a relative path resolve against. The scenario file's own directory seems most intuitive, flagging it rather than assuming.
Today, duplicate name values are rejected across the whole --tests-dir. Once loading is scoped to only the referenced files, does that check still need to hold directory wide, or just among the files actually loaded for a run.
Out of scope
Not touching the X | list[X] sweeping annotation issue or the wider parsing refactor. Those need their own scoping discussion.
Problem
Parser.parse()insrc/cloudai/parser.pyglobs and fully parses every.tomlin--tests-dirand every hook toml, even when a scenario only references a few of them. Filtering to the tests actually used happens after everything is already loaded and validated.This is hard to fix in isolation. A scenario references a test by its internal
namefield viatest_name, not by file path. The only way to know which file has which name is to open and parse it. There is no name to file index, so lazy loading has nothing to work from without a change to how tests are referenced.Proposal
Two small, sequential PRs.
PR 1: relative path test references. Add an optional path based reference on
TestRunModel, alongside the existingtest_namelookup. Fully backward compatible. Existing scenarios keep working unchanged. A[[Tests]]entry could point directly at a toml relative to the scenario file instead of relying on directory wide name resolution.PR 2: lazy load referenced tomls. Depends on PR 1. When every
[[Tests]]entry in a scenario uses a path reference, skip the eager glob and parse of the whole--tests-dirand hooks dir, and load only the files actually referenced. If a scenario mixes path and name references, fall back to today's eager behavior for the name based ones. Full mixed mode laziness can be a later follow up.Open questions
What should a relative path resolve against. The scenario file's own directory seems most intuitive, flagging it rather than assuming.
Today, duplicate
namevalues are rejected across the whole--tests-dir. Once loading is scoped to only the referenced files, does that check still need to hold directory wide, or just among the files actually loaded for a run.Out of scope
Not touching the
X | list[X]sweeping annotation issue or the wider parsing refactor. Those need their own scoping discussion.