Add examples/supported/: examples tested as product behaviour (#104) - #445
Conversation
`examples/` holds 111 pipelines of which 3 compile. They are the
project's primary documentation surface, so a reader has no way to tell
which parts of the syntax are real.
examples/supported/ is the answer: a small set that is tested as product
behaviour. Every file in it must validate, execute through the CLI,
execute through the Python API, produce the same normalised result
document from both surfaces, and produce the artifacts and declared
outputs its expectations record. tests/test_supported_examples.py
enforces all five and fails if a file is added without an entry
declaring what it should do, so the set cannot grow past its coverage.
Five to start, each exercising semantics pinned in the preceding PRs:
typed parameters and declared outputs; parallel fan-out with a dependent
join; `condition:` with `on_false` routing and skipped steps; timeout,
bounded retry and `on_failure: continue`; step-result references.
06_failure_policy exits 1 on purpose. Not every supported example is a
success case, and the expectations table records the intended exit code
rather than assuming zero -- a pipeline containing a failing step is
supposed to say so.
Writing 04_conditions found two validator defects, both of which would
have hit any user writing the documented conditional syntax:
- `{{ content | length > 10 }}` was reported as an unknown filter
called "length > 10". The filter name was taken as everything after
the pipe up to a parenthesis, so any operator following a filter
became part of its name. It is now the leading identifier.
- A condition referencing a pipeline *parameter* was reported as a
dependency on a non-existent task. The dependency validators only
ever received the task list, so they could not know that `content`
was a declared parameter rather than a missing step. Declared
parameter, input and loop-variable names are now passed in.
Neither fix moved the existing catalog: still 108 failing, now out of
116. Those failures are elsewhere, and this PR does not claim otherwise.
10_model_fallback from the planned set is deliberately absent. There is
no model fallback -- select_model raises rather than substituting a
model the pipeline did not ask for -- and an example must not be written
for behaviour that does not exist. The README records why, along with
the other planned-but-absent numbers, so the gaps read as deliberate.
Blocking suite 515 -> 541 passed, 13 skipped, 0 failed, 0 xfailed.
Collection 3310 -> 3336.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
CI 9/9 green. The legacy tally moved by +3, which is above the ±2 noise floor I had established, so I checked rather than dismissing it. It is the same flakiness family.
Evidence they are flaky rather than caused by this change:
Revising the noise floor upwards, againOn #443 I put the run-to-run swing at "at least ±2", based on two tests flipping. It is at least ±3, and the cluster is larger than the individual tests I had named — That makes the aggregate legacy count less useful than I have been treating it. The named |
Step 3 of the sequence. Partially addresses #104.
examples/holds 111 pipelines of which 3 compile. They are the project's primary documentation surface, so a reader has no way to tell which parts of the syntax are real.examples/supported/is the answer: a small set tested as product behaviour. Every file mustorchestrator validatetests/test_supported_examples.pyenforces all five and fails if a file is added without an entry declaring what it should do — so the set cannot grow past its coverage. 26 tests over 5 examples.01_hello_filesystem02_parallel_fanout_fanin04_conditionscondition:,on_falserouting, skipped steps06_failure_policytimeout, bounded retry,on_failure: continue, honest exit code07_templates_and_outputsEach exercises semantics pinned in #441–#444, which is why they can be real specs rather than aspirations.
06_failure_policyexits 1 on purpose. Not every supported example is a success case, and the expectations table records the intended exit code rather than assuming zero — a pipeline containing a failing step is supposed to say so.Writing 04_conditions found two validator defects
Both would hit any user writing the documented conditional syntax:
{{ content | length > 10 }}was reported as an unknown filter called"length > 10". The filter name was taken as everything after the pipe up to a parenthesis, so any operator following a filter became part of its name. It is now the leading identifier.A condition referencing a pipeline parameter was reported as a dependency on a non-existent task. The dependency validators only ever received the task list, so they could not know
contentwas a declared parameter rather than a missing step. Declared parameter, input and loop-variable names are now passed in.I found these only because I wrote an example using the syntax the docs describe — which is the argument for this directory existing.
What these fixes did not do
Neither moved the existing catalog: still 108 failing, now out of 116. Those failures are elsewhere. I am stating this explicitly because "fixed two validator bugs" invites the assumption that the catalog improved, and it did not.
Deliberate absences
The numbering follows the planned set, so gaps read as intentional:
03_data_etl,05_loop— the tools andfor_each/whileexist but are not yet pinned by contract tests.08,09,11— model pipelines; they cannot run hermetically in the blocking layer and belong in the live job.10_model_fallback— absent on purpose. There is no model fallback:select_modelraises rather than substituting a model the pipeline did not ask for (Pin timeout and retry semantics; establish that fallback fails closed #444). An example must not be written for behaviour that does not exist, and writing one would have quietly created pressure to implement silent model substitution — the exact thing the Dartmouth cost-control work exists to prevent.12–14— checkpoint/resume, sub-pipelines, web research; each needs its own acceptance job first.Not in this PR
The other 108 examples are untouched. Moving them to
experimental/andarchive/is mechanical but touches every docs link that references them, and it is a separate change from establishing the tested set. This PR makes the supported set exist; quarantining the rest is the follow-up.Numbers
uv lock --check/git diff --check/ docs drift