Skip to content

static guards

Jump to bottom
Kadyapam edited this page Oct 6, 2026 · 4 revisions

Static guards in noetl-server

A static guard here is a #[test] that reads the crate's own source as text and asserts a structural invariant — "every insert site also calls the observer", "every decision outcome has a pinned metric series", "the skip is reachable from a live call path". They exist because the defects this codebase actually ships are rarely wrong logic. They are mechanisms that exist and never run: an inert gate, a metric on a registry nothing scrapes, a wired function with no callers. A normal unit test cannot see that class, because the code under test is correct in isolation — what is wrong is that nothing reaches it.

This page records what each guard prevents, and the two text-scanning hazards that have produced false passes in this crate. Read this before deleting a guard that looks like noise. Every one of them was written after the thing it checks for had already happened at least once.

Why they are tests and not CI scripts

⚠ Corrected 2026-10-05. This section used to say no cargo test runs in CI on any NoETL Rust repo, citing noetl/ai-meta#232. That stopped being true: #232 was closed on 2026-10-02, and test.yml here runs cargo test --all-targets on every pull request — the guards below gate merges.

So they are tests rather than CI scripts for a better reason than "there is no runner": a guard that lives beside the thing it guards moves with it. A CI script reading src/ from outside drifts the moment a file is renamed, and nothing fails until someone notices the script matched zero sites.

playbooks/drift-audit.sh in ai-meta still mechanises some of the same classes, because it can see across repos and against the cluster, which a #[test] cannot.

The guards

services/certified_fold.rs — the chain-certified fold skip

guard prevents
every_event_insert_site_observes_the_chain An event-insert site that does not feed the certificate observer. The skip is only sound if the observer sees every event; a missed site silently diverges the digest. Compares INSERT-site count to observe( count per file, whole-file (see Truncation below).
the_fold_skip_is_reachable_from_a_live_call_path The #484 defect: the skip was wired into fold_from_postgres, which has no callers. The feature was merged, flag-gated, documented — and inert.
the_wired_skip_actually_avoids_the_second_rebuild The skip existing but not actually avoiding work.
the_flag_off_never_skips_through_the_wiring The flag failing to disable the path it gates.
the_rebuild_cache_registers_its_eviction_hook Deleting register_rebuild_evictor — which broke nothing observable until this guard existed. Unbounded cache growth under the flag.
every_decision_outcome_has_a_pinned_series A FoldDecision variant with no pinned metric. Prometheus prunes label families until they fire, so an unpinned outcome reads "absent" identically to "zero" — the exact reason the flag-on-but-inert state took an hour to diagnose on prod rather than a minute.
the_server_pins_the_series_at_startup The pinning function existing but never being called from main.
the_inert_vs_off_discriminator_is_exported Losing the ability to tell "flag off" from "flag on but inert" from a scrape alone.
the_runtime_proof_is_the_ramps_skipped_counter Treating observed>0 as success. The pass condition is observed>0 AND skipped incrementing; observed>0, skipped=0 is inert, which is a failure.
splitting_on_cfg_test_hides_code_that_whole_file_scanning_sees Reintroducing the truncation idiom in this file. See below.

services/catalog.rs — tool-kind validation

guard prevents
a_list_form_tool_is_validated_and_not_silently_skipped validate_tool_kinds validating an empty set. tool: is written as a sequence as well as a mapping, and the walker read only the mapping — so for the sequence form it collected nothing, the guard ran, returned successfully, and checked nothing.
a_playbook_mixing_both_tool_shapes_has_every_kind_collected A walker that handles one shape, or only the first item of a list. Asserts a count, which a rejection-only test cannot.

The shape that was invisible

serde_yaml::Value::get("kind") on a Sequence returns None, and the recursion cannot recover it: each sequence item is a {name, kind, …} mapping with no tool key of its own, so the push never fired.

Measured against the noetl/e2e fixture corpus — 166 files carrying a tool: key, which is the denominator:

files affected kinds missed
before 59 of 166 219 of 1,178
after 0 0 genuine

Shape distribution in that corpus: 20 sequence-only, 109 mapping-only, and 37 using both forms in one file — so the mixed case is ordinary, not contrived.

Hand-verified on fixtures/playbooks/vars_test/test_vars_block.yaml: five list-form blocks each carrying kind: python, from which the old walker extracted zero.

⚠ The probe's raw "after" figure is 3, not 0. All three are ground-truth over-counting — a kind: Playbook inside an HTTP prompt string and two inside Python code strings. They are text in string literals, not tool kinds. Recorded so nobody later chases 3 as a residual gap.

Consequence before the fix: a typo'd or unknown tool kind in list form registered successfully and failed only when the step ran — the precise failure noetl/ai-meta#256 exists to prevent, for the shape it did not cover.

The sequence arm labels each finding with the tool item's own name where present. Without it, a step with several tools reports every kind against the step name and the error does not say which tool was wrong.

db/queries/catalog.rs — the catalog listing shape

guard prevents
the_kind_filter_is_case_insensitive_on_both_the_column_and_the_parameter A case-sensitive kind filter. ⚠ The failure is a partial result, not an empty one — see below. Asserts LOWER(kind) = $1 across all four row-source shapes and that both bind sites take the lowercased value, because fixing either half alone is still wrong.
a_listing_selects_no_heavy_column_verbatim A listing paying for a body it discards. Asserts the property, not the column names already fixed.
latest_only_and_the_filters_apply_to_both_the_count_and_the_page The count and the page disagreeing about what "matching" means.

The partial-result failure mode

/api/catalog/list filtered kind case-sensitively while the column holds both spellings. Measured on prod 2026-10-06 across 1,527 rows:

kind stored rows
'Playbook' 875
'playbook' 650
'mcp' 2

So {"resource_type":"playbook"} returned 650 of 1,525 playbooks and "Playbook" returned 875.

⚠ Neither returned zero, and that is why it survived four months. An empty result gets reported as a bug; a 43%-complete one gets believed. Nothing in tests/ mentioned resource_type — 0 of 27 files — and the pre-existing shape guard asserted page.contains("kind = $1"), which pinned the buggy predicate as correct while passing Some("Playbook") to it.

Cause was two halves, each insufficient alone:

  1. register has lowercased unconditionally since f0301e86 (2026-06-09), but that fix normalised future writes and never backfilled, so earlier rows keep the capitalised spelling.
  2. The listing predicate bound the caller's string verbatim.

Fixing only the predicate makes a caller sending Playbook match nothing; fixing only the parameter still misses the 875 legacy rows. ingress.rs already had this right — WHERE LOWER(kind) = 'subscription' was the only LOWER(kind) in the tree.

⚠ If an index on kind is ever added it must be ON noetl.catalog (LOWER(kind)). There is none today — the table carries only the PK and UNIQUE (path, version) — so the function call costs nothing at present.

handlers/ehdb_eventlog_mirror.rs — the mirror and insert registry

guard prevents
every_event_insert_in_the_crate_mirrors_or_is_registered A new noetl.event writer that neither mirrors nor is explicitly registered as mirrored-by-caller. Holds assert_eq!(found.len(), 5) and files.len() > 50 so a broken walk fails instead of reading clean.
the_embedded_shadow_covers_every_site_the_mirror_covers The shadow and the mirror covering different site sets.
every_in_tx_event_insert_is_mirrored An in-transaction insert that skips the mirror.
the_chokepoint_mirrors_before_it_forks Mirroring after a branch, so one arm is missed.
mirror_rows_has_no_unaccounted_early_return A control-flow bypass — a text search cannot see an early return. This is the #486 lesson in guard form.
every_mirrored_insert_persists_what_it_mirrors / no_event_insert_persists_fewer_columns_than_the_row_carries / insert_column_and_bind_counts_agree A truncated row: an insert that drops a column the record carries (noetl/ai-meta#327).
gating_the_sink_mirror_did_not_break_the_insert_registry A feature gate silently emptying the registry.
the_guards_truncation_point_hides_no_production_code The precondition the two count-comparing guards above depend on. See below.
every_send_error_kind_is_pinned_at_zero An unpinned error label — same absent-is-not-zero hazard as above.

handlers/ehdb_projection_serve.rs + handlers/ehdb_embedded.rs — pinned label sets

Added 2026-10-05 (noetl/ai-meta#415).

guard prevents
every_refuse_reason_is_pinned_and_distinct a RefuseReason variant whose label is not in PROJECTION_SERVE_REFUSALS, so its series stays absent while the other four read 0
every_shadow_outcome_is_pinned_and_nothing_is_pinned_that_cannot_fire the same for ShadowVerdict plus the three labels callers pass as literals (opened, open_failed, append_failed)
every_read_outcome_is_pinned_and_nothing_is_pinned_that_cannot_fire the same for ReadVerdict, plus a source assertion that the verify handler still records engine_unavailable

Each uses a total match, so a new variant fails the build here rather than quietly adding an unpinned label. Each also asserts the reverse direction — nothing pinned that no caller can produce.

⭐ The reverse direction found a live bug on its first run

EMBEDDED_READ_OUTCOMES pins four values. ReadVerdict::label() produces three. The fourth, engine_unavailable, existed in exactly two places: that const, and a JSON response body in ehdb_embedded_verify. That arm returned early without touching the metric, and the only other call site passes verdict.label(), which cannot produce it.

So noetl_ehdb_embedded_read_total{outcome="engine_unavailable"} was registered, pinned at 0, and could never increment — telling anyone who read it that the embedded engine was always available, while the condition it names did happen (flag off, or the open failed).

A pinned series nothing can increment is worse than an absent one: absence at least prompts the question. That is why the reverse assertion is not symmetry for its own sake, and why the read guard reaches into the handler's source — what regressed is a missing call, and only a call-site assertion can see a missing call.

Hazard 3 — a mutation battery where no mutant was applied

The six-mutation battery for the guards above reported 5 of 5 FIRES on its first run and was worthless. The shell helper never forwarded its arguments to python3 -c, so no mutation was applied; the non-zero exits came from a sed that changed an array's declared length without changing its contents, i.e. from E0308.

Two later mutants had the same shape in reverse — value removed, length left alone.

A mutant that fails to COMPILE proves nothing about the guard. A battery here must:

  • assert the mutation applied (the applier prints MUTATED, or the case is skipped and reported as not-applied);
  • count compile errors separately and require 0;
  • establish a GREEN baseline first, one filter at a time — cargo test --lib a b c is a usage error, not three tests, and reads as a failing baseline, which makes every mutant look caught.

Hazard 1 — split("#[cfg(test)]") truncation

Several guards extract a file's production region as src.split("#[cfg(test)]").next() before counting. Two separate ways this produces a confident wrong answer:

It truncates at the first occurrence, which may be an inline attribute. Not a trailing test module. An #[cfg(test)] on a single item near the top of a file cuts almost everything away, and the guard then reports clean over a nearly empty slice.

Everything below the cut is invisible, including production code. Whether that matters depends on the assertion's shape, and it is worth checking per guard rather than assuming:

  • An assertion of the form contains(needle) fails safe. Truncation can only remove a needle, which makes the test stricter, never falsely satisfied. The MIRRORED_BY_CALLER leg and gating_the_sink_mirror_did_not_break_the_insert_registry are both this shape and are sound. They were reviewed and deliberately left unchanged.
  • An assertion that compares two counts within the slice can false-pass. A site hidden below the cut is absent from both sides, so the equality holds. the_embedded_shadow_covers_every_site_the_mirror_covers is this shape.

Demonstrated 2026-10-02 by planting an unmirrored production INSERT INTO noetl.event below handlers/events.rs's test modules:

the_embedded_shadow_covers_every_site_the_mirror_covers   2 passed   <- blind
every_event_insert_in_the_crate_mirrors_or_is_registered  1 passed   <- blind
the_guards_truncation_point_hides_no_production_code      1 FAILED   <- catches it

Both existing guards would have let that through. In events.rs the invisible region is lines 2103–6341 — 67% of the file — and both noetl/ai-meta#263 and noetl/ai-meta#332 were new direct-write sites in exactly that file.

Why the truncation is kept anyway

It is load-bearing in ehdb_eventlog_mirror.rs. That file carries 7 INSERT INTO noetl.event strings in its own tests, all 7 below its cut. Scanning it whole-file would make the crate walk flag the file itself as an unregistered writer. Measured: all five production files have 0 needles below their cut; this file has 7 of 7 below, correctly.

So the fix is a precondition on the extraction, not a rewrite of it: the_guards_truncation_point_hides_no_production_code scans the five production files the weak pair walks — handlers/events.rs, handlers/event_write.rs, handlers/internal.rs, db/queries/event.rs, services/internal.rs, and explicitly not ehdb_eventlog_mirror.rs — and asserts no needle sits below the first #[cfg(test)].

It fails closed in both edge cases, deliberately:

  • An early inline #[cfg(test)] moves the cut up, so more code looks "below" it. Stricter, not looser.
  • A production file with no test module is skipped before checked += 1, so assert_eq!(checked, scanned.len()) fails and a human re-reads it rather than the guard silently measuring nothing.

Prefer whole-file scanning where the file permits it. The coverage guard in services/certified_fold.rs scans whole-file and catches the same plant (3 INSERT site(s) but 2 observe call(s)), so it is strictly stronger here.

Hazard 2 — a guard that passes vacuously

A filter that matches zero tests exits 0. The amd64 Cloud Build reported OK this way: the test filter matched nothing and the step was green. When running a guard by name, check the count, not the exit code:

cargo test --locked --lib <filter>
# "test result: ok. 0 passed; ... 1177 filtered out"  <- this is a VACUOUS pass

Use --lib to target one binary; a bare cargo test <filter> prints a test result: line per test binary, and the last one is usually an integration binary where the filter matches nothing. Guards that scan a population should assert the population size (assert_eq!(checked, scanned.len(), ..), assert!(files.len() > 50)) so a broken scan fails rather than reads clean — the "print the denominator" rule from agents/rules/representation-drift.md.

Working on these guards

  • RED before GREEN, always. Plant the defect the guard exists to catch, confirm the guard fails, restore, confirm it passes. A guard that has never once fired is indistinguishable from a guard that cannot fire.
  • A plant must compile. Planting an inline #[cfg(test)] inside real code to demonstrate the truncation hazard does not compile, so it proves nothing. Use a synthetic-source comparison for that case.
  • Never run a blanket cargo fmt. It has twice churned unrelated files in this repo, and once split .merge(x.layer( across lines so every_privileged_router_is_gated_in_main reported 12 routers ungated. Nothing was ungated — the guard failed closed, correctly. Use rustfmt --edition 2021 --check <file> on the file you touched.

Hazard 4 — a guard that counts its own assertion messages

A source-scanning guard that reads its whole file with include_str! and counts a needle will also count the needle inside its own assertion messages, because those messages name it verbatim.

Concretely, from the catalog kind-filter guard:

let src = include_str!("catalog.rs");                 // the WHOLE file
let n = src.matches("kind_lower.as_deref()").count();
assert_eq!(n, 2, "...found {n} uses of kind_lower.as_deref()");
//                                       ^^^^^^^^^^^^^^^^^^^^ counted too

It reported 4 uses where the code has 2 and failed with the fix correctly in place. The symptom is the opposite of the usual one: not a false pass, but a false failure that invites "relaxing" a correct assertion until it agrees with the miscount.

The fix is to slice, and to slice with rfind. Cutting at the last #[cfg(test)] rather than the first, because catalog.rs has an earlier #[cfg(test)] block with real code after it — so Hazard 1's split(..).next() idiom would discard part of the region being measured and still report a plausible number. Then assert the slice is plausibly large before using it:

let whole = include_str!("catalog.rs");
let code = &whole[..whole.rfind("#[cfg(test)]").expect("test module")];
assert!(code.len() > 10_000, "slice is {} of {} bytes", code.len(), whole.len());

This is Hazard 2's lesson applied one level up: assert the extraction before asserting about it. A guard measuring the wrong region is the same defect whether the region is empty, truncated, or includes the guard itself.

Related

History

Created 2026-10-02 alongside noetl/server#490, which reviewed the four split("#[cfg(test)]") uses in ehdb_eventlog_mirror.rs and found two sound (fail-safe contains) and two with a latent false pass.

noetl-server

Reference

Design

Operations

Related repos

External

Clone this wiki locally