catalog secret material

Catalog registration refuses secret material

A catalog entry names a secret by reference — auth: "{{ db_credential }}", an alias the keychain resolves at step-execution time. The material itself lives in the keychain, envelope-sealed. POST /api/catalog/register enforces that split: an entry carrying recognisable secret material is refused.

Source: src/services/secret_material.rs, called from src/services/catalog.rs. Program: noetl/ai-meta#455 P5.

Why it is a hard refusal

Registration is the last point at which the answer can still be "no". The internal catalog is EHDB-backed and noetl.event is append-only, immutable and never purged, so a credential pasted inline into a playbook is not a mistake that can be cleaned up later — and it travels into command.issued when the step dispatches, which is the shape of the keychain drive leak.

⭐ The scan runs before the write. Scanning afterwards would persist the material and only then report an error; the log keeps it either way, so the order is the guarantee. A test asserts the call precedes queries::get_next_version, proven by mutation.

What fires

Only unambiguous markers:

marker	requires
PemBlock	-----BEGIN + PRIVATE KEY, or -----BEGIN CERTIFICATE
ServiceAccountJson	"private_key" and "client_email" and a -----BEGIN body
Jwt	eyJ-prefixed, three base64url segments, ≥80 chars
ProviderToken	AKIA/ASIA at exactly 20 chars; ghp_, github_pat_, xoxb-, xoxp-, sk-ant-, glpat-, AIza at the length that provider issues

No entropy heuristics, deliberately. A check that cries wolf is worse than no check because it teaches people to skim past the one finding that is real.

A value of the form {{ ... }} is checked first and never classified as material.

⚠⚠ The false positive that shaped the rule

Measured before shipping, against 570 YAML files (494 parsed, 26,081 string values) across noetl / travel / e2e / ops. It produced exactly one finding, and the finding was wrong:

ops/automation/agents/mcp/vertex-ai.yaml embeds Python that reads a service-account credential — info.get("private_key"), info.get("client_email") — so the code that handles a credential names both of its keys while containing no material at all. Requiring only the two key names would have broken a legitimate playbook's registration.

Requiring a PEM body fixed it: any genuine service-account key JSON carries its PEM inside private_key, so the extra condition costs no true positives. Re-measured over the same population: 0 findings in 26,081 values. The false positive is kept as a regression test.

⭐ And that fix initially made ServiceAccountJson unreachable — with a PEM required, the bare-PEM test, which ran first, shadowed it entirely. The specific check now runs first, because "a service-account key" is actionable in a way "a PEM block" is not. The test asserts the class is not PemBlock rather than merely is_some(); a plain is_some() would have passed on the shadowing value and hidden the dead class.

⚠⚠ The error never echoes the value

Findings carry the YAML path and the marker class — never the material, not even a 12-character prefix. A validator that quotes the offending string into its error writes the secret to the log, which is the thing it exists to prevent. Asserted directly.

The scan also publishes its denominator (scalars): a finding count without the population it came from is consistent with having examined nothing. That caught its own first measurement — a run against prod's 1527 catalog entries reported 0 findings with scalars_scanned=0, because /api/catalog/list omits content. Without the denominator that was a clean bill of health computed over zero bytes.

Scope

The scan walks the whole document, not secret-sounding keys. Material under description reaches the append-only log exactly as material under auth does, and a key-name allowlist would be a guess about which names are secret-bearing — the narrow-idiom mistake that made an earlier env read-set scan miss 96 of 152 variables.

If a legitimate entry is refused

The right fix is almost always to move the value into the keychain and reference it by alias. If an entry genuinely needs a material-shaped literal (a test fixture, an illustrative PEM in prose), that is a case worth discussing on the issue rather than loosening the marker — the measurement above is the baseline any loosening should be re-run against.

Related

• secrets and redaction — the response-boundary contract.
• runtime-registry — the sibling #455 work, and where the existence-vs-reachability lesson came from.