Direction proposal: error fidelity as the seventh guardrail property

[aanishs]

Proposing a direction rather than a single fix, so posting here instead of the tracker — the first concrete issue is filed (#74), and the rest follow individually if this resonates.

The $conformance scorecard grades six properties — PHI redaction, immutable audit, step-up auth, human-in-the-loop, tenant isolation, medical disclaimers. The probes verify those protections on successful responses; none of them probes what the layer does when the data layer fails. Today the stack converts errors into confident wrong answers at three independent layers:

Layer	Behavior today	Where
Upstream proxy	Any non-200 on search → {total: 0, entry: []}; any non-200/404 on read → None → facade reports 404 not found. An upstream 400 OperationOutcome, an expired OAuth token, and a genuinely empty record are indistinguishable.	r6/fhir_proxy.py L170–198
Local search	Only 7 params are read; anything else is silently ignored — Observation?date=ge2024 returns unfiltered results. The self link (routes.py L901–909) correctly echoes only applied params, but there's no warning entry, no strict mode, and the self link is skipped for _summary=count, omits context-id, and doesn't URL-encode.	r6/routes.py L734+, L901
MCP orchestrator (HTTP/SSE)	The read-path methods flatten non-OK responses to {error: "search failed with status 400"}, discarding the OperationOutcome the Flask layer produced; the HTTP and SSE transports never set isError and surface uncaught exceptions as JSON-RPC -32603 — while the stdio transport already does both correctly, so the transports disagree.	services/agent-orchestrator/src/tools.ts, index.ts

The compounding effect: a rejected upstream query flows through as a 200 empty bundle, fhir_search tells the agent "No Observation resources found matching criteria" in _mcp_summary, and the audit trail records search (upstream): 0 results for a search that never executed. For an agent, "no data" and "your query was rejected" require opposite reactions — and for a patient-facing deployment, "you have no observations" is a clinical statement the layer should never fabricate.

None of this needs invented machinery — FHIR R4 search already specifies the error contract (Prefer: handling=strict as a client request servers SHOULD honor; the self-link SHALL; search.mode="outcome" warning entries; SHALL-reject for unsupported modifiers), and MCP classifies tool failures as results with isError: true (which stdio here already emits).

The concrete work, if the direction lands

Each of these would be its own tracked issue (per the roadmap's "issue tracker is the canonical log" convention), and I'm offering the PRs:

1. Upstream proxy: propagate errors instead of empty bundles / fake 404s — the bug above; surfaced as sanitized allowlist OperationOutcomes (raw passthrough would leak; _rewrite_urls isn't sanitization), audit outcome='failure'. Filed: Upstream proxy converts FHIR errors into empty search results (and auth failures into 404s) #74.
2. MCP HTTP/SSE transports: isError + sanitized OperationOutcome bodies, aligning with stdio's existing behavior via a transport-layer adapter.
3. Local search: honor Prefer: handling=strict, emit lenient-mode warning entries, harden the self link, reject unsupported modifiers — with semantically safe suggestions only (never datetime → _lastUpdated; modification time isn't clinical time).
4. Make action_policy.yaml load-bearing, then expose it as a describe contract — today the YAML isn't loaded at runtime, isn't in the orchestrator image, and contradicts enforcement (e.g. fhir_propose_write); corrective errors should point at a contract the runtime actually obeys.
5. Grade it: "error fidelity" $conformance probes, split into a local profile (deterministic everywhere) and a mock-upstream proxy profile — so the property is verifiable the same way the other six are, and a graded F is on the board even before the fixes land.

Deliberate limits: param-level correction only (no StructureDefinition-level errors, consistent with Known Limitations); no changes to write semantics, step-up, or redaction; ignore-by-default stays for callers that don't opt into strict, so nothing breaks.

Context: this grew out of medplum/medplum#9616 — agents recover from structured errors far better than from silent or transport failures — applied to the guardrail layer itself. "Verifiable, not marketing" is the best line in this repo; this direction extends it to the failure paths. Does this fit where you want the project to go?

[aks129]

This is one of the best contributions this project has had — and yes, it lands. Error fidelity is the seventh guardrail property. You've articulated something I felt but hadn't named: the six properties all probe what the layer protects on the success path, and none of them probe what it does when the data layer fails. A guardrail layer that turns a rejected query into a confident "you have no observations" is failing at its actual job, and "verifiable, not marketing" has to extend to the failure paths or it's only half true.

The framing that sold me: for an agent, "no data" and "your query was rejected" require opposite reactions, and for a patient-facing deployment "you have no observations" is a clinical statement the layer must never fabricate. That's exactly right, and it's the kind of thing that's invisible until someone names it.

On the concrete plan — I'm in on all five, in your suggested order:

1. Upstream proxy — merged (fix(proxy): surface upstream FHIR errors instead of empty results and fake 404s #81). The sanitized-allowlist OperationOutcome approach (not raw passthrough, since free text leaks) was the right call, and the tests that plant a patient name + internal hostname and assert neither leaks are exactly the evidence that makes this reviewable. That's piece one of the seventh property already on main.
2. MCP HTTP/SSE transports → isError + sanitized bodies, aligned with stdio. Yes. The transports disagreeing is a real bug, and stdio already showing the correct shape means we have a reference implementation in-repo. A transport-layer adapter is the right level.
3. Local search: Prefer: handling=strict, lenient-mode warning entries, hardened self link, reject unsupported modifiers. Yes — and your "semantically safe suggestions only (never datetime → _lastUpdated, modification time isn't clinical time)" constraint is the kind of judgment that makes me trust the rest. Ignore-by-default staying the default is correct; nothing should break for callers who don't opt into strict.
4. Make action_policy.yaml load-bearing, then expose it as a describe contract. Strong agree, and this is the sharpest one: a corrective error that points at a contract the runtime doesn't actually obey is worse than no contract. Making it load-bearing first, then describable, is the right sequence.
5. Grade it — split into a deterministic local profile and a mock-upstream proxy profile, so a graded F is on the board before the fixes land. This is the part I love most: it makes the property real the same way the other six are real, and it means we can watch the grade climb as the PRs merge. Put the probes in the same harness (r6/conformance/) so $conformance emits it as property 7.

How I'd like to run it, if you're up for it: file the four remaining items as tracked issues (per the "issue tracker is the canonical log" convention). I'll label a couple of them help wanted so others can join — the transport adapter and the search-strict work are both scoped enough for another contributor to take a slice while you drive the conformance profiles, which are the load-bearing part. And I'd like to credit you as the author of the property in the docs when it lands — this is your design, not just your code.

One bigger-picture note: this is also exactly the kind of thing that makes the project matter outside itself. "Agents recover from structured errors far better than from silent failures" (your Medplum #9616 lineage) is a claim the whole agent-on-FHIR ecosystem needs to internalize, and a graded, verifiable error-fidelity property is how you make it undeniable. That's a contribution to the field, not just the repo.

Thank you for leading with a direction instead of a drive-by fix. Tag me on the issues as you file them and let's build it. 🙏

[aanishs]

Thank you, I’m fully in. I’ll file the four remaining issues in the agreed order and tag you on each. I’m happy to drive the conformance profiles/property-seven work, while leaving the transport adapter and strict-search slices open for help wanted contributors. Really appreciate the trust and the offer to credit the property’s authorship—let’s build it.

[aanishs]

Status: both bookends are on main. #81 landed the proxy fix (item 1), and #108 landed the grading (item 5) — $conformance now reports error fidelity as property seven. It landed with the honest baseline on the board — B (6/7), error fidelity F — and #117 then implemented the local strict-handling behavior against those probes and took the board back to Grade A (7/7): the grade-climbs-as-fixes-merge loop working exactly as designed, faster than I expected.

On the filed issues: #117 covers the core of #94, so I'll follow up there to reframe what remains (self-link hardening and the search-parameter registry); #93 (transport adapter) is still open and scoped for the help wanted label whenever you want to add it; and I'm picking up #95 (action_policy) next.