What happens when the system of record is stale or unreachable — hold, or allow? #19
Unanswered
Nisarg-Vibhakar
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Been reading through the adapters ahead of v1. The allow/hold/block decision assumes the system of record answers. I'm curious what the intended behaviour is when it doesn't, and whether that's configurable per adapter.
Three cases that seem relevant:
1. Unreachable. The Postgres/HTTP adapter times out. Does the verdict default to
hold(fail closed) orallow(fail open)? Fail-open is the tempting default because it keeps agents moving, but it makes ActionRail silently absent exactly when the underlying system is having a bad day — which is when a wrong tool call is most likely.2. Stale but reachable. A read replica answers in 5ms with data from 40 seconds ago. The check passes, the argument was correct 40 seconds ago, and the action is wrong now. This one worries me more than the timeout, because nothing looks broken. Is there a freshness/lag signal an adapter can surface, or is that the integrator's problem?
3. Partially available. Two of three adapters answer. Is a verdict issued on partial evidence, or held?
Context for why I care: I built a grounded retrieval layer for a state emergency-operations platform where the model had to cite a source or abstain, and the hardest failure was never "no answer" — it was a confident answer from a section that had been superseded. Abstention is easy to state and hard to calibrate; the stale band is where it actually gets decided.
Happy to write the stale-replica case up as a test if that's useful for v1.
All reactions