Repository navigation
AI Audit Evidence, Part 5: Five Sources in One File Do Not Check Each Other #90
rain6fish
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Has the same conclusion been answered by two paths that share nothing?
Our evidence package puts five sources into one file: the authorization snapshot, the decision evidence, the audit row, the side-effect row and the operation audit row. A root digest pins them together. The package verifies offline and the signature matches.
What that proves is that the five were signed together. It does not prove that they said anything to each other. A digest only answers whether the group was swapped out.
One sentence: cross verification is not "find more sources". It is one conclusion answered independently by two non-sharing paths, with the system itself able to notice when they disagree.
If that is enough, the rest is a table, a code fragment and three admissions.
1. What counts, and what is only packaging
Three criteria. Miss one and it does not count.
By those criteria, bundling five sources and hashing the bundle is packaging. Its value is real: the bundle cannot be altered without breaking the hash. But "cannot be altered" answers "was it changed", not "do the parts corroborate each other". Those two questions get conflated because both travel under the word integrity.
2. Our own account: two and a half
Out of five conclusions, one meets all three criteria.
The authorization row is half. A recomputation does happen, but for one of five checks. A reader sees "authorization was reviewed" and what was reviewed is a single item.
Read that list as three grades: complete, meaning a second source that is actually compared; half, meaning a second source with one item compared; and none, meaning no second source at all.
There is a fourth grade that is not on the list, and it costs the most: both sources are present and nobody compares them.
We run two chains, an AI audit chain and an operation audit chain, and the same request lands in both. The material is there, the write points do not overlap, and it looks exactly like two independent paths. But no line of code pairs them by action. They are juxtaposed rather than corroborating.
It is not on the list of five because we never treated "the two chains describe the same event" as a conclusion the package has to answer. The five are what the package claims. This one nobody claimed. The expensive kind of false cross is not missing material. It is material that nobody remembers to compare.
3. Why the one that counts, counts
Independence: one side is what this call said it would write, the other is what the ledger already recorded. They are produced in different places, so an error in one does not automatically appear in the other.
Detectability: a mismatch becomes a state, disputed, with the difference kept as evidence. Not two fields waiting to be noticed.
The cost, stated plainly: it triggers in one situation, when a retry with the same arguments hits the idempotency key.
4. Something a reader asked out of us
This section is about us.
We published a five-layer evidence model: that a record exists, its source and integrity, its link to execution, coverage, and the verification of the result. A reader replied that a layer was missing. Some claims have no record at all. "This system complies with X", with nothing attached.
He was right, and it was worse than he put it. Our first layer says that a record exists is not the same as the event happened, and it presupposes that a record exists. A claim with nothing attached sits outside the model: it neither satisfies nor violates it. It cannot enter.
We added the layer, numbered L0. To be explicit: it was added after publication, and the published version does not contain it. If you go back to the article, you will not find it there.
That exchange was itself a cross check. Our own conclusion was asked again by someone who does not share our view, in a different form, and the different form found the gap. The same question in the same wording returns the same blind spot.
5. Four checks you can run
Closing
Back to the file. It can prove it was not altered. It cannot prove the parts corroborate each other.
To prove that, each conclusion has to name its second source, and the ones that cannot get labelled single source. The list will look bad. It will at least be accurate.
The mechanism above is taken from KeelBase, an open-source runtime that sits between an AI agent and a business system, re-checks identity, authorization, scope, risk and confirmation state before a tool runs, and writes every decision into a record that can be verified offline. Cross verification is a way of reading here, not a shipped feature; KeelBase is its implementation and evidence carrier.
Repository: https://github.com/rain6fish/KeelBase
Three things stated plainly. One of five conclusions is a complete cross check; the rest are half or nothing. The authorization row reviews a single item out of five. And L0 was added after publication, so anyone citing it should carry that sentence along.
Drafted with an AI assistant. The system, the positions and the mistakes are mine.
All reactions