Repository navigation
AI Audit Evidence, Part 4: What Can One AI Audit Record Actually Prove? #88
rain6fish
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
An audit record is on the table: one confirmation, one execution, one side effect. The hash chain verifies.
What does that prove?
One sentence: one record does not prove all five layers: a valid signature reaches only the second, and coverage and the external result are separate checks.
Most people read it as two things: the event happened, and the record is true. There are five different questions inside that sentence, and one record usually answers only one or two of them.
This piece gives a five-layer evidence model. If you only want the conclusion, you can stop here. The rest is the failure modes and the five claims behind that sentence, and you can skip it. The bold sentence in each section carries the point.
1. Five questions, not five fields
A record can answer five questions, and they are not the same thing. Before all five there is one entry question:
It is not one of the five, but without it none of the other five can be asked. Its most common form is not a blank page. It is a statement: "this system complies with X", with nothing attached. The correct conclusion there is
unknown, notfailed, because you cannot tell whether nothing was recorded or nothing was kept. Reading it as "fine" treats the absence of evidence as evidence.The five layers themselves:
The layers build on each other, but they do not upgrade automatically. A higher layer cannot be inferred from a lower layer's fields. A signature does not add coverage. A coverage statement does not verify an external result.
2. What one real record looks like
This is a record from a local run on 2026-10-15. The tool is
create_followup_task, risk level R3, approved from the pending list:The three rows together are "one operation".
But there is a non-obvious detail: audit row #27 has
toolandoutcomeempty. The tool name exists only inside thedetailstring, and the result can only be inferred fromis_error=0. A later row in the same database (#37, later that day) hastool=create_followup_taskandoutcome=executed. The same kind of record can look different at different times.The record is not false. It is authentic, on the chain, and the hashes are continuous. It simply does not explain itself. A reader who does not know the
detailconvention cannot say what ran or what the result was.3. Reading the record layer by layer
hashandprev_hash. It does not prove the tail was not deleted or that nothing was omitted. The confirmation and side-effect rows are not on this chain.executed, and the effect iscrm_task:10. It does not prove this effect was produced by this confirmation; the link is by tool, argument hash and time, not one foreign key.after_snapshotis present. It does not make the change comparable, becausebefore_snapshotis null, and it does not prove the external target's state.The strongest layer in this record is layer 2. The weakest is layer 5. It is a real record, not a complete piece of evidence.
4. Two counterexamples
Counterexample A: a valid signature over an incomplete record.
Audit row #27 is on the chain, with continuous
hashandprev_hash. But itstoolandoutcomeare empty, and the side-effect row has nobefore_snapshot. So:A signature authenticates bytes, not meaning. A real, signed record can still fail to answer the question it appears to answer.
Counterexample B: a complete record with an unknown business result.
This is a design-level scenario, not a test of the record above. A write is sent to an external system. The local side-effect row says
executed, but the external system is unreachable and has no query endpoint. The correct conclusion isunknown, notsuccess.A complete local record only says the local side is complete. An unverified external state cannot be written as success.
5. Not a field checklist
The five layers are not a field checklist. They are five refutable claims, preceded by one entry claim:
These claims do not try to prove that AI never makes mistakes. They ask which guarantees a system can offer when AI misreads, overreaches, retries, times out or hits an exception, and how far each guarantee can be verified.
They also explain why adding fields does not add guarantees: a signature at L2 does not make coverage at L4 true, and coverage at L4 does not make an external result at L5 verified. Each layer needs its own evidence.
The test of these claims is not more terminology. The test is whether they can point at a counterexample, narrow a guarantee, or turn a vague promise into a verifiable criterion. All five are candidates, not proven theorems.
6. What we have, and what we do not
What exists: the confirmation's execution axis (
decided_at,execution_claimed_at,executed_at,execution_error); the audit hash chain; side-effect rows with before and after snapshots; the coverage statement; the audit reader role.Boundaries: the record above already shows three. Structured fields can be empty. A side-effect row can lack a before snapshot. The confirmation and side-effect rows are not on the audit chain. Beyond that: without an external anchor, tail completeness can only be
unknown; an unreachable external result isunknown; the operation audit row scope is not yet narrowed by organisation.What is not done: the five-layer model is not an API or a product feature. It is a way to read evidence, which you can apply to any agent runtime without waiting for us to ship five endpoints.
7. Five checks you can run
unknown, absent, or pretended complete?unknown?Closing
Back to the opening: what a record proves depends on which layer you are asking about.
What we do: split evidence into five layers and say what each cannot prove; show a real record where "authentic but incomplete" is visible; keep external results and tail completeness as
unknowninstead of writing them as success.The mechanism above is taken from KeelBase, an open-source runtime that sits between an AI agent and a business system, re-checks identity, authorization, scope, risk and confirmation state before a tool runs, and writes every decision into a record that can be verified offline. The five-layer model is a way to read evidence, not a shipped feature; KeelBase is its implementation and evidence carrier.
Repository: https://github.com/rain6fish/KeelBase
Four things stated plainly. The record above comes from a local run, not production data. The five-layer model is a method, not a shipped feature. External results and tail completeness remain
unknown, and the operation audit row scope is still an open item. The five layers are candidates, and counterexamples are welcome.An honest record does not claim all five layers are full. It says where each layer stops.
Drafted with an AI assistant. The system, the positions and the mistakes are mine.
All reactions