Repository navigation
AI Audit Evidence, Part 3: An Auditor Is Not an Admin #87
rain6fish
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
An auditor has to review what an AI did.
The cheapest way to let them do that is to hand them an admin account. But then the person they are supposed to review becomes themselves.
This piece is about building a read-only auditor role: its reach is an explicit list, every field is rebuilt from a whitelist, row scope comes from the account rather than the request, and the signed evidence stays whole and unprojected.
1. Admin is the cheapest answer, and the wrong one
An audit reader's job is to review what someone else did. They need evidence, not operational authority.
But in most systems read access follows write access: whoever can manage everything can obviously read everything. So the shortest path to "let them see the audit" is to give them
admin. The cost is that they also gain the ability to change policy, approve requests and execute tools, which is exactly what they came to review.What is needed is not a weaker admin. It is a role whose reach can be enumerated item by item. If you cannot enumerate it, it is still an admin.
2. The role's reach is a list
Our auditor role gets five rules, and every one grants
readonly:That list is everything the role can touch. Everything else is denied by the policy guard by default.
It deliberately gets none of the
USERownership rules. This role is a reader of the governance record, not a user of the application. It should have no write action either. The day one of these rules grows awrite, the role stops being an auditor.The shape of the rule is worth one sentence too. The underlying role seeds originally had only a subject and an owner field, and every seed implied
manage. We added an optionalactions; omitting it still meansmanage, so no existing role changed behaviour. Writing it is how a role narrows itself. Read access is written down, not inferred from management.3. Fields: rebuilt from a whitelist, not deleted from a full row
The role decides which surface you can reach. The field list decides what you can see on that surface.
We keep one whitelist and rebuild the projection from it:
That is not a style choice. It is a choice about defaults. Deleting from a full row defaults to visible; rebuilding from a whitelist defaults to invisible.
The fields left off the list have reasons. A conversation id is a pointer into a person's conversation. IP and device ids bind activity to a device or a person. A user's feedback about the assistant is not the audit record. Delegation context is free text that may quote a prompt. The bounded business intent and the agent and caller ids carry the attribution a reader actually needs.
One test injects a key that is not on the list at runtime and asserts it does not appear in the projection. It is not testing deletion. It is testing the default.
4. Rows: organisation from the account, period from the request
Beyond the role and the fields there is the question of which rows. The same surface, with the same fields, must still show different rows to different readers.
The by-id path and the list call the same scope function. Two implementations would eventually disagree.
5. Aggregates need the scope too, and so does the cache key
Narrowing the list while leaving the counts narrow-free is the same as not doing it.
The auditor's stats, cost, report and coverage statement all read through the same organisation scope. One thing is easy to miss: the aggregates are cached for 60 seconds, and the cache key has to carry the scope.
We hit that class of defect. With no scope in the key, the first reader's organisation answer was served to the next reader for a minute. The fix is not to clear the cache. The fix is to put the scope in the key:
all,org:<id>orown:<id>. It is the same class as a verification verdict computed for one database being served for another. A cache does not know who is asking, so the key has to.One thing we do not slice by organisation: the hash chain inside a signed evidence package. Slicing a chain and verifying the slice produces a false conclusion, so the chain keeps its full scope. Rows can be narrowed. A chain cannot.
6. What stays closed
Listing what the auditor can reach matters as much as listing what they still cannot:
Three points deserve their own paragraph:
userId; it does not apply the reader's account scope. This is an open item from the role's widening, and saying so here keeps the five rules from implying that every surface is already scoped.7. The scene: widening one surface at a time
Our first cut changed only the audit controller to
read AuditLog. Operation audit still judgedmanage all, so the auditor reached that surface and got a 403.That was not an omission. It was deliberate: open one surface, see whether it is enough, then widen one surface at a time. Later, operation audit, governance policy, tool inventory and the approval queue each got their own
readsubject. Writes, approvals and execution never entered the widening.The order has a second benefit. Each surface can be verified on its own: what should be visible is visible, what should be blocked is blocked. Opening everything at once makes a hole impossible to attribute.
8. Five checks you can run
Closing
Back to the opening: giving someone the evidence is not giving them the system.
What we do: five read-only rules define the role's entire reach; fields are rebuilt from a whitelist and new columns are invisible by default; organisation comes from the account and the period from the request; an out-of-scope row returns 404; aggregates and cache keys carry the same scope; signed evidence stays whole.
The mechanism above is taken from KeelBase, an open-source runtime that sits between an AI agent and a business system, re-checks identity, authorization, scope, risk and confirmation state before a tool runs, and writes every decision into a record that can be verified offline.
Repository: https://github.com/rain6fish/KeelBase
Four things stated plainly. The role is widened one surface at a time: everything outside the list is still unreachable, and that is default deny rather than unfinished work. Signed evidence cannot be projected, so privacy minimisation has an explicit boundary here. An auditor cannot decide approvals or serve as a flow approver. If they could, they would become the person they are reviewing. And operation audit's row scope is not narrowed by organisation yet: the read access is open, the scope is not, and that is an open item rather than a finished surface.
Read-only is not "a little less permission". It is a reach you can enumerate.
Drafted with an AI assistant. The system, the positions and the mistakes are mine.
All reactions