Repository navigation
My concern is what the person or organization relying on an AI-assisted operation can actually depend on when the workflow reports that its work is complete #4526
Replies: 2 comments
|
I appreciate the work going into AGT and the related discussions. I would like to make one part of this request more concrete: a supported evaluation toolkit that helps adopting organizations perform independent evaluation of the operation they are relying on. The organization would continue to define its purpose, authority, required conditions, and acceptable evidence. Microsoft could provide the reusable evidence interfaces, test cases, and working example that make those requirements testable in practice. The maintenance example above offers a manageable starting point:
The example should also let an evaluator establish whether the restriction actually took effect. A controller reporting that it blocked an action is itself a claim that needs appropriate evidence. Independent evaluation would therefore cover the connection between the organization’s requirement, the evidence, the evaluation, and actual operational behavior—including how the controller implements the rule. The evaluator needs a credible way to challenge the system’s account, using evidence beyond its own records where the claim requires it. The evaluation method should also be open to examination. For the operator, the result should make clear what has been established, what remains unresolved, which action is restricted, and what evidence or authorized decision is needed next. Could AGT provide or maintain a reference evaluation toolkit around this example, either directly or through a supported companion integration? It would need documented evidence requirements, coverage limits, expected results for successful and failed cases, and a way for another party to repeat the checks. That would give users and contributors a concrete basis for testing the complete connection—and give adopting organizations practical tools for carrying out the responsibility they retain. |
|
I am sharing this as an end user and operator. I am not selling or promoting a product. My purpose is to explain a necessary operational concept: people and organizations relying on AI-assisted work need a practical, supported way to evaluate it independently and connect those findings to their authorized decisions. For optional background, I have attached my Governing Plane overview, which sets out the reasoning behind this request. It explains how organizational requirements, observation, execution controls, independent evaluation, and accountable authorization fit together. A central principle is that every component whose account is relied upon—including the controller and the evaluation process—must remain open to examination. The practical question for this discussion remains the supported evaluation example described above. AGT could provide that capability through its own architecture, a companion toolkit, or a maintained integration. The papers offer context for evaluating whether those connections are sufficient. 02_Volume_1_Define_Reality.docx |
Uh oh!
There was an error while loading. Please reload this page.
I am approaching this as an end user and operator. My concern is what the person or organization relying on an AI-assisted operation can actually depend on when the workflow reports that its work is complete.
Several discussions here already address important parts of that question:
These discussions raise a shared operational need. I would like to ask the maintainers for a clear engineering decision on how that need will be supported.
The gap matters to the person relying on the operation
AGT’s limitations documentation acknowledges that recording attempted actions and policy decisions does not establish that an external objective was achieved. It also identifies post-action verification hooks and outcome attestation as planned work.
That is a useful acknowledgment. The operational problem remains until there is a dependable connection between the required condition, the evidence establishing it, and permission to proceed.
An agent can have the correct identity, pass every applicable configured control, perform the permitted tool call, and leave an accurate record. The organization may still lack adequate evidence that the information was suitable, the applicable authority supported the operation, the correct target was reached, or the required effect occurred.
Those questions are part of the operation the end user has to manage.
An ordinary maintenance example
Consider these five states:
Successful governance of the first three steps does not establish the last two.
If the required inspection evidence is missing, the system should preserve that fact and apply the organization’s rule. Return to service might remain on hold while further repair, inspection, evidence collection, or escalation continues.
The system does not need to invent an answer. It needs to prevent an unestablished condition from silently becoming the basis for the next consequential action.
An authorized exception should also remain identifiable as an exception. Approval to proceed under restricted conditions does not turn missing evidence into a verified result.
The supported capability being requested
Please provide a supported mechanism through which an adopting organization can bind its authorized operational requirements to controlled execution, required evidence, evaluation of that evidence, and the conditions under which subsequent action is permitted.
The organization would define what must be established, who holds authority, what evidence is acceptable, and what restrictions or exceptions apply. Microsoft could supply the reusable machinery that:
The evaluation determines what the evidence supports. The organization’s authorized rules determine what that permits.
The customer retaining responsibility does not mean every customer should have to design and maintain this connection from scratch.
A request for a clear maintainer response
The existing proposals provide concrete starting points. I would appreciate an answer to the following:
From the operator’s perspective, an assortment of useful components becomes dependable governance only when the necessary connections are made effective throughout the operation.
As agents gain more ability to act on an organization’s behalf, this needs to be treated as a deployment requirement. The need has already been identified here. I am asking for a clear decision and a supported implementation path.
All reactions