Tell the planner which edges the policy denies, before it wastes a round on one - #88
Merged
Conversation
…und on one The planner's system prompt names the catalog, the START/END literals and the structural rules, and its own comments say why: stating a rule up front is cheaper than three wasted rounds. The rule models actually trip over was the one it never stated. Observed with qwen3:8b against the incident registry — the policy denied `*->deploy`, and the planner proposed an edge into `deploy` in all three rounds (`edge_denied`; `edge_denied` + `cycle`; `edge_denied`) until the loop stopped `admission_refused`. About 3.5 minutes of local inference spent discovering one sentence. The refusal came back every round, and `edge_denied` names the check, not the rule: "no edge may enter `deploy`, ever" was never on the page. `EdgePolicy.disclosure()` and `NodePolicy.disclosure()` render a policy's deny rules one line each — `edges into 'deploy' are denied by policy — do not propose them` — and `PlannerNode(edge_policy=..., node_policy=...)` puts them directly under the catalog, which is the other half of the same statement: here is what exists, here is what may not be wired. The shipped loop builders hand the planner the same policy *object* the checker holds, so the prompt cannot describe a policy the gate is not applying. Allow rules and the default are left out — they say what is permitted, which the catalog covers — and so is `ask`, whose remedy is an approval rather than a different proposal. The refusal side is enriched to match. `EdgeRule` carries the `reason` `NodeRule` already had, `PolicyEngine.edge_policy()` compiles it out of the document instead of dropping it, and `policy/edge_denied` quotes it, so a planner reads why and not only what. None of this is enforcement. No check consults a disclosure, the admission gate is byte-identical, and a model that ignores what it was told is refused exactly as one that was never told — pinned by a test comparing the rejections of a disclosed and an undisclosed planner field by field, and by the shipped demo, whose scripted round 1 still proposes the denied deploy and is still refused. Fixes #45 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #45
What was wrong
DEFAULT_PLANNER_SYSTEM_PROMPTstates the catalog, the START/END literals and the structural rules — and its own comments explain the philosophy: "stating the rule up front is cheaper than three wasted rounds." The one rule models actually trip over was never stated. Observed with qwen3:8b against the incident registry: the goal said "find the cause and propose a fix", the policy denied*->deploy, and the planner proposed an edge intodeployin all three rounds (edge_denied;edge_denied+cycle;edge_denied) until the loop stoppedadmission_refused— roughly 3.5 minutes of local inference. The refusal came back every round, andedge_deniednames the check, not the rule: "no edge may enterdeploy, ever" was never on the page.What this does
Both halves of the issue, because they are the same information arriving at two different moments and each is weak alone — the disclosure prevents the first wasted round, the enriched refusal tells a model that ignored it how wide the denial is.
Disclosure.
EdgePolicy.disclosure()andNodePolicy.disclosure()render a policy'sdenyrules one line each.PlannerNode(edge_policy=…, node_policy=…)puts them directly under the catalog — the other half of the same statement: here is what exists, here is what may not be wired. The shipped builders (examples.plan_incident.build_loop,stdlib.build_loop, and thereforegrapharc plan) hand the planner the same policy object the checker holds, so the prompt cannot describe a policy the gate is not applying.Allow rules and the default are left out, per the issue — they say what is permitted, which the catalog covers.
askis left out too, for a different reason: its remedy is to obtain an approval, not to propose something else, so "do not propose them" would be the wrong sentence.Node denials are included (A policy's
resource = "node"rules were compiled by nobody, so a denied kind ran #77'sNodePolicy), rendering asnodes of kind 'deploy' are denied by policy — do not propose them. A registered-but-denied kind is exactly the shape that reads as an invitation.Refusal feedback.
EdgeRulegains thereasonfieldNodeRulealready had,PolicyEngine.edge_policy()compiles it out of the document instead of dropping it, andpolicy/edge_deniedquotes it. The generated document's"Deploy changes are dangerous…"now reaches the planner instead of dying at compile time.Enforcement is unchanged — explicitly
The admission gate is byte-identical in behaviour. No check consults a disclosure;
PlannerNodestill decides nothing and holds the policies only to print them; the only change insideAdmissionChecker._check_policyis that the detail string may carry the rule'sreason(decide()was refactored torule_for()+.action, same tiering, same answer). A model that ignores the disclosure — or was never given one — is refused by the same code, with the same codes, subjects and remedies.That is pinned twice:
test_disclosure_does_not_move_the_enforcement_into_the_prompt— two planners, one shown the deny rule and one not, produce the same proposal; the gate's rejections are compared field by field (model_dump()), and must be equal.test_the_shipped_loop_discloses_its_edge_policy_to_the_planner— the shipped demo, wired asgrapharc planwires it. The scripted round 1 still proposes the denieddeployand is still refusededge_denied; round 2 replans and the run reachesgoal_met.grapharc planoutput is unchanged, matching the README transcript.Before / after
The system prompt
grapharc planhands the model, tail end. Before:After:
With a policy document that carried a
reason, the line becomes:and the same words reach the refusal:
Tests
New, and each confirmed failing before the change:
tests/test_admission.py— the deny rule is in the prompt handed to the model; thereasontravels with it; node denials are disclosed beside edge denials; a policy that denies nothing adds nothing (prompt byte-identical to before); every rule shape renders (into/out of/from…to/ glob / catch-all),askandallowdo not; duplicates collapse;edge_deniedquotes the reason; enforcement unchanged.tests/test_planner_loop.py— the shipped loop discloses its policy and still refuses round 1; disclosed and undisclosed runs produce identical rejections.tests/test_policy_engine.py— the document'sreasonsurvives compilation intoEdgeRuleand reaches the disclosure.1861 passed, 12 deselected;ruff check grapharc testsclean.docs/cookbook/05-governance.mdgains a runnable snippet (executed and diffed bytest_cookbook_governance.pylike every other one) next to the paragraph that already says the prompt is a courtesy and the gate is the enforcement. CHANGELOG entry added under## Unreleased.🤖 Generated with Claude Code