Skip to content

Tell the planner which edges the policy denies, before it wastes a round on one - #88

Merged
Shashankss1205 merged 2 commits into
mainfrom
fix/issue-45
Aug 4, 2026
Merged

Tell the planner which edges the policy denies, before it wastes a round on one#88
Shashankss1205 merged 2 commits into
mainfrom
fix/issue-45

Conversation

@Shashankss1205

Copy link
Copy Markdown
Collaborator

Fixes #45

What was wrong

DEFAULT_PLANNER_SYSTEM_PROMPT states the catalog, the START/END literals and the structural rules — and its own comments explain the philosophy: "stating the rule up front is cheaper than three wasted rounds." The one rule models actually trip over was never stated. Observed with qwen3:8b against the incident registry: the goal said "find the cause and propose a fix", the policy denied *->deploy, and the planner proposed an edge into deploy in all three rounds (edge_denied; edge_denied + cycle; edge_denied) until the loop stopped admission_refused — roughly 3.5 minutes of local inference. The refusal came back every round, and edge_denied names the check, not the rule: "no edge may enter deploy, ever" was never on the page.

What this does

Both halves of the issue, because they are the same information arriving at two different moments and each is weak alone — the disclosure prevents the first wasted round, the enriched refusal tells a model that ignored it how wide the denial is.

  1. Disclosure. EdgePolicy.disclosure() and NodePolicy.disclosure() render a policy's deny rules one line each. PlannerNode(edge_policy=…, node_policy=…) puts them directly under the catalog — the other half of the same statement: here is what exists, here is what may not be wired. The shipped builders (examples.plan_incident.build_loop, stdlib.build_loop, and therefore grapharc plan) hand the planner the same policy object the checker holds, so the prompt cannot describe a policy the gate is not applying.

    Allow rules and the default are left out, per the issue — they say what is permitted, which the catalog covers. ask is left out too, for a different reason: its remedy is to obtain an approval, not to propose something else, so "do not propose them" would be the wrong sentence.

    Node denials are included (A policy's resource = "node" rules were compiled by nobody, so a denied kind ran #77's NodePolicy), rendering as nodes of kind 'deploy' are denied by policy — do not propose them. A registered-but-denied kind is exactly the shape that reads as an invitation.

  2. Refusal feedback. EdgeRule gains the reason field NodeRule already had, PolicyEngine.edge_policy() compiles it out of the document instead of dropping it, and policy/edge_denied quotes it. The generated document's "Deploy changes are dangerous…" now reaches the planner instead of dying at compile time.

Enforcement is unchanged — explicitly

The admission gate is byte-identical in behaviour. No check consults a disclosure; PlannerNode still decides nothing and holds the policies only to print them; the only change inside AdmissionChecker._check_policy is that the detail string may carry the rule's reason (decide() was refactored to rule_for() + .action, same tiering, same answer). A model that ignores the disclosure — or was never given one — is refused by the same code, with the same codes, subjects and remedies.

That is pinned twice:

  • test_disclosure_does_not_move_the_enforcement_into_the_prompt — two planners, one shown the deny rule and one not, produce the same proposal; the gate's rejections are compared field by field (model_dump()), and must be equal.
  • test_the_shipped_loop_discloses_its_edge_policy_to_the_planner — the shipped demo, wired as grapharc plan wires it. The scripted round 1 still proposes the denied deploy and is still refused edge_denied; round 2 replans and the run reaches goal_met. grapharc plan output is unchanged, matching the README transcript.

Before / after

The system prompt grapharc plan hands the model, tail end. Before:

Available node kinds:
- deploy: push to production
- patch: write a fix for the cause
- triage: classify the incident and gather context
- verify: prove the fix works against evidence

After:

Available node kinds:
- deploy: push to production
- patch: write a fix for the cause
- triage: classify the incident and gather context
- verify: prove the fix works against evidence

Denied by policy. The admission checker refuses these; it is deterministic code and this list is only telling you in advance:
- edges into 'deploy' are denied by policy — do not propose them

With a policy document that carried a reason, the line becomes:

- edges into 'deploy' are denied by policy — do not propose them: Deploy changes are dangerous and require a human operator

and the same words reach the refusal:

- [policy/edge_denied] triage -> deploy: the edge policy denies this transition: kind 'triage' (proposed as 'triage') -> kind 'deploy' (proposed as 'deploy'): Deploy changes are dangerous and require a human operator the decision is made on the registry kind, not the name you chose: renaming the node will not change it — route through a permitted kind

Tests

New, and each confirmed failing before the change:

  • tests/test_admission.py — the deny rule is in the prompt handed to the model; the reason travels with it; node denials are disclosed beside edge denials; a policy that denies nothing adds nothing (prompt byte-identical to before); every rule shape renders (into / out of / from…to / glob / catch-all), ask and allow do not; duplicates collapse; edge_denied quotes the reason; enforcement unchanged.
  • tests/test_planner_loop.py — the shipped loop discloses its policy and still refuses round 1; disclosed and undisclosed runs produce identical rejections.
  • tests/test_policy_engine.py — the document's reason survives compilation into EdgeRule and reaches the disclosure.

1861 passed, 12 deselected; ruff check grapharc tests clean.

docs/cookbook/05-governance.md gains a runnable snippet (executed and diffed by test_cookbook_governance.py like every other one) next to the paragraph that already says the prompt is a courtesy and the gate is the enforcement. CHANGELOG entry added under ## Unreleased.

🤖 Generated with Claude Code

Shashankss1205 and others added 2 commits August 5, 2026 00:29
…und on one

The planner's system prompt names the catalog, the START/END literals and the
structural rules, and its own comments say why: stating a rule up front is
cheaper than three wasted rounds. The rule models actually trip over was the
one it never stated. Observed with qwen3:8b against the incident registry —
the policy denied `*->deploy`, and the planner proposed an edge into `deploy`
in all three rounds (`edge_denied`; `edge_denied` + `cycle`; `edge_denied`)
until the loop stopped `admission_refused`. About 3.5 minutes of local
inference spent discovering one sentence. The refusal came back every round,
and `edge_denied` names the check, not the rule: "no edge may enter `deploy`,
ever" was never on the page.

`EdgePolicy.disclosure()` and `NodePolicy.disclosure()` render a policy's deny
rules one line each — `edges into 'deploy' are denied by policy — do not
propose them` — and `PlannerNode(edge_policy=..., node_policy=...)` puts them
directly under the catalog, which is the other half of the same statement:
here is what exists, here is what may not be wired. The shipped loop builders
hand the planner the same policy *object* the checker holds, so the prompt
cannot describe a policy the gate is not applying. Allow rules and the default
are left out — they say what is permitted, which the catalog covers — and so
is `ask`, whose remedy is an approval rather than a different proposal.

The refusal side is enriched to match. `EdgeRule` carries the `reason`
`NodeRule` already had, `PolicyEngine.edge_policy()` compiles it out of the
document instead of dropping it, and `policy/edge_denied` quotes it, so a
planner reads why and not only what.

None of this is enforcement. No check consults a disclosure, the admission
gate is byte-identical, and a model that ignores what it was told is refused
exactly as one that was never told — pinned by a test comparing the rejections
of a disclosed and an undisclosed planner field by field, and by the shipped
demo, whose scripted round 1 still proposes the denied deploy and is still
refused.

Fixes #45

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Shashankss1205
Shashankss1205 merged commit 62a1832 into main Aug 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

planner: the system prompt withholds the edge policy, so models burn rounds proposing denied edges

1 participant