Skip to content

feat(hooks): add 'type: model' hook and integrate pre_tool_use into approval flow#2546

Merged
dgageot merged 1 commit intodocker:mainfrom
dgageot:board/llm-as-judge-for-auto-approving-tool-1563505f
Apr 28, 2026
Merged

feat(hooks): add 'type: model' hook and integrate pre_tool_use into approval flow#2546
dgageot merged 1 commit intodocker:mainfrom
dgageot:board/llm-as-judge-for-auto-approving-tool-1563505f

Conversation

@dgageot
Copy link
Copy Markdown
Member

@dgageot dgageot commented Apr 27, 2026

Summary

Add a first-class type: model hook handler so users can wire LLM-as-X hooks (judge, summarizer, classifier, …) entirely from YAML — no Go code, no shell glue, no jq+curl. Combined with the well-known pre_tool_use_decision schema and a small approval-flow change, it delivers a working LLM-as-a-judge that actually auto-approves tool calls.

hooks:
  pre_tool_use:
    - matcher: "shell|edit_file|mcp:.*"
      hooks:
        - type: model
          model: openai/gpt-4o-mini
          timeout: 15
          schema: pre_tool_use_decision
          prompt: |
            You are a security judge. Tool: {{ .ToolName }}
            Args: {{ .ToolInput | toJSON }}
            Reads under the working dir are safe. Writes to ~/.ssh are deny.
permissions:
  deny:  ["shell:cmd=*sudo*", "shell:cmd=*rm -rf*", "edit_file:path=/etc/*"]
  allow: ["read_*"]

That's it — the framework owns provider construction, streaming, structured output, JSON parsing, error fallback, and verdict shaping.

What's in the PR

1. New type: model hook framework

  • New HookTypeModel factory (pkg/hooks/model_handler.go) with a pluggable Backend/ModelClient seam, Go text/template prompt rendering exposing the hook Input, plus toJSON and truncate template helpers.
  • Pluggable ResponseShape registry: a shape interprets the model's reply as a hook Output, with an optional paired latest.StructuredOutput schema. The default shape passes the reply through as additional_context (turn_start summarizers, etc.).
  • New schema fields on HookDefinition: model, prompt, schema. Conditional validation: when type == "model", both model and prompt are required (enforced in the Go validator and agent-schema.json).
  • Runtime wiring: pkg/runtime/model_hook.go provides a real provider-backed ModelClient that builds a provider.Provider per call from a model spec, threads WithStructuredOutput when a schema is supplied, and streams the reply.

2. Well-known pre_tool_use_decision shape

Ships with the package (auto-registered):

  • Strict JSON schema ({decision: "allow"|"ask"|"deny", reason: string}) for providers that honor structured output (OpenAI, …).
  • Lenient JSON-in-text parser (handles markdown fences and surrounding prose) for providers that ignore the schema.
  • Output is shaped into a HookSpecificOutput.PermissionDecision verdict the runtime's approval flow understands.

3. Runtime: pre_tool_use becomes an active approval gate

The pre_tool_use hook used to fire inside runTool, after the user had already been asked. So a Deny wasted a click and an Allow had no effect on the prompt. This PR moves it into executeWithApproval:

1. --yolo                                → run
2. session/team permissions: deny        → block
3. session/team permissions: allow       → run
4. session/team permissions: forceAsk    → ask user
5. pre_tool_use hooks                    → Allow (run) | Ask (escalate) | Deny (block)   ← NEW
6. ReadOnlyHint                          → run
7. default                               → ask user
  • New Result.Decision / Result.DecisionReason fields, populated by a most-restrictive aggregator (Deny > Ask > Allow).
  • Hook UpdatedInput rewrites are applied before the verdict branch, so all downstream paths see the modified call.
  • Two new ApprovalSource constants — pre_tool_use_hook_allow and pre_tool_use_hook_deny — so the existing on_tool_approval_decision observer event fires for hook verdicts too. Telemetry / audit gets full visibility for free.

4. Tests (~30 new sub-tests)

File Covers
pkg/hooks/aggregate_test.go Most-restrictive ordering across multiple hooks; non-PreToolUse leaves Decision empty.
pkg/hooks/model_handler_test.go Factory validation (missing fields, bad template, unknown schema), prompt rendering, structured-output threading, lenient JSON parsing, error propagation, truncate helper, nil-client misuse.
pkg/runtime/pre_tool_use_approval_test.go End-to-end: Allow auto-approves without prompt; Deny blocks without prompt; Ask escalates to prompt; deterministic Deny beats hook Allow; deterministic Allow short-circuits the hook.
pkg/runtime/hooks_wiring_test.go The model factory is registered by NewLocalRuntime.

5. Docs / examples

  • examples/llm_judge.yaml — full working configuration showing the 3-layer pattern (deterministic permissions → LLM judge → user prompt).
  • docs/configuration/hooks/index.md — "LLM as a Judge" section rewritten around type: model, plus a security-considerations subsection.
  • agent-schema.json — new fields with if/then conditional validation for type: model.

Behavior change to flag

Pre-existing pre_tool_use shell-script hooks that returned Deny used to fire after the user had been prompted. They now fire before. This is strictly an improvement (no more wasted clicks on hooks that were going to deny) but is observable for any audit log keyed on hook-vs-prompt timing. Allow-style verdicts gain new effect: they previously did nothing; they now skip the user prompt.

Validation

  • golangci-lint run ./...0 issues
  • go test ./...all packages pass, no FAIL
  • Schema round-trip test (pkg/config) validates the new YAML shape.
  • go build ./... clean.

Stats

18 files changed, 1519 insertions(+), 37 deletions(-)


Built incrementally with docker agent from a blank slate ("how do I implement an LLM judge?") through to a shipped type: model framework with reviewer feedback applied. See commit message for details.

@dgageot dgageot requested a review from a team as a code owner April 27, 2026 16:17
trungutt
trungutt previously approved these changes Apr 28, 2026
@dgageot dgageot force-pushed the board/llm-as-judge-for-auto-approving-tool-1563505f branch 2 times, most recently from bbdfe37 to 5f3db5b Compare April 28, 2026 07:36
rumpl
rumpl previously approved these changes Apr 28, 2026
@dgageot dgageot force-pushed the board/llm-as-judge-for-auto-approving-tool-1563505f branch from 5f3db5b to a445589 Compare April 28, 2026 07:58
rumpl
rumpl previously approved these changes Apr 28, 2026
…pproval flow

Add a first-class `type: model` hook handler so users can wire LLM-as-X hooks (judge, summarizer, classifier, ...) entirely from YAML, with no Go code.

Highlights:

- New `HookTypeModel` factory in pkg/hooks with a Go text/template prompt, an injectable ModelClient, and a pluggable ResponseShape registry. Tests cover validation, templating, structured-output threading, and lenient JSON parsing.

- Well-known `pre_tool_use_decision` shape ships with the package: the model is asked for {decision, reason} (strict JSON schema for providers that honor it; tolerant parser otherwise), and the result becomes a permission_decision verdict.

- Runtime model_hook.go wires a real provider-backed ModelClient from a model spec (provider/model).

- Move pre_tool_use dispatch from runTool into executeWithApproval. Allow now auto-approves (skips user prompt), Ask escalates to it, Deny blocks BEFORE prompting (no more wasted clicks). New Result.Decision / DecisionReason carries the most-restrictive verdict (Deny > Ask > Allow). Deterministic permissions: rules still win, so the hook only sees the ambiguous middle.

- Schema & docs: new model/prompt/schema fields on HookDefinition with conditional 'if type == model' validation; LLM-as-a-judge doc section rewritten around type: model; examples/llm_judge.yaml migrated and slimmed down.

- Removed: pkg/hooks/builtins/llm_judge.go (~370 LOC) and its tests, superseded by the generic framework. The judge use case is now ~30 LOC of YAML.

Behavior change to flag in changelog: pre-existing pre_tool_use shell-script hooks fire BEFORE the user prompt instead of after. This is strictly an improvement (Deny no longer wastes a user click) but is observable for any hook that relied on running post-approval.

Assisted-By: docker-agent
@dgageot dgageot force-pushed the board/llm-as-judge-for-auto-approving-tool-1563505f branch from a445589 to 3a6990e Compare April 28, 2026 08:18
@dgageot dgageot merged commit 4ca9713 into docker:main Apr 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants