Skip to content

Add Phase 9 orchestration evaluation quality gates#1040

Merged
paullizer merged 1 commit into
feature/orchestrationfrom
feature/orchestration-phase-09-evaluation-quality-gates
Jul 15, 2026
Merged

Add Phase 9 orchestration evaluation quality gates#1040
paullizer merged 1 commit into
feature/orchestrationfrom
feature/orchestration-phase-09-evaluation-quality-gates

Conversation

@paullizer

Copy link
Copy Markdown
Contributor

Summary

Implements Phase 9 of chat orchestration as an evaluation and quality-gate slice on top of the merged Phase 8A/8B contracts.

  • Adds six deterministic golden scenarios covering M365 profile plus selected image, SQL metrics, missing public-profile evidence, selected-image Q&A, current Fairfax rules, and governed-agent evidence-only central synthesis.
  • Adds fixed-schema privacy-safe evaluation events for terminal run/source outcomes, recommendation creation/decision/outcome, latency, citation yield, missing evidence, and unsupported-fact counts.
  • Adds a repeatable repository-local quality-gate runner and an opt-in five-scenario controlled live-smoke manifest/result contract.
  • Repairs orchestration-adjacent stale version, Blueprint, helper-fixture, and false-green test assertions.
  • Updates the application version, feature documentation, and release notes to 0.250.068.

Privacy And Governance

  • Hashes run correlation values and reports discovered agents only as governed_agent.
  • Allowlists capability classes, reason codes, statuses, and task profiles.
  • Excludes prompts, evidence/response text, canonical agent and object IDs, private scope details, inaccessible counts, endpoints, secrets, hidden tools, action names, and action arguments from evaluation events.
  • Leaves existing authorization, exact source-turn revalidation, ETag decision/resume idempotency, process-restart reconciliation, and central finalization behavior unchanged.

Validation

  • 203 passed, 5 skipped using scripts/run_phase9_orchestration_quality_gates.py.
  • The five skips are explicitly environment-gated browser/live scenarios.
  • All changed and new Python files passed py_compile.
  • Full-file Broken Access Control guardrail passed for all 4 changed application Python files.
  • Full-file XSS sink guardrail passed for all 4 changed application Python files.
  • Route-policy suites, diagnostics, and tracked/untracked whitespace checks passed.
  • Controlled live smoke was not executed because no explicit live manifest/auth configuration was supplied; its local contract test passed and live execution remains opt-in.

Scope Boundary

Phase 10 remains responsible for generalized output types, broader multi-agent execution, Foundry or action-attached discovery, consequential/write approval, durable in-flight execution, and complex workflows.

Refs #1021

@paullizer
paullizer merged commit 1f36f56 into feature/orchestration Jul 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant