Skip to content

feat(skill): add run-assert-eval skill and eight worked example domains - #303

Open
alex ngo (ango10) wants to merge 77 commits into
mainfrom
ango10/assert-acs-skill-main
Open

feat(skill): add run-assert-eval skill and eight worked example domains#303
alex ngo (ango10) wants to merge 77 commits into
mainfrom
ango10/assert-acs-skill-main

Conversation

@ango10

Copy link
Copy Markdown
Collaborator

Summary

Adds run-assert-eval, a skill that drives the full Clarity → ASSERT → ACS loop from inside an
MCP-capable IDE, plus eight worked example domains produced with it and a skill-first README entrypoint.

Motivation / linked issue

ASSERT's entry cost was writing an eval_config.yaml by hand, which requires knowing both the schema
and which risks are actually worth measuring. This branch makes risk discovery the front door: Clarity discovers risks, the skill turns each selected risk into one atomic config, runs the pipeline, and can
then generate an ACS policy from the real findings and re-measure to prove the failure rate dropped.

Changes

Skill .claude/skills/run-assert-eval/

  • SKILL.md: canonical instructions. Clarity MCP (run_clarity) is a required front door. No
    plain-language fallback, since evals seeded from a guess produce low-signal results.
  • Three workflows: measure-clarity-failures.md (discovery → measurement),
    govern-and-remeasure.md (ACS → governed run → delta), diagnose-acs-delta.md (symptom-indexed
    fallback for when the delta comes out wrong).
  • clarity_intake.py parser, with unit tests and fixtures.
  • SETUP-CHECKLIST.md for one-time MCP setup and end-to-end verification.
  • Mirrored for Copilot (.github/prompts/) and Cursor (.cursor/rules/).

Examples eight domains built end-to-end with the skill, each shipping Clarity Protocol/
(discovered risks), evals/<risk>/eval_config.yaml (one atomic config per risk), agent.py, and a
README: azure_doc_qa, billing_support_agent, career_health_assessment, change_control_agent,
prompt_agents, science_research_agent, travel_planner_langgraph, travel_planner_neurosan.

  • Skill-generated ACS artifacts are intentionally stripped: the governed entrypoint is an output of
    the workflow, not a checked-in file.

Framework ACS CLI fixes

  • assert_ai/cli.py: assert-ai acs … now loads .env and refreshes Azure auth mode, so
    acs generate resolves credentials the same way assert-ai run does. The ACS subcommands
    don't import the runner, so the project .env was previously never loaded.
  • assert_ai/cli.py: acs validate now explains that annotator-conditioned policies
    (input.annotations.*) cannot fire in offline validation, so an unblocked/0-handled result
    is expected rather than a policy defect.
  • assert_ai/integrations/acs/{prompt_builder,validate}.py: policy generation/validation fixes,
    with tests in tests/test_acs_{prompt_builder,validate}.py.

Docs

  • Root README.md: new skill-first Get started covering onboard → explore → use → ACS. Also repoints
    a quick-install command that broke when example configs moved under evals/<risk>/.
  • Example READMEs now headline Impermissible Behavior violated / Permissible Behavior violated
    instead of the raw policy_violation / overrefusal keys

Testing

Checklist

  • Tests pass locally (pytest and/or viewer checks as applicable).
  • Docs updated if behavior or public API changed.
  • No secrets, credentials, or customer data committed.
  • No breaking change, or a CHANGELOG.md entry is included.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants