feat(skill): add run-assert-eval skill and eight worked example domains - #303
Open
alex ngo (ango10) wants to merge 77 commits into
Open
feat(skill): add run-assert-eval skill and eight worked example domains#303alex ngo (ango10) wants to merge 77 commits into
alex ngo (ango10) wants to merge 77 commits into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
run-assert-eval, a skill that drives the full Clarity → ASSERT → ACS loop from inside anMCP-capable IDE, plus eight worked example domains produced with it and a skill-first README entrypoint.
Motivation / linked issue
ASSERT's entry cost was writing an
eval_config.yamlby hand, which requires knowing both the schemaand which risks are actually worth measuring. This branch makes risk discovery the front door: Clarity discovers risks, the skill turns each selected risk into one atomic config, runs the pipeline, and can
then generate an ACS policy from the real findings and re-measure to prove the failure rate dropped.
Changes
Skill
.claude/skills/run-assert-eval/SKILL.md: canonical instructions. Clarity MCP (run_clarity) is a required front door. Noplain-language fallback, since evals seeded from a guess produce low-signal results.
measure-clarity-failures.md(discovery → measurement),govern-and-remeasure.md(ACS → governed run → delta),diagnose-acs-delta.md(symptom-indexedfallback for when the delta comes out wrong).
clarity_intake.pyparser, with unit tests and fixtures.SETUP-CHECKLIST.mdfor one-time MCP setup and end-to-end verification..github/prompts/) and Cursor (.cursor/rules/).Examples eight domains built end-to-end with the skill, each shipping
Clarity Protocol/(discovered risks),
evals/<risk>/eval_config.yaml(one atomic config per risk),agent.py, and aREADME:
azure_doc_qa,billing_support_agent,career_health_assessment,change_control_agent,prompt_agents,science_research_agent,travel_planner_langgraph,travel_planner_neurosan.the workflow, not a checked-in file.
Framework ACS CLI fixes
assert_ai/cli.py:assert-ai acs …now loads.envand refreshes Azure auth mode, soacs generateresolves credentials the same wayassert-ai rundoes. The ACS subcommandsdon't import the runner, so the project
.envwas previously never loaded.assert_ai/cli.py:acs validatenow explains that annotator-conditioned policies(
input.annotations.*) cannot fire in offline validation, so an unblocked/0-handled resultis expected rather than a policy defect.
assert_ai/integrations/acs/{prompt_builder,validate}.py: policy generation/validation fixes,with tests in
tests/test_acs_{prompt_builder,validate}.py.Docs
README.md: new skill-first Get started covering onboard → explore → use → ACS. Also repointsa quick-install command that broke when example configs moved under
evals/<risk>/.instead of the raw
policy_violation/overrefusalkeysTesting
assert-ai run→ judged artifacts underartifacts/results/). Testing matrix found here: https://onedrive.cloud.microsoft/:b:/a@tm94la6n/r/_layouts/15/onedrive.aspx?id=%2Fa%40tm94la6n%2FDocuments%2FClarityAssertAcsResultsV0%2E2Release%2FClarityAssertAcsV0%2E2Metrics%2Epdf&parent=%2Fa%40tm94la6n%2FDocuments%2FClarityAssertAcsResultsV0%2E2Release&share=cQrJBugNB1yZTrZSjrMxSfPfEgUCaq4K6MgvoSpLTJpXVJACgQChecklist
pytestand/or viewer checks as applicable).CHANGELOG.mdentry is included.