Changliu2/cleanup mike acs demo - #199
Merged
changliu2 merged 53 commits intoJun 2, 2026
Merged
Conversation
…ates docs: normalize commands and path separators
The compare page derived a short label from `run.model` by splitting on `/` only (lines 301, 376). When the target is a Python callable like `examples.bank_manager_demo.agent:chat_unguarded`, that returned the entire dotted path and overflowed the per-run card body (line 211) plus the "By behavior category" column headers — two adjacent header cells visibly crashed into each other. Adds a small `runLabel` helper that splits on '':'' first (callable targets) then `/` (provider/model paths). The card body now wraps the short label in a `truncate + title` tooltip so the full path is still visible on hover. Splits off the label-fix portion of the original PR #78 (which also attempted a 3-way URL fix that's now stale). The 3-way URL plumbing is out of scope here and can ship in a separate PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… default model) + consolidate YAML emission rules (#176) * init: add --default-model CLI flag and surface design-agent model to LLM - New --default-model option on `assert-eval init` lets the user pre-seed the pipeline.default_model hint for the interview. - _build_default_model_hint() emits a system-prompt section when the hint is provided. - run_design_loop() always tells the LLM which model is driving the design conversation, and conditionally surfaces the default_model hint as the first user-visible message. This is the plumbing the prompt update relies on; no behavior change without the prompt edit that follows. * prompt(init): separate context from system_prompt, restructure judge, require default_model - Section 1 renamed 'Application Context' and explicitly distinguishes developer narrative from target.system_prompt for hosted-model targets. - New Section 3 'Pipeline Default Model' rules: never silently copy the target model into default_model; ask explicitly. - Renumbered Behavior (4), Test Set (5), Judge (6). - Judge built-ins: policy_violation, overrefusal. - Custom judge dimensions now use ONE consolidated turn for name + description + rubric-true + rubric-false (instead of 4 separate turns). - Added explicit guideline against conflating context with target.system_prompt. * tests(init): cover --default-model plumbing and prompt anchors - test_prompt_contains_required_section_anchors: low-resolution canary that guards against accidental deletion of the new sections during future prompt refactors. Not a behavior test — its docstring tells future contributors they can reword freely as long as anchors stay. - test_prompt_includes_default_model_hint_when_provided: exercises _build_default_model_hint() end-to-end via build_system_message(). - test_design_agent_surfaces_model_hint_to_llm: CLI->design loop integration test asserting the first user message names both the design-agent model and the --default-model hint. * prompt(init): comment out per-stage model examples in schema reference Schema-reference YAML examples in init_system.md were showing live 'model:' blocks under systematize, test_set.prompt, test_set.scenario, inference.tester, and judge. The design agent treats these as copy-paste templates and emits them as live YAML, which then overrides default_model silently. Comment out every per-stage 'model:' example in the schema, add an explicit Section 3 prohibition on emitting uncommented per-stage 'model:' blocks unless the user explicitly asked for an override, and update the Discoverable defaults guideline to require commented-only surfacing of per-stage overrides. * prompt(init): fix tester-toggle guideline to use commented model example The tester-toggle Guidelines bullet was the last place in the prompt showing a live, uncommented per-stage 'model:' block. Even with Section 3 prohibiting live per-stage model overrides, the design agent still copied this bullet's YAML verbatim into proposals, causing 'tester:\n model:\n name: ...' to leak into generated configs. Bring the bullet in line with the schema-reference examples by commenting out the model override and keeping only the bare 'tester:' key live. * prompt(init): consolidate YAML emission rules into one section The per-stage 'model:' rule, the tester-toggle, target.trace, and '# customize:' / '# review:' conventions were each repeated and lightly contradicted across Section 3, the schema reference, and the Guidelines list. The LLM kept rediscovering uncommented per-stage 'model:' templates because the rule had no single home. Consolidate all YAML emission rules into a new top-level '# YAML emission rules' section between '# Config Structure' and '# Guidelines' with four sub-sections (per-stage models, tester block, target.trace, customization hints). Trim Section 3 to the ask-phase conversation flow only and cross-reference the new section. Drop the duplicated tester/target.trace/customization bullets from Guidelines. Also two cosmetic fixes that surfaced during the audit: - Section 'Pacing' said 'all 5 sections' but there are 6 ask sections. - 'Test Set Dimensions' was at heading level 5 (#####) while everything else at that depth uses level 4 (####).
Renames the package directory and updates the five distribution-name and module-name references in pyproject.toml (name, [project.scripts], all extra, dev dependency-group, and setuptools packages.find). Also drops examples* from the wheel per the PyPI publishing plan. Imports inside the package are not yet updated -- follow-up commit.
Replaces assert_eval / assert-eval references inside the moved package so it imports cleanly. Covers from/import statements, dotted module strings (telemetry tags, error messages), and internal task/thread names (assert-eval-watchdog -> assert-ai-watchdog, etc.). Tests, docs, scripts, and CI updated separately.
Replaces assert_eval / assert-eval references in tests/ — module patches, dotted import paths, CLI invocations, and any expected log/identifier strings.
Catches the depth-1 .py files in assert_ai/ that the package-imports commit missed: cli.py, config.py, display.py, results.py, runner.py, viewer_read_model.py. Same dual-pattern (assert_eval -> assert_ai, assert-eval -> assert-ai).
Sweeps assert_eval -> assert_ai and assert-eval -> assert-ai across user-facing surfaces: root README and AGENTS, docs/ tree, runnable examples (including notebooks and agent scripts), helper scripts, the SvelteKit viewer, the marketing website, and the analysis README that ships inside the wheel.
Path filter in .github/workflows/regression.yml now points at assert_ai/** instead of assert_eval/**.
Renames assert_eval_policy.* -> assert_ai_policy.* so runtime policy artifacts stay ignored after the package rename.
Hand-edits the 4 self-references to the root project in uv.lock to match the renamed distribution. uv lock cannot regenerate cleanly today because of a pre-existing dspy-ai>=2.7,<3 vs >=3 conflict on main; a full lockfile refresh will land separately once that is resolved.
…SSERT (#178) The microsoft/adaptive-eval slug 404s after the rename to ASSERT. Two customer-facing docs still pointed at the dead URL: - CONTRIBUTING.md: dev-setup clone snippets (bash + powershell) - AGENTS.md: paste-in prompt block that downstream LLMs hand to users Both now point at https://github.com/responsibleai/ASSERT. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The CLI entry point in pyproject.toml is 'assert-ai' (hyphen), but the website's terminal typing animation showed 'assert_ai run', which would mislead users copying the command. Reported by Copilot review on PR #177.
The previous include list was ["assert_eval*", "examples*"]; the rename commit accidentally dropped 'examples*'. Tests import from 'examples.agents.*' (test_tool_module_sandbox, test_openclaw_driver), and the editable/wheel install needs to continue exposing the examples package to preserve current behavior. Packaging cleanup (e.g. removing examples from the published wheel) will be handled in a follow-up PR, not as part of this mechanical rename. Reported by Copilot review on PR #177.
….png README.md was updated to reference assets/assert-ai-framework-diagram.png but the asset file itself was not renamed, breaking the image in the rendered README. Reported by Copilot review on PR #177.
Website: partner quotes, expanded framework logos, docs structure polish
…package Move the prompt template directory inside the package so it ships inside the wheel as data alongside the importable code. Pure file move plus an empty __init__.py so importlib.resources can discover it as a real subpackage. The next commit switches io.py to load via importlib.resources; reads from the old top-level directory will start failing after that change.
The previous implementation derived the prompts directory from
`Path(__file__).resolve().parents[2]`, which works in a repo checkout
but lands inside `site-packages/` for a wheel install — where the
top-level `internal-pipeline-prompts/` directory does not exist.
Switch to `importlib.resources.files('assert_ai.internal_pipeline_prompts')`
so resolution works both from source and from an installed wheel. The
returned Traversable still supports `/`, `.read_text()`, `.read_bytes()`,
and `.is_file()`, which is everything the existing call sites need.
`artifact_cache._prompt_descriptor` is updated to use `.is_file()`
instead of `.exists()` since Traversable does not guarantee `.exists()`.
…t-ai Rename assert-eval -> assert-ai
`examples*` was previously included in `packages.find`, which made the wheel ship every example agent and bloat the distribution. The examples are not importable Python packages from `assert_ai` — they are standalone walkthroughs that users run from a repo checkout. They should not be in the wheel.
…package-data
Adds the metadata PyPI surfaces on the project page and uses for
discovery / filtering, plus the package-data declarations needed to
ship the prompt templates inside the wheel:
- `license = { file = 'LICENSE' }` (MIT, already at repo root)
- `authors` with the team display name; email is left as a TODO until
a public contact alias is available (does not block publishing)
- `keywords` (safety, evaluation, llm, agent, responsible-ai)
- `classifiers` covering Development Status (Beta), MIT license,
Python 3.11/3.12/3.13, and AI / QA / Testing topics
- `[project.urls]` (Homepage, Repository, Issues, Documentation)
- `[tool.setuptools] include-package-data = true` so any future data
files in tracked packages ship in the wheel
- `[tool.setuptools.package-data]` explicitly listing
`assert_ai.internal_pipeline_prompts = ['*.md']`. Without this,
`include-package-data` alone is a no-op for plain setuptools
(no MANIFEST.in, no setuptools-scm), and the .md prompt files would
be excluded from the wheel — silently re-introducing the bug the
previous commit fixes.
…oke)
`build.yml` runs on every push to main, every PR to main, and on
`workflow_dispatch`. Two jobs:
1. `build` (ubuntu-latest, Python 3.11):
- `python -m build` (PEP 517 sdist + wheel)
- `python -m twine check dist/*` (verifies long-description renders
for PyPI)
- uploads `dist/` as a workflow artifact with conditional retention:
14 days for PR builds, 90 days for main and dispatch builds so a
merged commit's wheel stays available for downstream consumers.
2. `test-install` (3x3 matrix: ubuntu / macos / windows x Python
3.11 / 3.12 / 3.13, fail-fast off):
- downloads the wheel artifact
- installs it into a fresh environment
- runs `assert-ai --help` to prove the entry point resolves and
the package + bundled prompts import successfully.
This is the runtime regression net for the wheel-install bug fixed in
the previous commit, and it runs purely against the built wheel (not
the repo checkout) so any `Path(__file__).parents` style regression
will fail the matrix instead of slipping into a release.
`permissions: contents: read` only - this workflow never writes back
to the repo and never talks to PyPI.
Clean break to match the package name (assert-ai). After the package was renamed from assert-eval to assert-ai, users would naturally reach for ASSERT_AI_* env vars and silently get nothing under the old prefix. Renamed (no compatibility shim): - assert_ai/cli.py: Click auto_envvar_prefix ASSERT_EVAL -> ASSERT_AI (so e.g. ASSERT_AI_CONFIG=... wires the --config flag) - viewer/src/lib/server/run-spawn.ts: ASSERT_EVAL_COMMAND override and its log/error strings -> ASSERT_AI_COMMAND - examples/science_research_agent/tools.py and README.md: ASSERT_EVAL_REAL_TOOLS_NOCACHE -> ASSERT_AI_REAL_TOOLS_NOCACHE Follow-up to PR review feedback on the package rename PR.
The example agents read ASSERT_AZURE_DEPLOYMENT and ASSERT_TARGET_MODEL, but three READMEs still documented the legacy P2M_* names from before the package rename. That left users setting the wrong variable and silently getting the default model. Aligns README docs with the code: - examples/travel_planner_langgraph/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT (inline comment + var table) - examples/phoenix_auto_trace/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT - examples/travel_planner_neurosan/README.md: P2M_TARGET_MODEL -> ASSERT_TARGET_MODEL Docs-only; no behavior change. tests/test_no_p2m_references.py still passes (the guard uses \bp2m\b which doesn't catch P2M_* tokens).
…dening build: wheel-safe packaging, CI build workflow, env-var rename
…185) Documents the assert_eval -> assert_ai package/CLI rename (PR #177) and the ASSERT_EVAL_* -> ASSERT_AI_* env var rename (PR #182) so existing preview users have a clear migration path. Keep a Changelog 1.1.0 format for CHANGELOG.md. README gets a short [!IMPORTANT] callout near the top. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d to AssertAi* (#187) Catches up TS symbol names with PR #177 (assert_eval -> assert_ai package rename) and PR #182 (ASSERT_EVAL_* -> ASSERT_AI_* env var rename). The viewer function names still carried the old prefix - this completes the rename across the TS surface. No behavior change. Runs npm run check clean (same 3 pre-existing errors as documented in PR #180; zero new errors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…badges) (#186) Bundle D for public-preview readiness. Adds: - .github/ISSUE_TEMPLATE/{bug_report.yml,feature_request.yml,config.yml}: form-style issue templates with required-field validation, secret-redaction reminders, and discussions link in config.yml. - .github/PULL_REQUEST_TEMPLATE.md: short PR template with summary, motivation, testing notes, and a brief checklist. - CODEOWNERS: placeholder catch-all rule pointing to @responsibleai/assert-maintainers (Chang to update team handle). - README badges: CI build status (from PR #182's build.yml workflow), supported Python versions (3.11/3.12/3.13 matching CI matrix), license. PyPI badge intentionally omitted until the package is published. No code change. No CHANGELOG.md (parallel PR delivers that). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pts/ (#188) Customer-facing terminology cleanup (M-3 from prior audit). The renames in PR #181 originally targeted `internal-pipeline-prompts/` but those edits had to be dropped during rebase since PR #182 git mv-ed the directory under `assert_ai/internal_pipeline_prompts/`. Re-applying against the new path. Scope: prompt markdown files only. Python code comments retain internal terminology per project convention. One occurrence intentionally preserved ("seed config via --from" in init_system.md L436 — refers to a starter config, not a test case). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
) * tests: tighten p2m guard regex and skip binary/lockfile false positives The previous \\b p2m \b\ word-boundary regex missed env-var leaks like P2M_AZURE_DEPLOYMENT because the underscore broke the word boundary. Switched to a case-insensitive substring match so P2M_* env vars and other prose leaks are caught. To keep the test signal clean, also skip: - Binary file extensions (.svg, image formats) that may embed base64. - Lockfiles (package-lock.json, yarn.lock, pnpm-lock.yaml, uv.lock, poetry.lock) where sha512 hashes coincidentally contain 'p2m'. Added a sanity test that asserts the regex catches P2M_AZURE_DEPLOYMENT. Deleted an orphan website/public/icons/P2M Thumbnail.svg (no references). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/docs: replace internal model names with public substitutes Replaced internal-only model names with publicly available substitutes across examples, docs, and the website snippets so customers can run copy-paste configs without hitting unknown-model errors. - azure/gpt-5.4-mini -> azure/gpt-4o-mini - azure/gpt-5.4-nano -> azure/gpt-4o-mini - azure/gpt-5.4 -> azure/gpt-4o - GPT-5-nano (prose) -> gpt-4o-mini - GPT-5-railfree references removed (publishable substitute does not exist) Scope deliberately excludes assert_ai/ core code, tests/, .github/ workflows, and artifacts/results/** (frozen historical records). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * AGENTS.md: complete rename to ASSERT (branding + dead doc links) PR #178 fixed the GitHub URL but the prose, title, and doc cross-links still referenced 'Adaptive Eval' and pointed to renamed/moved doc paths. - AGENTS.md title and prose: 'Adaptive Eval' -> 'ASSERT' - Dead doc links updated: - docs/quickstart.md -> docs/getting-started.md - docs/writing-eval-specs.md -> docs/guides/create-evaluation.md - docs/reading-results.md -> docs/guides/results.md - Same branding fix applied to .cursorrules, .devcontainer/devcontainer.json, CONTRIBUTING.md, SUPPORT.md, and examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/change_control_agent: genericize internal Microsoft infra references The change_control_agent example referenced internal Microsoft deployment/ops systems (Safefly, Ev2, R2D, ICM, ADO, ChangeKeep) which are unfamiliar to external customers and leak internal context. Renamed to generic enterprise vocabulary so the example reads as a generic change-control pattern: - submit_to_safefly -> submit_to_deployment_gateway - submit_to_ev2 -> submit_to_rollout_service - submit_to_r2d -> submit_to_release_readiness - create_ado_change_request -> create_change_request - get_icm_incident -> get_incident - ChangeKeep -> ChangeFlow - SAFEFLY-<id> -> DEPLOYGATE-<id> - 'Safefly', 'Ev2', 'R2D' -> 'Deployment Gateway', 'Rollout Service', 'Release Readiness' - 'ADO', 'ICM' -> 'change-tracker', 'incident-tracker' - 'internal change-management assistant' -> 'change-management assistant' Both function names and the string-literal handles used as dict keys/ identifiers in tools.py were renamed so the example remains functional. README updated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/incident_triage_agent: rewrite README opening for customer context The README opening framed this example as an 'ACS efficacy demo' with an 'A -> C demo path', cross-linked an internal draft PR (#88), and referenced a bank-manager demo that does not ship in this repo. Rewrote the title, opening paragraph, and TL;DR to present the example as a generic incident-triage agent evaluation. Preserved the variant tables, eval-config matrix, and provenance section unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/README: fix broken paths, add missing example entries - Quickstart YAML path: pipes/ -> prompt_agents/ (the pipes/ directory was renamed long ago; the README pointed at a 404). - Added table rows for four examples that exist on disk but were not listed: travel_planner_neurosan, change_control_agent, azure_doc_qa, benchmark. - Fixed dead doc links: docs/reference/cli.md -> docs/cli/commands.md. - Removed broken link to docs/case-study-incident-triage-joint.md (file does not exist). - Updated the layout block to match the current directory structure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * README: add inline Quick install block Surface a copy-pasteable install/run snippet above the 4-column 'Get started' table so first-time readers don't have to click through to find the bootstrap commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * scripts + pyproject: misc jargon and dead-reference cleanup - scripts/migrate_artifacts_to_pr23_vocab.py: docstring no longer references 'PR #23 vocabulary'; describes the migration in customer- neutral terms. - pyproject.toml: trimmed dspy pin comment to drop the bank-manager reference (that demo does not ship in this repo). - scripts/README.md, scripts/scenario_failure_prediction.py: replaced the relationship-entanglement-v1 suite name with the placeholder <your-suite-name> so examples are generic. - scripts/judge_stability_experiment.py: dropped internal model name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gration callouts (#189) The repo should read like a clean v1 launch, not a project mid-migration. Private-preview era transitions (assert_eval -> assert_ai, ASSERT_EVAL_* -> ASSERT_AI_*, prompts directory move) are noise in public docs because there were no public releases to break. Existing private-preview users are notified out-of-band. - README.md: drops the [!IMPORTANT] migration callout from PR #185. Added a bundled-viewer bullet to the capability list. - CHANGELOG.md: rewritten to empty [Unreleased] scaffold (Keep a Changelog 1.1.0), ready to populate when v0.1 ships. - .github templates: replaced stale `assert-eval` placeholders with `assert-ai` so contributor-facing examples match the current CLI. No code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…th after assert_ai rename (#190) The subprocess invocation in tests/test_runtime_safety.py::test_run_stage_coro_does_not_block_subprocess_exit_when_worker_leaked spawned a fresh sys.executable but did not forward the parent interpreter's import paths. In CI, pip install -e . populates site-packages so import assert_ai works inside the subprocess; in any environment where pytest is the only thing putting the project root on sys.path (developer running pytest without first installing, or a leftover venv from before the rename), the subprocess hits ModuleNotFoundError: No module named 'assert_ai' and the test fails before it can even exercise the leaked-worker shutdown path. Fix: build PYTHONPATH from the parent's sys.path and pass it via env= to subprocess.run. Works whether assert_ai is editable-installed or only discovered through pytest's rootdir hook. Pre-existing failure since PR #177 (assert_eval -> assert_ai rename) — the import statement was correctly renamed but the underlying env-propagation gap was unmasked once the package name no longer matched any stale install left in dev venvs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…ates Minthigpen/docs readme updates
Self-contained example comparing an unguarded LangGraph bank-manager agent against the same agent guarded by the new Agent Control Specification (ACS) runtime. Two ASSERT callables, two eval configs, two frozen n=100 result snapshots for viewer playback. Contents (scoped to examples/bank_manager_agent_shield/): - agent.py: chat_unguarded + chat_guarded_acs - mcp_server.py: mock banking MCP server - acs/manifest.yaml + acs/policy/bank_manager.rego: stateless ACS policy (SSN input, sensitivity-scoped read/transfer gates, approval/admin-mode gates, prompt-injection scrubber) - eval_unguarded_v2.yaml: baseline (owns systematize+test_set) - eval_guarded_acs.yaml: ACS variant (reuses baseline test_set) - results/: frozen n=100 artifacts for both variants ACS integration uses the idiomatic SDK orchestration helpers: control.run() for input/output gating around the agent execution, control.run_tool() per MCP tool for pre/post tool-call gating. Per-turn state (transfer_approved, admin_mode_active, account_sensitivity) is tracked by the host wrapper and threaded into each snapshot, since ACS is stateless by design. Headline n=100 (same test set across both variants): unguarded: safety_violation 39%, unjustified_refusal 2% ACS: safety_violation 9%, unjustified_refusal 2% README documents both usage paths: seed the committed results into artifacts/results/ and view them, or run the full pipeline end-to-end (requires agent_control_specification SDK + opa binary on PATH).
…usal Re-judged the existing n=100 inference outputs against the renamed rubric (no inference re-run). Updated taxonomy.json behavior_categories to match the names referenced by the committed test_set.jsonl so the viewer renders. Regenerated .viewer/ caches. Results (n=100, byte-identical inference): unguarded: policy_violation 40%, overrefusal 4% ACS: policy_violation 5%, overrefusal 15%
…ded.yaml The _v2 suffix is no longer meaningful; drop it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ariant
The unguarded variant-a-unguarded-n100 artifacts contained 4 scenario
rows that failed with stop_reason="target_error":
test_case_000053, test_case_000063, test_case_000065, test_case_000090
Root cause: examples/bank_manager_agent_shield/agent.py:309 calls
asyncio.run(_run_agent_async(...)) inside a worker thread. asyncio.run
creates a new event loop and refuses to nest, so under concurrent
scenario load the TaskGroup unwinds with an unhandled-errors crash.
These are inference errors, NOT ACS policy blocks. The ACS-guarded
variant ran clean at n=100.
This commit drops the 4 affected rows from every downstream artifact
in variant-a-unguarded-n100/ so the headline numbers reflect only the
96 cases that actually ran:
- inference_set.jsonl: 100 -> 96 rows
- scores.jsonl: 100 -> 96 rows
- .viewer/viewer_audit_rows.json: 50 -> 46 (the 4 were scenarios)
- .viewer/viewer_prompt_rows.json: 50 -> 50 (unaffected)
- .viewer/viewer_score_index.json: byte offsets rebuilt
- .viewer/viewer_transcript_index.json: byte offsets rebuilt
- .viewer/viewer_run_manifest.json: source_files size_bytes patched
- metrics.json: stages.judge.calls 100 -> 96 (skipped rows weren't
actually judged)
Recomputed unguarded headline from the surviving 96 scored rows:
policy_violation 42% (was 39%), overrefusal 4% (was 2%). README
updated to reflect n=96 for unguarded and n=100 for ACS-guarded.
Follow-up suggestion: fix chat_unguarded / chat_guarded_acs to not
call asyncio.run from within a thread. Either restructure as a
coroutine and let the caller await it, or use a long-lived loop via
asyncio.new_event_loop()/loop.run_until_complete() guarded by a lock.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ager_agent_control
Matches the broader ACS = Agent Control Specification framing. Folder
rename is via git mv to preserve blame. All in-repo references (Python
imports, YAML targets, frozen artifact configs, viewer rows, README,
scripts/render_trade_off.py) updated:
- 712 underscore-form occurrences (bank_manager_agent_shield)
- 12 dash-form occurrences (bank-manager-agent-shield) — suite IDs
inside taxonomy.json, config.yaml, and README copy commands
Internal artifact directory names (variant-a-unguarded-n100,
variant-e-guarded-acs-n100) are unchanged — those are run names, not
suite names.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The systematize stage regenerated taxonomy.json with 4 formal category names
but test_set.jsonl + downstream artifacts retained the older descriptive names,
causing the viewer to 500 at viewer/src/lib/server/data.ts:184 (behaviorDefinition
strict lookup).
Maps every test case's dimensions.behavior + top-level behavior fields to the
canonical formal name across:
- examples/bank_manager_agent_control/results/test_set.jsonl
- per-variant inference_set.jsonl, scores.jsonl, .viewer/*.json
Mapping:
authorized banking tool use -> Authorization-gated action handling
accurate financial statements ... -> Correct tool use and customer-facing banking help
confidentiality and social-engineering -> split per-case into Confidential data withholding
OR Resistance to impersonation and prompt injection
Each of the 4 formal categories now has >= 1 test case (the 4th was empty before).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Catches up the demo README to the post-rebrand CLI binary name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-ai, ASSERT_AI_*) After rebasing onto main, sweeps the demo for stale terminology: - p2m run -> assert-ai run (in YAML comments + README) - p2m/stages/* -> assert_ai/stages/* (in YAML comments) - ASSERT_EVAL_* -> ASSERT_AI_* (env vars) - assert_eval -> assert_ai (module paths, if any) - assert-eval -> assert-ai (CLI binary) - drops references to eval_guarded_v2.yaml / eval_guarded_v3.yaml (now eval_guarded_acs.yaml only) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ent.py Drops in the canonical Phoenix auto_instrument pattern at the top of agent.py so LangChain / OpenAI / MCP tool calls flow into Phoenix without any framework config. Optional via try/except — demo still runs without arize-phoenix-otel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…y is single-sided The "Permissible requests failed" / "Not-permissible requests failed" card pair only makes sense when the taxonomy has BOTH permissible and not-permissible behaviors. For single-sided taxonomies the not-aligned bucket would render an empty "no relevant judgments" tile that's noise. Now returns null/null when the permissibility index has < 2 distinct values, so the run-detail page's `policyViolationOnPermissible || policyViolationOnNotPermissible` gate naturally skips the section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds chat_unguarded_prompted callable that runs the same raw LangGraph agent as chat_unguarded but with a defensive addendum appended to the system prompt (no tool gating). Reuses the suite-root frozen test_set + same judge dims for a fair 3-way comparison: bare baseline (a) vs prompt-engineering (c) vs ACS-guarded (e). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Motivation / linked issue
Changes
Testing
Checklist
pytestand/or viewer checks as applicable).CHANGELOG.mdentry is included.