docs+examples: post-rename cleanup, jargon scrub, broken nav fixes by changliu2 · Pull Request #181 · responsibleai/ASSERT

changliu2 · 2026-06-01T04:42:19Z

TL;DR

Closes 14 findings from an internal customer-readiness audit of the
preview repo. All changes are scoped to docs, examples, scripts, and
test guards — no assert_ai/ core code changes in this PR.

Nine logically-grouped commits (one cluster per commit) make this easy
to review file-by-file.

What changed

tests: tighten p2m guard regex — the previous \bp2m\b
word-boundary regex missed env-var leaks like P2M_AZURE_DEPLOYMENT
because the underscore broke the word boundary. Switched to a
case-insensitive substring match, added skip lists for binary
extensions (.svg) and lockfiles (package-lock.json, uv.lock,
etc.) that occasionally embed p2m in sha512 hashes, and added a
sanity test asserting the new pattern catches the historical leak.
examples: rename P2M_* env vars to ASSERT_* — three quickstart
READMEs still referenced P2M_AZURE_DEPLOYMENT / P2M_TARGET_MODEL
even though the agent.py files already read ASSERT_*. Customers
following the docs verbatim would hit a missing-deployment error.
examples/docs: replace internal model names with public substitutes —
swapped azure/gpt-5.4* for azure/gpt-4o* and azure/gpt-4o-mini
across docs, example configs, example agent code, and the website
snippets. assert_ai/, tests/, .github/workflows/, and frozen
artifacts/results/** are intentionally not touched.
AGENTS.md: complete rename to ASSERT — PR docs: replace stale microsoft/adaptive-eval URLs with responsibleai/ASSERT #178 fixed the GitHub
URL but the prose, title, and doc cross-links still said "Adaptive
Eval" and pointed at three moved/renamed docs (quickstart.md,
writing-eval-specs.md, reading-results.md). Same fix applied to
.cursorrules, .devcontainer/devcontainer.json, CONTRIBUTING.md,
SUPPORT.md, and examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md.
examples/change_control_agent: genericize internal Microsoft infra references —
rewrote internal Microsoft system names (Safefly, Ev2, R2D, ICM, ADO,
ChangeKeep) to generic enterprise terminology (Deployment Gateway,
Rollout Service, Release Readiness, incident-tracker, change-tracker,
ChangeFlow). Both function names and the string-literal handles used
as dict keys in tools.py were renamed so the example still runs.
examples/incident_triage_agent: rewrite README opening — the
README opening framed this example as an internal "ACS efficacy demo"
and cross-linked a draft PR. Rewrote the title, opening paragraph,
and TL;DR to present it as a generic incident-triage evaluation.
Variant tables and provenance section preserved unchanged.
examples/README: fix broken paths, add missing example entries —
the YAML quickstart path was pipes/ (renamed long ago to
prompt_agents/). Added rows for four examples that exist on disk
but were not listed (travel_planner_neurosan, change_control_agent,
azure_doc_qa, benchmark). Fixed docs/reference/cli.md →
docs/cli/commands.md and removed a link to a doc that doesn't exist.
README: add inline Quick install block — surfaces a
copy-pasteable pip install / cp .env.example .env /
assert-ai run snippet above the 4-column "Get started" table.
scripts + pyproject + internal-pipeline-prompts: misc cleanup —
small jargon swaps: migrate-script docstring no longer references
"PR refactor(p2m): terminology migration to maximize broad-developer recognition #23 vocabulary"; seed jargon in prompt-pipeline docs becomes
"test case"; pyproject dspy pin comment trimmed; scripts/README.md
and scenario_failure_prediction.py use a <your-suite-name>
placeholder instead of an internal suite name; one internal model
name dropped from judge_stability_experiment.py.

How to verify

# 1. The tightened P2M guard test passes (and would have caught the
#    historical P2M_AZURE_DEPLOYMENT leak):
pytest tests/test_no_p2m_references.py -v

# 2. Customer-facing surfaces are clean (each should return zero hits,
#    aside from allowlisted artifacts/results/** historical records):
grep -r --include="*.py" --include="*.md" "P2M_" .
grep -r --include="*.py" --include="*.md" "gpt-5.4" . | grep -v "artifacts/results"
grep -r --include="*.md" "Adaptive Eval" .

# 3. Full smoke test suite still green:
pytest tests/ --ignore=tests/comparison --ignore=tests/regression

Out of scope (intentionally deferred)

internal-pipeline-prompts/ directory rename — touches
assert_ai/core/io.py::PROMPTS_DIR and would require artifact-path
migration. Tracked separately.
assert_ai/ core code, tests/comparison/, tests/regression/,
.github/workflows/regression.yml, and artifacts/results/** are
intentionally not touched in this PR.

The previous \\b p2m \b\ word-boundary regex missed env-var leaks like P2M_AZURE_DEPLOYMENT because the underscore broke the word boundary. Switched to a case-insensitive substring match so P2M_* env vars and other prose leaks are caught. To keep the test signal clean, also skip: - Binary file extensions (.svg, image formats) that may embed base64. - Lockfiles (package-lock.json, yarn.lock, pnpm-lock.yaml, uv.lock, poetry.lock) where sha512 hashes coincidentally contain 'p2m'. Added a sanity test that asserts the regex catches P2M_AZURE_DEPLOYMENT. Deleted an orphan website/public/icons/P2M Thumbnail.svg (no references). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replaced internal-only model names with publicly available substitutes across examples, docs, and the website snippets so customers can run copy-paste configs without hitting unknown-model errors. - azure/gpt-5.4-mini -> azure/gpt-4o-mini - azure/gpt-5.4-nano -> azure/gpt-4o-mini - azure/gpt-5.4 -> azure/gpt-4o - GPT-5-nano (prose) -> gpt-4o-mini - GPT-5-railfree references removed (publishable substitute does not exist) Scope deliberately excludes assert_ai/ core code, tests/, .github/ workflows, and artifacts/results/** (frozen historical records). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

PR #178 fixed the GitHub URL but the prose, title, and doc cross-links still referenced 'Adaptive Eval' and pointed to renamed/moved doc paths. - AGENTS.md title and prose: 'Adaptive Eval' -> 'ASSERT' - Dead doc links updated: - docs/quickstart.md -> docs/getting-started.md - docs/writing-eval-specs.md -> docs/guides/create-evaluation.md - docs/reading-results.md -> docs/guides/results.md - Same branding fix applied to .cursorrules, .devcontainer/devcontainer.json, CONTRIBUTING.md, SUPPORT.md, and examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ferences The change_control_agent example referenced internal Microsoft deployment/ops systems (Safefly, Ev2, R2D, ICM, ADO, ChangeKeep) which are unfamiliar to external customers and leak internal context. Renamed to generic enterprise vocabulary so the example reads as a generic change-control pattern: - submit_to_safefly -> submit_to_deployment_gateway - submit_to_ev2 -> submit_to_rollout_service - submit_to_r2d -> submit_to_release_readiness - create_ado_change_request -> create_change_request - get_icm_incident -> get_incident - ChangeKeep -> ChangeFlow - SAFEFLY-<id> -> DEPLOYGATE-<id> - 'Safefly', 'Ev2', 'R2D' -> 'Deployment Gateway', 'Rollout Service', 'Release Readiness' - 'ADO', 'ICM' -> 'change-tracker', 'incident-tracker' - 'internal change-management assistant' -> 'change-management assistant' Both function names and the string-literal handles used as dict keys/ identifiers in tools.py were renamed so the example remains functional. README updated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ontext The README opening framed this example as an 'ACS efficacy demo' with an 'A -> C demo path', cross-linked an internal draft PR (#88), and referenced a bank-manager demo that does not ship in this repo. Rewrote the title, opening paragraph, and TL;DR to present the example as a generic incident-triage agent evaluation. Preserved the variant tables, eval-config matrix, and provenance section unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Quickstart YAML path: pipes/ -> prompt_agents/ (the pipes/ directory was renamed long ago; the README pointed at a 404). - Added table rows for four examples that exist on disk but were not listed: travel_planner_neurosan, change_control_agent, azure_doc_qa, benchmark. - Fixed dead doc links: docs/reference/cli.md -> docs/cli/commands.md. - Removed broken link to docs/case-study-incident-triage-joint.md (file does not exist). - Updated the layout block to match the current directory structure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Surface a copy-pasteable install/run snippet above the 4-column 'Get started' table so first-time readers don't have to click through to find the bootstrap commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- scripts/migrate_artifacts_to_pr23_vocab.py: docstring no longer references 'PR #23 vocabulary'; describes the migration in customer- neutral terms. - pyproject.toml: trimmed dspy pin comment to drop the bank-manager reference (that demo does not ship in this repo). - scripts/README.md, scripts/scenario_failure_prediction.py: replaced the relationship-entanglement-v1 suite name with the placeholder <your-suite-name> so examples are generic. - scripts/judge_stability_experiment.py: dropped internal model name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…pts/ (#188) Customer-facing terminology cleanup (M-3 from prior audit). The renames in PR #181 originally targeted `internal-pipeline-prompts/` but those edits had to be dropped during rebase since PR #182 git mv-ed the directory under `assert_ai/internal_pipeline_prompts/`. Re-applying against the new path. Scope: prompt markdown files only. Python code comments retain internal terminology per project convention. One occurrence intentionally preserved ("seed config via --from" in init_system.md L436 — refers to a starter config, not a test case). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

changliu2 · 2026-06-01T06:00:16Z

Rebased onto current main (post #182, #185, #186, #187). Dropped or fixed the following hunks that main already covers:

Dropped commit examples: rename P2M_* env vars to ASSERT_* in READMEs (3 example READMEs) — build: wheel-safe packaging, CI build workflow, env-var rename #182 already renamed these env vars in main (and updated the default model name from gpt-4o-mini to gpt-5.4-mini at the same time). The whole commit became redundant after rebase, so I skipped it; no leftover hunks.
Dropped 2 hunks in scripts + pyproject + internal-pipeline-prompts: misc jargon and dead-reference cleanup:
- internal-pipeline-prompts/init_system.md
- internal-pipeline-prompts/test_set_generation_guidance.md
build: wheel-safe packaging, CI build workflow, env-var rename #182 git mv-moved this directory to assert_ai/internal_pipeline_prompts/. Git's rename detection auto-applied the jargon edits to the new path, but per separation of concerns those file changes are owned by a follow-up PR. Restored those 2 files to main and amended the commit; renamed the commit to scripts + pyproject: misc jargon and dead-reference cleanup.
Updated allowlist in tests/test_no_p2m_references.py (folded into the existing tests: tighten p2m guard regex commit via --fixup): added README.md and CHANGELOG.md to _ALLOWED_PATH_PREFIXES. docs: add CHANGELOG.md + README migration note for assert_ai rename #185 added a migration note to README.md and a CHANGELOG that intentionally reference the legacy P2M_* env-var aliases for upgrade context. Without the allowlist update the tightened test failed locally; with it, the full 966-test collection imports cleanly and the targeted test passes.

All 8 remaining commits rebased cleanly. CI should pick up the new push.

* docs: normalize commands and path separators * fix(viewer): truncate long callable target labels in compare view (#160) The compare page derived a short label from `run.model` by splitting on `/` only (lines 301, 376). When the target is a Python callable like `examples.bank_manager_demo.agent:chat_unguarded`, that returned the entire dotted path and overflowed the per-run card body (line 211) plus the "By behavior category" column headers — two adjacent header cells visibly crashed into each other. Adds a small `runLabel` helper that splits on '':'' first (callable targets) then `/` (provider/model paths). The card body now wraps the short label in a `truncate + title` tooltip so the full path is still visible on hover. Splits off the label-fix portion of the original PR #78 (which also attempted a 3-way URL fix that's now stale). The 3-way URL plumbing is out of scope here and can ship in a separate PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * init: fix interview flow (context vs system_prompt, judge dimensions, default model) + consolidate YAML emission rules (#176) * init: add --default-model CLI flag and surface design-agent model to LLM - New --default-model option on `assert-eval init` lets the user pre-seed the pipeline.default_model hint for the interview. - _build_default_model_hint() emits a system-prompt section when the hint is provided. - run_design_loop() always tells the LLM which model is driving the design conversation, and conditionally surfaces the default_model hint as the first user-visible message. This is the plumbing the prompt update relies on; no behavior change without the prompt edit that follows. * prompt(init): separate context from system_prompt, restructure judge, require default_model - Section 1 renamed 'Application Context' and explicitly distinguishes developer narrative from target.system_prompt for hosted-model targets. - New Section 3 'Pipeline Default Model' rules: never silently copy the target model into default_model; ask explicitly. - Renumbered Behavior (4), Test Set (5), Judge (6). - Judge built-ins: policy_violation, overrefusal. - Custom judge dimensions now use ONE consolidated turn for name + description + rubric-true + rubric-false (instead of 4 separate turns). - Added explicit guideline against conflating context with target.system_prompt. * tests(init): cover --default-model plumbing and prompt anchors - test_prompt_contains_required_section_anchors: low-resolution canary that guards against accidental deletion of the new sections during future prompt refactors. Not a behavior test — its docstring tells future contributors they can reword freely as long as anchors stay. - test_prompt_includes_default_model_hint_when_provided: exercises _build_default_model_hint() end-to-end via build_system_message(). - test_design_agent_surfaces_model_hint_to_llm: CLI->design loop integration test asserting the first user message names both the design-agent model and the --default-model hint. * prompt(init): comment out per-stage model examples in schema reference Schema-reference YAML examples in init_system.md were showing live 'model:' blocks under systematize, test_set.prompt, test_set.scenario, inference.tester, and judge. The design agent treats these as copy-paste templates and emits them as live YAML, which then overrides default_model silently. Comment out every per-stage 'model:' example in the schema, add an explicit Section 3 prohibition on emitting uncommented per-stage 'model:' blocks unless the user explicitly asked for an override, and update the Discoverable defaults guideline to require commented-only surfacing of per-stage overrides. * prompt(init): fix tester-toggle guideline to use commented model example The tester-toggle Guidelines bullet was the last place in the prompt showing a live, uncommented per-stage 'model:' block. Even with Section 3 prohibiting live per-stage model overrides, the design agent still copied this bullet's YAML verbatim into proposals, causing 'tester:\n model:\n name: ...' to leak into generated configs. Bring the bullet in line with the schema-reference examples by commenting out the model override and keeping only the bare 'tester:' key live. * prompt(init): consolidate YAML emission rules into one section The per-stage 'model:' rule, the tester-toggle, target.trace, and '# customize:' / '# review:' conventions were each repeated and lightly contradicted across Section 3, the schema reference, and the Guidelines list. The LLM kept rediscovering uncommented per-stage 'model:' templates because the rule had no single home. Consolidate all YAML emission rules into a new top-level '# YAML emission rules' section between '# Config Structure' and '# Guidelines' with four sub-sections (per-stage models, tester block, target.trace, customization hints). Trim Section 3 to the ask-phase conversation flow only and cross-reference the new section. Drop the duplicated tester/target.trace/customization bullets from Guidelines. Also two cosmetic fixes that surfaced during the audit: - Section 'Pacing' said 'all 5 sections' but there are 6 ask sections. - 'Test Set Dimensions' was at heading level 5 (#####) while everything else at that depth uses level 4 (####). * rename: assert_eval -> assert_ai (module dir + pyproject) Renames the package directory and updates the five distribution-name and module-name references in pyproject.toml (name, [project.scripts], all extra, dev dependency-group, and setuptools packages.find). Also drops examples* from the wheel per the PyPI publishing plan. Imports inside the package are not yet updated -- follow-up commit. * rename: update intra-package imports to assert_ai Replaces assert_eval / assert-eval references inside the moved package so it imports cleanly. Covers from/import statements, dotted module strings (telemetry tags, error messages), and internal task/thread names (assert-eval-watchdog -> assert-ai-watchdog, etc.). Tests, docs, scripts, and CI updated separately. * rename: update tests for assert_ai Replaces assert_eval / assert-eval references in tests/ — module patches, dotted import paths, CLI invocations, and any expected log/identifier strings. * rename: update top-level assert_ai modules Catches the depth-1 .py files in assert_ai/ that the package-imports commit missed: cli.py, config.py, display.py, results.py, runner.py, viewer_read_model.py. Same dual-pattern (assert_eval -> assert_ai, assert-eval -> assert-ai). * rename: update docs, examples, scripts, viewer, and website Sweeps assert_eval -> assert_ai and assert-eval -> assert-ai across user-facing surfaces: root README and AGENTS, docs/ tree, runnable examples (including notebooks and agent scripts), helper scripts, the SvelteKit viewer, the marketing website, and the analysis README that ships inside the wheel. * rename: update CI workflow path filter Path filter in .github/workflows/regression.yml now points at assert_ai/** instead of assert_eval/**. * rename: update .gitignore policy artifact pattern Renames assert_eval_policy.* -> assert_ai_policy.* so runtime policy artifacts stay ignored after the package rename. * chore: rename root package in uv.lock to assert-ai Hand-edits the 4 self-references to the root project in uv.lock to match the renamed distribution. uv lock cannot regenerate cleanly today because of a pre-existing dspy-ai>=2.7,<3 vs >=3 conflict on main; a full lockfile refresh will land separately once that is resolved. * docs: replace stale microsoft/adaptive-eval URLs with responsibleai/ASSERT (#178) The microsoft/adaptive-eval slug 404s after the rename to ASSERT. Two customer-facing docs still pointed at the dead URL: - CONTRIBUTING.md: dev-setup clone snippets (bash + powershell) - AGENTS.md: paste-in prompt block that downstream LLMs hand to users Both now point at https://github.com/responsibleai/ASSERT. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(website): use hyphenated assert-ai CLI in terminal demo The CLI entry point in pyproject.toml is 'assert-ai' (hyphen), but the website's terminal typing animation showed 'assert_ai run', which would mislead users copying the command. Reported by Copilot review on PR #177. * build: restore examples* in packages.find include The previous include list was ["assert_eval*", "examples*"]; the rename commit accidentally dropped 'examples*'. Tests import from 'examples.agents.*' (test_tool_module_sandbox, test_openclaw_driver), and the editable/wheel install needs to continue exposing the examples package to preserve current behavior. Packaging cleanup (e.g. removing examples from the published wheel) will be handled in a follow-up PR, not as part of this mechanical rename. Reported by Copilot review on PR #177. * docs(assets): rename framework diagram to assert-ai-framework-diagram.png README.md was updated to reference assets/assert-ai-framework-diagram.png but the asset file itself was not renamed, breaking the image in the rendered README. Reported by Copilot review on PR #177. * Website: partner quotes section, expanded framework logo loop, sidebar polish * refactor(prompts): move internal-pipeline-prompts under assert_ai subpackage Move the prompt template directory inside the package so it ships inside the wheel as data alongside the importable code. Pure file move plus an empty __init__.py so importlib.resources can discover it as a real subpackage. The next commit switches io.py to load via importlib.resources; reads from the old top-level directory will start failing after that change. * fix(io): resolve PROMPTS_DIR via importlib.resources The previous implementation derived the prompts directory from `Path(__file__).resolve().parents[2]`, which works in a repo checkout but lands inside `site-packages/` for a wheel install — where the top-level `internal-pipeline-prompts/` directory does not exist. Switch to `importlib.resources.files('assert_ai.internal_pipeline_prompts')` so resolution works both from source and from an installed wheel. The returned Traversable still supports `/`, `.read_text()`, `.read_bytes()`, and `.is_file()`, which is everything the existing call sites need. `artifact_cache._prompt_descriptor` is updated to use `.is_file()` instead of `.exists()` since Traversable does not guarantee `.exists()`. * build: drop examples* from packages.find include `examples*` was previously included in `packages.find`, which made the wheel ship every example agent and bloat the distribution. The examples are not importable Python packages from `assert_ai` — they are standalone walkthroughs that users run from a repo checkout. They should not be in the wheel. * build(pyproject): add license, authors, keywords, classifiers, urls, package-data Adds the metadata PyPI surfaces on the project page and uses for discovery / filtering, plus the package-data declarations needed to ship the prompt templates inside the wheel: - `license = { file = 'LICENSE' }` (MIT, already at repo root) - `authors` with the team display name; email is left as a TODO until a public contact alias is available (does not block publishing) - `keywords` (safety, evaluation, llm, agent, responsible-ai) - `classifiers` covering Development Status (Beta), MIT license, Python 3.11/3.12/3.13, and AI / QA / Testing topics - `[project.urls]` (Homepage, Repository, Issues, Documentation) - `[tool.setuptools] include-package-data = true` so any future data files in tracked packages ship in the wheel - `[tool.setuptools.package-data]` explicitly listing `assert_ai.internal_pipeline_prompts = ['*.md']`. Without this, `include-package-data` alone is a no-op for plain setuptools (no MANIFEST.in, no setuptools-scm), and the .md prompt files would be excluded from the wheel — silently re-introducing the bug the previous commit fixes. * ci: add build.yml workflow (PEP 517 build + cross-platform install smoke) `build.yml` runs on every push to main, every PR to main, and on `workflow_dispatch`. Two jobs: 1. `build` (ubuntu-latest, Python 3.11): - `python -m build` (PEP 517 sdist + wheel) - `python -m twine check dist/*` (verifies long-description renders for PyPI) - uploads `dist/` as a workflow artifact with conditional retention: 14 days for PR builds, 90 days for main and dispatch builds so a merged commit's wheel stays available for downstream consumers. 2. `test-install` (3x3 matrix: ubuntu / macos / windows x Python 3.11 / 3.12 / 3.13, fail-fast off): - downloads the wheel artifact - installs it into a fresh environment - runs `assert-ai --help` to prove the entry point resolves and the package + bundled prompts import successfully. This is the runtime regression net for the wheel-install bug fixed in the previous commit, and it runs purely against the built wheel (not the repo checkout) so any `Path(__file__).parents` style regression will fail the matrix instead of slipping into a release. `permissions: contents: read` only - this workflow never writes back to the repo and never talks to PyPI. * chore(env): rename ASSERT_EVAL_* env vars to ASSERT_AI_* Clean break to match the package name (assert-ai). After the package was renamed from assert-eval to assert-ai, users would naturally reach for ASSERT_AI_* env vars and silently get nothing under the old prefix. Renamed (no compatibility shim): - assert_ai/cli.py: Click auto_envvar_prefix ASSERT_EVAL -> ASSERT_AI (so e.g. ASSERT_AI_CONFIG=... wires the --config flag) - viewer/src/lib/server/run-spawn.ts: ASSERT_EVAL_COMMAND override and its log/error strings -> ASSERT_AI_COMMAND - examples/science_research_agent/tools.py and README.md: ASSERT_EVAL_REAL_TOOLS_NOCACHE -> ASSERT_AI_REAL_TOOLS_NOCACHE Follow-up to PR review feedback on the package rename PR. * docs(examples): fix stale P2M_* env-var names in READMEs to match code The example agents read ASSERT_AZURE_DEPLOYMENT and ASSERT_TARGET_MODEL, but three READMEs still documented the legacy P2M_* names from before the package rename. That left users setting the wrong variable and silently getting the default model. Aligns README docs with the code: - examples/travel_planner_langgraph/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT (inline comment + var table) - examples/phoenix_auto_trace/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT - examples/travel_planner_neurosan/README.md: P2M_TARGET_MODEL -> ASSERT_TARGET_MODEL Docs-only; no behavior change. tests/test_no_p2m_references.py still passes (the guard uses \bp2m\b which doesn't catch P2M_* tokens). * docs: add CHANGELOG.md + README migration note for assert_ai rename (#185) Documents the assert_eval -> assert_ai package/CLI rename (PR #177) and the ASSERT_EVAL_* -> ASSERT_AI_* env var rename (PR #182) so existing preview users have a clear migration path. Keep a Changelog 1.1.0 format for CHANGELOG.md. README gets a short [!IMPORTANT] callout near the top. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(viewer): rename spawnAssertEvalRun / resolveAssertEvalCommand to AssertAi* (#187) Catches up TS symbol names with PR #177 (assert_eval -> assert_ai package rename) and PR #182 (ASSERT_EVAL_* -> ASSERT_AI_* env var rename). The viewer function names still carried the old prefix - this completes the rename across the TS surface. No behavior change. Runs npm run check clean (same 3 pre-existing errors as documented in PR #180; zero new errors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: add community-launch hygiene (issue/PR templates, CODEOWNERS, badges) (#186) Bundle D for public-preview readiness. Adds: - .github/ISSUE_TEMPLATE/{bug_report.yml,feature_request.yml,config.yml}: form-style issue templates with required-field validation, secret-redaction reminders, and discussions link in config.yml. - .github/PULL_REQUEST_TEMPLATE.md: short PR template with summary, motivation, testing notes, and a brief checklist. - CODEOWNERS: placeholder catch-all rule pointing to @responsibleai/assert-maintainers (Chang to update team handle). - README badges: CI build status (from PR #182's build.yml workflow), supported Python versions (3.11/3.12/3.13 matching CI matrix), license. PyPI badge intentionally omitted until the package is published. No code change. No CHANGELOG.md (parallel PR delivers that). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(prompts): rename "seed" -> "test case" in internal_pipeline_prompts/ (#188) Customer-facing terminology cleanup (M-3 from prior audit). The renames in PR #181 originally targeted `internal-pipeline-prompts/` but those edits had to be dropped during rebase since PR #182 git mv-ed the directory under `assert_ai/internal_pipeline_prompts/`. Re-applying against the new path. Scope: prompt markdown files only. Python code comments retain internal terminology per project convention. One occurrence intentionally preserved ("seed config via --from" in init_system.md L436 — refers to a starter config, not a test case). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs+examples: post-rename cleanup, jargon scrub, broken nav fixes (#181) * tests: tighten p2m guard regex and skip binary/lockfile false positives The previous \\b p2m \b\ word-boundary regex missed env-var leaks like P2M_AZURE_DEPLOYMENT because the underscore broke the word boundary. Switched to a case-insensitive substring match so P2M_* env vars and other prose leaks are caught. To keep the test signal clean, also skip: - Binary file extensions (.svg, image formats) that may embed base64. - Lockfiles (package-lock.json, yarn.lock, pnpm-lock.yaml, uv.lock, poetry.lock) where sha512 hashes coincidentally contain 'p2m'. Added a sanity test that asserts the regex catches P2M_AZURE_DEPLOYMENT. Deleted an orphan website/public/icons/P2M Thumbnail.svg (no references). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/docs: replace internal model names with public substitutes Replaced internal-only model names with publicly available substitutes across examples, docs, and the website snippets so customers can run copy-paste configs without hitting unknown-model errors. - azure/gpt-5.4-mini -> azure/gpt-4o-mini - azure/gpt-5.4-nano -> azure/gpt-4o-mini - azure/gpt-5.4 -> azure/gpt-4o - GPT-5-nano (prose) -> gpt-4o-mini - GPT-5-railfree references removed (publishable substitute does not exist) Scope deliberately excludes assert_ai/ core code, tests/, .github/ workflows, and artifacts/results/** (frozen historical records). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * AGENTS.md: complete rename to ASSERT (branding + dead doc links) PR #178 fixed the GitHub URL but the prose, title, and doc cross-links still referenced 'Adaptive Eval' and pointed to renamed/moved doc paths. - AGENTS.md title and prose: 'Adaptive Eval' -> 'ASSERT' - Dead doc links updated: - docs/quickstart.md -> docs/getting-started.md - docs/writing-eval-specs.md -> docs/guides/create-evaluation.md - docs/reading-results.md -> docs/guides/results.md - Same branding fix applied to .cursorrules, .devcontainer/devcontainer.json, CONTRIBUTING.md, SUPPORT.md, and examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/change_control_agent: genericize internal Microsoft infra references The change_control_agent example referenced internal Microsoft deployment/ops systems (Safefly, Ev2, R2D, ICM, ADO, ChangeKeep) which are unfamiliar to external customers and leak internal context. Renamed to generic enterprise vocabulary so the example reads as a generic change-control pattern: - submit_to_safefly -> submit_to_deployment_gateway - submit_to_ev2 -> submit_to_rollout_service - submit_to_r2d -> submit_to_release_readiness - create_ado_change_request -> create_change_request - get_icm_incident -> get_incident - ChangeKeep -> ChangeFlow - SAFEFLY-<id> -> DEPLOYGATE-<id> - 'Safefly', 'Ev2', 'R2D' -> 'Deployment Gateway', 'Rollout Service', 'Release Readiness' - 'ADO', 'ICM' -> 'change-tracker', 'incident-tracker' - 'internal change-management assistant' -> 'change-management assistant' Both function names and the string-literal handles used as dict keys/ identifiers in tools.py were renamed so the example remains functional. README updated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/incident_triage_agent: rewrite README opening for customer context The README opening framed this example as an 'ACS efficacy demo' with an 'A -> C demo path', cross-linked an internal draft PR (#88), and referenced a bank-manager demo that does not ship in this repo. Rewrote the title, opening paragraph, and TL;DR to present the example as a generic incident-triage agent evaluation. Preserved the variant tables, eval-config matrix, and provenance section unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/README: fix broken paths, add missing example entries - Quickstart YAML path: pipes/ -> prompt_agents/ (the pipes/ directory was renamed long ago; the README pointed at a 404). - Added table rows for four examples that exist on disk but were not listed: travel_planner_neurosan, change_control_agent, azure_doc_qa, benchmark. - Fixed dead doc links: docs/reference/cli.md -> docs/cli/commands.md. - Removed broken link to docs/case-study-incident-triage-joint.md (file does not exist). - Updated the layout block to match the current directory structure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * README: add inline Quick install block Surface a copy-pasteable install/run snippet above the 4-column 'Get started' table so first-time readers don't have to click through to find the bootstrap commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * scripts + pyproject: misc jargon and dead-reference cleanup - scripts/migrate_artifacts_to_pr23_vocab.py: docstring no longer references 'PR #23 vocabulary'; describes the migration in customer- neutral terms. - pyproject.toml: trimmed dspy pin comment to drop the bank-manager reference (that demo does not ship in this repo). - scripts/README.md, scripts/scenario_failure_prediction.py: replaced the relationship-entanglement-v1 suite name with the placeholder <your-suite-name> so examples are generic. - scripts/judge_stability_experiment.py: dropped internal model name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: clean-slate the public-facing surface — drop private-preview migration callouts (#189) The repo should read like a clean v1 launch, not a project mid-migration. Private-preview era transitions (assert_eval -> assert_ai, ASSERT_EVAL_* -> ASSERT_AI_*, prompts directory move) are noise in public docs because there were no public releases to break. Existing private-preview users are notified out-of-band. - README.md: drops the [!IMPORTANT] migration callout from PR #185. Added a bundled-viewer bullet to the capability list. - CHANGELOG.md: rewritten to empty [Unreleased] scaffold (Keep a Changelog 1.1.0), ready to populate when v0.1 ships. - .github templates: replaced stale `assert-eval` placeholders with `assert-ai` so contributor-facing examples match the current CLI. No code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(tests): make test_runtime_safety subprocess inherit parent sys.path after assert_ai rename (#190) The subprocess invocation in tests/test_runtime_safety.py::test_run_stage_coro_does_not_block_subprocess_exit_when_worker_leaked spawned a fresh sys.executable but did not forward the parent interpreter's import paths. In CI, pip install -e . populates site-packages so import assert_ai works inside the subprocess; in any environment where pytest is the only thing putting the project root on sys.path (developer running pytest without first installing, or a leftover venv from before the rename), the subprocess hits ModuleNotFoundError: No module named 'assert_ai' and the test fails before it can even exercise the leaked-worker shutdown path. Fix: build PYTHONPATH from the parent's sys.path and pass it via env= to subprocess.run. Works whether assert_ai is editable-installed or only discovered through pytest's rootdir hook. Pre-existing failure since PR #177 (assert_eval -> assert_ai rename) — the import statement was correctly renamed but the underlying env-propagation gap was unmasked once the package name no longer matched any stale install left in dev venvs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: rename CLI/python refs and clean user-facing wording * docs: refine docs index and migration wording * Clean up of migration terminology and doc updates * docs: rename CLI/python refs and clean user-facing wording * docs: refine docs index and migration wording * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Restore init example context in getting started docs * examples(bank-manager): ACS-vs-unguarded comparison demo Self-contained example comparing an unguarded LangGraph bank-manager agent against the same agent guarded by the new Agent Control Specification (ACS) runtime. Two ASSERT callables, two eval configs, two frozen n=100 result snapshots for viewer playback. Contents (scoped to examples/bank_manager_agent_shield/): - agent.py: chat_unguarded + chat_guarded_acs - mcp_server.py: mock banking MCP server - acs/manifest.yaml + acs/policy/bank_manager.rego: stateless ACS policy (SSN input, sensitivity-scoped read/transfer gates, approval/admin-mode gates, prompt-injection scrubber) - eval_unguarded_v2.yaml: baseline (owns systematize+test_set) - eval_guarded_acs.yaml: ACS variant (reuses baseline test_set) - results/: frozen n=100 artifacts for both variants ACS integration uses the idiomatic SDK orchestration helpers: control.run() for input/output gating around the agent execution, control.run_tool() per MCP tool for pre/post tool-call gating. Per-turn state (transfer_approved, admin_mode_active, account_sensitivity) is tracked by the host wrapper and threaded into each snapshot, since ACS is stateless by design. Headline n=100 (same test set across both variants): unguarded: safety_violation 39%, unjustified_refusal 2% ACS: safety_violation 9%, unjustified_refusal 2% README documents both usage paths: seed the committed results into artifacts/results/ and view them, or run the full pipeline end-to-end (requires agent_control_specification SDK + opa binary on PATH). * examples(bank-manager): rename judge dims to policy_violation/overrefusal Re-judged the existing n=100 inference outputs against the renamed rubric (no inference re-run). Updated taxonomy.json behavior_categories to match the names referenced by the committed test_set.jsonl so the viewer renders. Regenerated .viewer/ caches. Results (n=100, byte-identical inference): unguarded: policy_violation 40%, overrefusal 4% ACS: policy_violation 5%, overrefusal 15% * chore(bank-manager-acs): rename eval_unguarded_v2.yaml to eval_unguarded.yaml The _v2 suffix is no longer meaningful; drop it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(bank-manager-acs): drop 4 target_error scenarios from unguarded variant The unguarded variant-a-unguarded-n100 artifacts contained 4 scenario rows that failed with stop_reason="target_error": test_case_000053, test_case_000063, test_case_000065, test_case_000090 Root cause: examples/bank_manager_agent_shield/agent.py:309 calls asyncio.run(_run_agent_async(...)) inside a worker thread. asyncio.run creates a new event loop and refuses to nest, so under concurrent scenario load the TaskGroup unwinds with an unhandled-errors crash. These are inference errors, NOT ACS policy blocks. The ACS-guarded variant ran clean at n=100. This commit drops the 4 affected rows from every downstream artifact in variant-a-unguarded-n100/ so the headline numbers reflect only the 96 cases that actually ran: - inference_set.jsonl: 100 -> 96 rows - scores.jsonl: 100 -> 96 rows - .viewer/viewer_audit_rows.json: 50 -> 46 (the 4 were scenarios) - .viewer/viewer_prompt_rows.json: 50 -> 50 (unaffected) - .viewer/viewer_score_index.json: byte offsets rebuilt - .viewer/viewer_transcript_index.json: byte offsets rebuilt - .viewer/viewer_run_manifest.json: source_files size_bytes patched - metrics.json: stages.judge.calls 100 -> 96 (skipped rows weren't actually judged) Recomputed unguarded headline from the surviving 96 scored rows: policy_violation 42% (was 39%), overrefusal 4% (was 2%). README updated to reflect n=96 for unguarded and n=100 for ACS-guarded. Follow-up suggestion: fix chat_unguarded / chat_guarded_acs to not call asyncio.run from within a thread. Either restructure as a coroutine and let the caller await it, or use a long-lived loop via asyncio.new_event_loop()/loop.run_until_complete() guarded by a lock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(bank-manager-acs): rename bank_manager_agent_shield to bank_manager_agent_control Matches the broader ACS = Agent Control Specification framing. Folder rename is via git mv to preserve blame. All in-repo references (Python imports, YAML targets, frozen artifact configs, viewer rows, README, scripts/render_trade_off.py) updated: - 712 underscore-form occurrences (bank_manager_agent_shield) - 12 dash-form occurrences (bank-manager-agent-shield) — suite IDs inside taxonomy.json, config.yaml, and README copy commands Internal artifact directory names (variant-a-unguarded-n100, variant-e-guarded-acs-n100) are unchanged — those are run names, not suite names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(bank-manager-acs): remap test_set behaviors to formal taxonomy names The systematize stage regenerated taxonomy.json with 4 formal category names but test_set.jsonl + downstream artifacts retained the older descriptive names, causing the viewer to 500 at viewer/src/lib/server/data.ts:184 (behaviorDefinition strict lookup). Maps every test case's dimensions.behavior + top-level behavior fields to the canonical formal name across: - examples/bank_manager_agent_control/results/test_set.jsonl - per-variant inference_set.jsonl, scores.jsonl, .viewer/*.json Mapping: authorized banking tool use -> Authorization-gated action handling accurate financial statements ... -> Correct tool use and customer-facing banking help confidentiality and social-engineering -> split per-case into Confidential data withholding OR Resistance to impersonation and prompt injection Each of the 4 formal categories now has >= 1 test case (the 4th was empty before). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(bank-manager-acs): rename assert-eval -> assert-ai in demo README Catches up the demo README to the post-rebrand CLI binary name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(bank-manager-acs): catch up demo to post-rebrand naming (assert-ai, ASSERT_AI_*) After rebasing onto main, sweeps the demo for stale terminology: - p2m run -> assert-ai run (in YAML comments + README) - p2m/stages/* -> assert_ai/stages/* (in YAML comments) - ASSERT_EVAL_* -> ASSERT_AI_* (env vars) - assert_eval -> assert_ai (module paths, if any) - assert-eval -> assert-ai (CLI binary) - drops references to eval_guarded_v2.yaml / eval_guarded_v3.yaml (now eval_guarded_acs.yaml only) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(bank-manager-acs): add Phoenix 2-line auto-instrumentation to agent.py Drops in the canonical Phoenix auto_instrument pattern at the top of agent.py so LangChain / OpenAI / MCP tool calls flow into Phoenix without any framework config. Optional via try/except — demo still runs without arize-phoenix-otel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(viewer/metrics): suppress permissibility-split cards when taxonomy is single-sided The "Permissible requests failed" / "Not-permissible requests failed" card pair only makes sense when the taxonomy has BOTH permissible and not-permissible behaviors. For single-sided taxonomies the not-aligned bucket would render an empty "no relevant judgments" tile that's noise. Now returns null/null when the permissibility index has < 2 distinct values, so the run-detail page's `policyViolationOnPermissible || policyViolationOnNotPermissible` gate naturally skips the section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(bank-manager-acs): add variant-c prompt-engineering intervention Adds chat_unguarded_prompted callable that runs the same raw LangGraph agent as chat_unguarded but with a defensive addendum appended to the system prompt (no tool gating). Reuses the suite-root frozen test_set + same judge dims for a fair 3-way comparison: bare baseline (a) vs prompt-engineering (c) vs ACS-guarded (e). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Minsoo Thigpen <mithigpe@microsoft.com> Co-authored-by: changliu2 <99364750+changliu2@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: tangym <tangym@users.noreply.github.com> Co-authored-by: sooyeonni <29706402+sooyeonni@users.noreply.github.com> Co-authored-by: sooyeonni <dusl1209@naver.com> Co-authored-by: Minsoo Thigpen <minsoo.thigpen@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Mike Shi <peichengshi@microsoft.com> Co-authored-by: Chang Liu <changliu2@microsoft.com>

This was referenced Jun 1, 2026

build: wheel-safe packaging, CI build workflow, env-var rename #182

Merged

docs(prompts): rename "seed" -> "test case" in internal_pipeline_prompts/ #188

Merged

changliu2 and others added 8 commits June 1, 2026 01:58

README: add inline Quick install block

92409e9

Surface a copy-pasteable install/run snippet above the 4-column 'Get started' table so first-time readers don't have to click through to find the bootstrap commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

changliu2 force-pushed the changliu2/docs-examples-rename-leftovers-and-jargon-scrub branch from 5cdfd1c to f4de998 Compare June 1, 2026 05:59

changliu2 merged commit 70d0984 into main Jun 1, 2026
12 checks passed

changliu2 deleted the changliu2/docs-examples-rename-leftovers-and-jargon-scrub branch June 1, 2026 06:04

Copilot AI mentioned this pull request Jun 1, 2026

docs: metrics.json is token-usage telemetry, not aggregate eval rates — no rename #192

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs+examples: post-rename cleanup, jargon scrub, broken nav fixes - #181

docs+examples: post-rename cleanup, jargon scrub, broken nav fixes#181
changliu2 merged 8 commits into
mainfrom
changliu2/docs-examples-rename-leftovers-and-jargon-scrub

changliu2 commented Jun 1, 2026

Uh oh!

changliu2 commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

changliu2 commented Jun 1, 2026

TL;DR

What changed

How to verify

Out of scope (intentionally deferred)

Uh oh!

changliu2 commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant