build: wheel-safe packaging, CI build workflow, env-var rename by tangym · Pull Request #182 · responsibleai/ASSERT

tangym · 2026-06-01T04:59:18Z

Summary

Hardens packaging and the build/release surface for assert-ai so the wheel is publishable and installable cleanly, and standardizes the env-var prefix.

Commits (small + ordered)

refactor(prompts): move internal-pipeline-prompts/ under assert_ai/ as a subpackage so prompt assets ship inside the wheel instead of being a sibling repo-only directory.
fix(io): resolve PROMPTS_DIR via importlib.resources (files()) so prompt lookup works whether the package is installed from a wheel, an editable install, or sdist — no reliance on __file__ path math.
build: drop examples* from packages.find include so the examples/ tree is no longer accidentally packaged into the wheel.
build(pyproject): flesh out package metadata — license, authors, keywords, classifiers, project URLs, and explicit package-data for the bundled prompt .md files. Required for a publishable wheel and PyPI listing.
ci: add .github/workflows/build.yml — PEP 517 build + twine check on every PR/push, plus a 3-OS × 3-Python (3.11/3.12/3.13) pip install <wheel> + assert-ai --help smoke matrix. Uploads dist/ as an artifact (14d on PR, 90d on main).
chore(env): rename ASSERT_EVAL_* env vars to ASSERT_AI_* to match the package rename in Rename assert-eval -> assert-ai #177. Includes the matching string update in viewer/src/lib/server/run-spawn.ts for ASSERT_AI_COMMAND so the viewer's spawn override stays consistent.
docs(examples): fix stale P2M_* env-var names in example READMEs so docs match the code.

Verification

python -m build → clean wheel + sdist
twine check dist/* → PASSED
Wheel contains the 12 packaged prompt .md files
Smoke install + assert-ai --help works in a clean venv
Guard test tests/test_no_p2m_references.py passes
All commits authored as tangym

Follow-up

Tracked as a separate issue: one internal TypeScript symbol rename in the viewer (spawnAssertEvalRun/resolveAssertEvalCommand → *AssertAi*). Kept out of this PR to avoid mixing viewer-side changes with packaging work.

…package Move the prompt template directory inside the package so it ships inside the wheel as data alongside the importable code. Pure file move plus an empty __init__.py so importlib.resources can discover it as a real subpackage. The next commit switches io.py to load via importlib.resources; reads from the old top-level directory will start failing after that change.

The previous implementation derived the prompts directory from `Path(__file__).resolve().parents[2]`, which works in a repo checkout but lands inside `site-packages/` for a wheel install — where the top-level `internal-pipeline-prompts/` directory does not exist. Switch to `importlib.resources.files('assert_ai.internal_pipeline_prompts')` so resolution works both from source and from an installed wheel. The returned Traversable still supports `/`, `.read_text()`, `.read_bytes()`, and `.is_file()`, which is everything the existing call sites need. `artifact_cache._prompt_descriptor` is updated to use `.is_file()` instead of `.exists()` since Traversable does not guarantee `.exists()`.

`examples*` was previously included in `packages.find`, which made the wheel ship every example agent and bloat the distribution. The examples are not importable Python packages from `assert_ai` — they are standalone walkthroughs that users run from a repo checkout. They should not be in the wheel.

…package-data Adds the metadata PyPI surfaces on the project page and uses for discovery / filtering, plus the package-data declarations needed to ship the prompt templates inside the wheel: - `license = { file = 'LICENSE' }` (MIT, already at repo root) - `authors` with the team display name; email is left as a TODO until a public contact alias is available (does not block publishing) - `keywords` (safety, evaluation, llm, agent, responsible-ai) - `classifiers` covering Development Status (Beta), MIT license, Python 3.11/3.12/3.13, and AI / QA / Testing topics - `[project.urls]` (Homepage, Repository, Issues, Documentation) - `[tool.setuptools] include-package-data = true` so any future data files in tracked packages ship in the wheel - `[tool.setuptools.package-data]` explicitly listing `assert_ai.internal_pipeline_prompts = ['*.md']`. Without this, `include-package-data` alone is a no-op for plain setuptools (no MANIFEST.in, no setuptools-scm), and the .md prompt files would be excluded from the wheel — silently re-introducing the bug the previous commit fixes.

…oke) `build.yml` runs on every push to main, every PR to main, and on `workflow_dispatch`. Two jobs: 1. `build` (ubuntu-latest, Python 3.11): - `python -m build` (PEP 517 sdist + wheel) - `python -m twine check dist/*` (verifies long-description renders for PyPI) - uploads `dist/` as a workflow artifact with conditional retention: 14 days for PR builds, 90 days for main and dispatch builds so a merged commit's wheel stays available for downstream consumers. 2. `test-install` (3x3 matrix: ubuntu / macos / windows x Python 3.11 / 3.12 / 3.13, fail-fast off): - downloads the wheel artifact - installs it into a fresh environment - runs `assert-ai --help` to prove the entry point resolves and the package + bundled prompts import successfully. This is the runtime regression net for the wheel-install bug fixed in the previous commit, and it runs purely against the built wheel (not the repo checkout) so any `Path(__file__).parents` style regression will fail the matrix instead of slipping into a release. `permissions: contents: read` only - this workflow never writes back to the repo and never talks to PyPI.

Clean break to match the package name (assert-ai). After the package was renamed from assert-eval to assert-ai, users would naturally reach for ASSERT_AI_* env vars and silently get nothing under the old prefix. Renamed (no compatibility shim): - assert_ai/cli.py: Click auto_envvar_prefix ASSERT_EVAL -> ASSERT_AI (so e.g. ASSERT_AI_CONFIG=... wires the --config flag) - viewer/src/lib/server/run-spawn.ts: ASSERT_EVAL_COMMAND override and its log/error strings -> ASSERT_AI_COMMAND - examples/science_research_agent/tools.py and README.md: ASSERT_EVAL_REAL_TOOLS_NOCACHE -> ASSERT_AI_REAL_TOOLS_NOCACHE Follow-up to PR review feedback on the package rename PR.

The example agents read ASSERT_AZURE_DEPLOYMENT and ASSERT_TARGET_MODEL, but three READMEs still documented the legacy P2M_* names from before the package rename. That left users setting the wrong variable and silently getting the default model. Aligns README docs with the code: - examples/travel_planner_langgraph/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT (inline comment + var table) - examples/phoenix_auto_trace/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT - examples/travel_planner_neurosan/README.md: P2M_TARGET_MODEL -> ASSERT_TARGET_MODEL Docs-only; no behavior change. tests/test_no_p2m_references.py still passes (the guard uses \bp2m\b which doesn't catch P2M_* tokens).

changliu2

LGTM 🚀 — clean closure of three follow-up items I had queued up for tomorrow:

M-2 (wheel safety): the importlib.resources.files() approach for PROMPTS_DIR is exactly the right fix — keeps the package importable from a wheel without any path gymnastics. Moving internal-pipeline-prompts/ under assert_ai/internal_pipeline_prompts/ also makes the package self-contained.
Env-var consistency: thanks for catching viewer/src/lib/server/run-spawn.ts and assert_ai/tools.py — those were on my list. Documentation in the 4 example READMEs also caught up.
CI build smoke (PEP 517 × 3-OS × 3-Python): great addition. This is table-stakes hygiene that several best-in-class OSS projects do and we were missing.

Heads up — I have an open follow-up PR (#181) that touches some of the same 4 example READMEs (different hunks: jargon cleanup + telemetry framing). I'll rebase #181 on top of this after it merges and drop any conflicting hunks. No action needed from you.

A couple of optional follow-ups I'll pick up separately (not blocking this PR):

CHANGELOG / README migration note for ASSERT_EVAL_* → ASSERT_AI_* so existing users find the rename
TypeScript symbol rename in viewer: spawnAssertEvalRun / resolveAssertEvalCommand → spawnAssertAi* (the env-var values are right, the function names still carry the old prefix)

Merging.

…185) Documents the assert_eval -> assert_ai package/CLI rename (PR #177) and the ASSERT_EVAL_* -> ASSERT_AI_* env var rename (PR #182) so existing preview users have a clear migration path. Keep a Changelog 1.1.0 format for CHANGELOG.md. README gets a short [!IMPORTANT] callout near the top. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…d to AssertAi* (#187) Catches up TS symbol names with PR #177 (assert_eval -> assert_ai package rename) and PR #182 (ASSERT_EVAL_* -> ASSERT_AI_* env var rename). The viewer function names still carried the old prefix - this completes the rename across the TS surface. No behavior change. Runs npm run check clean (same 3 pre-existing errors as documented in PR #180; zero new errors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…badges) (#186) Bundle D for public-preview readiness. Adds: - .github/ISSUE_TEMPLATE/{bug_report.yml,feature_request.yml,config.yml}: form-style issue templates with required-field validation, secret-redaction reminders, and discussions link in config.yml. - .github/PULL_REQUEST_TEMPLATE.md: short PR template with summary, motivation, testing notes, and a brief checklist. - CODEOWNERS: placeholder catch-all rule pointing to @responsibleai/assert-maintainers (Chang to update team handle). - README badges: CI build status (from PR #182's build.yml workflow), supported Python versions (3.11/3.12/3.13 matching CI matrix), license. PyPI badge intentionally omitted until the package is published. No code change. No CHANGELOG.md (parallel PR delivers that). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…pts/ (#188) Customer-facing terminology cleanup (M-3 from prior audit). The renames in PR #181 originally targeted `internal-pipeline-prompts/` but those edits had to be dropped during rebase since PR #182 git mv-ed the directory under `assert_ai/internal_pipeline_prompts/`. Re-applying against the new path. Scope: prompt markdown files only. Python code comments retain internal terminology per project convention. One occurrence intentionally preserved ("seed config via --from" in init_system.md L436 — refers to a starter config, not a test case). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: normalize commands and path separators * fix(viewer): truncate long callable target labels in compare view (#160) The compare page derived a short label from `run.model` by splitting on `/` only (lines 301, 376). When the target is a Python callable like `examples.bank_manager_demo.agent:chat_unguarded`, that returned the entire dotted path and overflowed the per-run card body (line 211) plus the "By behavior category" column headers — two adjacent header cells visibly crashed into each other. Adds a small `runLabel` helper that splits on '':'' first (callable targets) then `/` (provider/model paths). The card body now wraps the short label in a `truncate + title` tooltip so the full path is still visible on hover. Splits off the label-fix portion of the original PR #78 (which also attempted a 3-way URL fix that's now stale). The 3-way URL plumbing is out of scope here and can ship in a separate PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * init: fix interview flow (context vs system_prompt, judge dimensions, default model) + consolidate YAML emission rules (#176) * init: add --default-model CLI flag and surface design-agent model to LLM - New --default-model option on `assert-eval init` lets the user pre-seed the pipeline.default_model hint for the interview. - _build_default_model_hint() emits a system-prompt section when the hint is provided. - run_design_loop() always tells the LLM which model is driving the design conversation, and conditionally surfaces the default_model hint as the first user-visible message. This is the plumbing the prompt update relies on; no behavior change without the prompt edit that follows. * prompt(init): separate context from system_prompt, restructure judge, require default_model - Section 1 renamed 'Application Context' and explicitly distinguishes developer narrative from target.system_prompt for hosted-model targets. - New Section 3 'Pipeline Default Model' rules: never silently copy the target model into default_model; ask explicitly. - Renumbered Behavior (4), Test Set (5), Judge (6). - Judge built-ins: policy_violation, overrefusal. - Custom judge dimensions now use ONE consolidated turn for name + description + rubric-true + rubric-false (instead of 4 separate turns). - Added explicit guideline against conflating context with target.system_prompt. * tests(init): cover --default-model plumbing and prompt anchors - test_prompt_contains_required_section_anchors: low-resolution canary that guards against accidental deletion of the new sections during future prompt refactors. Not a behavior test — its docstring tells future contributors they can reword freely as long as anchors stay. - test_prompt_includes_default_model_hint_when_provided: exercises _build_default_model_hint() end-to-end via build_system_message(). - test_design_agent_surfaces_model_hint_to_llm: CLI->design loop integration test asserting the first user message names both the design-agent model and the --default-model hint. * prompt(init): comment out per-stage model examples in schema reference Schema-reference YAML examples in init_system.md were showing live 'model:' blocks under systematize, test_set.prompt, test_set.scenario, inference.tester, and judge. The design agent treats these as copy-paste templates and emits them as live YAML, which then overrides default_model silently. Comment out every per-stage 'model:' example in the schema, add an explicit Section 3 prohibition on emitting uncommented per-stage 'model:' blocks unless the user explicitly asked for an override, and update the Discoverable defaults guideline to require commented-only surfacing of per-stage overrides. * prompt(init): fix tester-toggle guideline to use commented model example The tester-toggle Guidelines bullet was the last place in the prompt showing a live, uncommented per-stage 'model:' block. Even with Section 3 prohibiting live per-stage model overrides, the design agent still copied this bullet's YAML verbatim into proposals, causing 'tester:\n model:\n name: ...' to leak into generated configs. Bring the bullet in line with the schema-reference examples by commenting out the model override and keeping only the bare 'tester:' key live. * prompt(init): consolidate YAML emission rules into one section The per-stage 'model:' rule, the tester-toggle, target.trace, and '# customize:' / '# review:' conventions were each repeated and lightly contradicted across Section 3, the schema reference, and the Guidelines list. The LLM kept rediscovering uncommented per-stage 'model:' templates because the rule had no single home. Consolidate all YAML emission rules into a new top-level '# YAML emission rules' section between '# Config Structure' and '# Guidelines' with four sub-sections (per-stage models, tester block, target.trace, customization hints). Trim Section 3 to the ask-phase conversation flow only and cross-reference the new section. Drop the duplicated tester/target.trace/customization bullets from Guidelines. Also two cosmetic fixes that surfaced during the audit: - Section 'Pacing' said 'all 5 sections' but there are 6 ask sections. - 'Test Set Dimensions' was at heading level 5 (#####) while everything else at that depth uses level 4 (####). * rename: assert_eval -> assert_ai (module dir + pyproject) Renames the package directory and updates the five distribution-name and module-name references in pyproject.toml (name, [project.scripts], all extra, dev dependency-group, and setuptools packages.find). Also drops examples* from the wheel per the PyPI publishing plan. Imports inside the package are not yet updated -- follow-up commit. * rename: update intra-package imports to assert_ai Replaces assert_eval / assert-eval references inside the moved package so it imports cleanly. Covers from/import statements, dotted module strings (telemetry tags, error messages), and internal task/thread names (assert-eval-watchdog -> assert-ai-watchdog, etc.). Tests, docs, scripts, and CI updated separately. * rename: update tests for assert_ai Replaces assert_eval / assert-eval references in tests/ — module patches, dotted import paths, CLI invocations, and any expected log/identifier strings. * rename: update top-level assert_ai modules Catches the depth-1 .py files in assert_ai/ that the package-imports commit missed: cli.py, config.py, display.py, results.py, runner.py, viewer_read_model.py. Same dual-pattern (assert_eval -> assert_ai, assert-eval -> assert-ai). * rename: update docs, examples, scripts, viewer, and website Sweeps assert_eval -> assert_ai and assert-eval -> assert-ai across user-facing surfaces: root README and AGENTS, docs/ tree, runnable examples (including notebooks and agent scripts), helper scripts, the SvelteKit viewer, the marketing website, and the analysis README that ships inside the wheel. * rename: update CI workflow path filter Path filter in .github/workflows/regression.yml now points at assert_ai/** instead of assert_eval/**. * rename: update .gitignore policy artifact pattern Renames assert_eval_policy.* -> assert_ai_policy.* so runtime policy artifacts stay ignored after the package rename. * chore: rename root package in uv.lock to assert-ai Hand-edits the 4 self-references to the root project in uv.lock to match the renamed distribution. uv lock cannot regenerate cleanly today because of a pre-existing dspy-ai>=2.7,<3 vs >=3 conflict on main; a full lockfile refresh will land separately once that is resolved. * docs: replace stale microsoft/adaptive-eval URLs with responsibleai/ASSERT (#178) The microsoft/adaptive-eval slug 404s after the rename to ASSERT. Two customer-facing docs still pointed at the dead URL: - CONTRIBUTING.md: dev-setup clone snippets (bash + powershell) - AGENTS.md: paste-in prompt block that downstream LLMs hand to users Both now point at https://github.com/responsibleai/ASSERT. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(website): use hyphenated assert-ai CLI in terminal demo The CLI entry point in pyproject.toml is 'assert-ai' (hyphen), but the website's terminal typing animation showed 'assert_ai run', which would mislead users copying the command. Reported by Copilot review on PR #177. * build: restore examples* in packages.find include The previous include list was ["assert_eval*", "examples*"]; the rename commit accidentally dropped 'examples*'. Tests import from 'examples.agents.*' (test_tool_module_sandbox, test_openclaw_driver), and the editable/wheel install needs to continue exposing the examples package to preserve current behavior. Packaging cleanup (e.g. removing examples from the published wheel) will be handled in a follow-up PR, not as part of this mechanical rename. Reported by Copilot review on PR #177. * docs(assets): rename framework diagram to assert-ai-framework-diagram.png README.md was updated to reference assets/assert-ai-framework-diagram.png but the asset file itself was not renamed, breaking the image in the rendered README. Reported by Copilot review on PR #177. * Website: partner quotes section, expanded framework logo loop, sidebar polish * refactor(prompts): move internal-pipeline-prompts under assert_ai subpackage Move the prompt template directory inside the package so it ships inside the wheel as data alongside the importable code. Pure file move plus an empty __init__.py so importlib.resources can discover it as a real subpackage. The next commit switches io.py to load via importlib.resources; reads from the old top-level directory will start failing after that change. * fix(io): resolve PROMPTS_DIR via importlib.resources The previous implementation derived the prompts directory from `Path(__file__).resolve().parents[2]`, which works in a repo checkout but lands inside `site-packages/` for a wheel install — where the top-level `internal-pipeline-prompts/` directory does not exist. Switch to `importlib.resources.files('assert_ai.internal_pipeline_prompts')` so resolution works both from source and from an installed wheel. The returned Traversable still supports `/`, `.read_text()`, `.read_bytes()`, and `.is_file()`, which is everything the existing call sites need. `artifact_cache._prompt_descriptor` is updated to use `.is_file()` instead of `.exists()` since Traversable does not guarantee `.exists()`. * build: drop examples* from packages.find include `examples*` was previously included in `packages.find`, which made the wheel ship every example agent and bloat the distribution. The examples are not importable Python packages from `assert_ai` — they are standalone walkthroughs that users run from a repo checkout. They should not be in the wheel. * build(pyproject): add license, authors, keywords, classifiers, urls, package-data Adds the metadata PyPI surfaces on the project page and uses for discovery / filtering, plus the package-data declarations needed to ship the prompt templates inside the wheel: - `license = { file = 'LICENSE' }` (MIT, already at repo root) - `authors` with the team display name; email is left as a TODO until a public contact alias is available (does not block publishing) - `keywords` (safety, evaluation, llm, agent, responsible-ai) - `classifiers` covering Development Status (Beta), MIT license, Python 3.11/3.12/3.13, and AI / QA / Testing topics - `[project.urls]` (Homepage, Repository, Issues, Documentation) - `[tool.setuptools] include-package-data = true` so any future data files in tracked packages ship in the wheel - `[tool.setuptools.package-data]` explicitly listing `assert_ai.internal_pipeline_prompts = ['*.md']`. Without this, `include-package-data` alone is a no-op for plain setuptools (no MANIFEST.in, no setuptools-scm), and the .md prompt files would be excluded from the wheel — silently re-introducing the bug the previous commit fixes. * ci: add build.yml workflow (PEP 517 build + cross-platform install smoke) `build.yml` runs on every push to main, every PR to main, and on `workflow_dispatch`. Two jobs: 1. `build` (ubuntu-latest, Python 3.11): - `python -m build` (PEP 517 sdist + wheel) - `python -m twine check dist/*` (verifies long-description renders for PyPI) - uploads `dist/` as a workflow artifact with conditional retention: 14 days for PR builds, 90 days for main and dispatch builds so a merged commit's wheel stays available for downstream consumers. 2. `test-install` (3x3 matrix: ubuntu / macos / windows x Python 3.11 / 3.12 / 3.13, fail-fast off): - downloads the wheel artifact - installs it into a fresh environment - runs `assert-ai --help` to prove the entry point resolves and the package + bundled prompts import successfully. This is the runtime regression net for the wheel-install bug fixed in the previous commit, and it runs purely against the built wheel (not the repo checkout) so any `Path(__file__).parents` style regression will fail the matrix instead of slipping into a release. `permissions: contents: read` only - this workflow never writes back to the repo and never talks to PyPI. * chore(env): rename ASSERT_EVAL_* env vars to ASSERT_AI_* Clean break to match the package name (assert-ai). After the package was renamed from assert-eval to assert-ai, users would naturally reach for ASSERT_AI_* env vars and silently get nothing under the old prefix. Renamed (no compatibility shim): - assert_ai/cli.py: Click auto_envvar_prefix ASSERT_EVAL -> ASSERT_AI (so e.g. ASSERT_AI_CONFIG=... wires the --config flag) - viewer/src/lib/server/run-spawn.ts: ASSERT_EVAL_COMMAND override and its log/error strings -> ASSERT_AI_COMMAND - examples/science_research_agent/tools.py and README.md: ASSERT_EVAL_REAL_TOOLS_NOCACHE -> ASSERT_AI_REAL_TOOLS_NOCACHE Follow-up to PR review feedback on the package rename PR. * docs(examples): fix stale P2M_* env-var names in READMEs to match code The example agents read ASSERT_AZURE_DEPLOYMENT and ASSERT_TARGET_MODEL, but three READMEs still documented the legacy P2M_* names from before the package rename. That left users setting the wrong variable and silently getting the default model. Aligns README docs with the code: - examples/travel_planner_langgraph/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT (inline comment + var table) - examples/phoenix_auto_trace/README.md: P2M_AZURE_DEPLOYMENT -> ASSERT_AZURE_DEPLOYMENT - examples/travel_planner_neurosan/README.md: P2M_TARGET_MODEL -> ASSERT_TARGET_MODEL Docs-only; no behavior change. tests/test_no_p2m_references.py still passes (the guard uses \bp2m\b which doesn't catch P2M_* tokens). * docs: add CHANGELOG.md + README migration note for assert_ai rename (#185) Documents the assert_eval -> assert_ai package/CLI rename (PR #177) and the ASSERT_EVAL_* -> ASSERT_AI_* env var rename (PR #182) so existing preview users have a clear migration path. Keep a Changelog 1.1.0 format for CHANGELOG.md. README gets a short [!IMPORTANT] callout near the top. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(viewer): rename spawnAssertEvalRun / resolveAssertEvalCommand to AssertAi* (#187) Catches up TS symbol names with PR #177 (assert_eval -> assert_ai package rename) and PR #182 (ASSERT_EVAL_* -> ASSERT_AI_* env var rename). The viewer function names still carried the old prefix - this completes the rename across the TS surface. No behavior change. Runs npm run check clean (same 3 pre-existing errors as documented in PR #180; zero new errors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: add community-launch hygiene (issue/PR templates, CODEOWNERS, badges) (#186) Bundle D for public-preview readiness. Adds: - .github/ISSUE_TEMPLATE/{bug_report.yml,feature_request.yml,config.yml}: form-style issue templates with required-field validation, secret-redaction reminders, and discussions link in config.yml. - .github/PULL_REQUEST_TEMPLATE.md: short PR template with summary, motivation, testing notes, and a brief checklist. - CODEOWNERS: placeholder catch-all rule pointing to @responsibleai/assert-maintainers (Chang to update team handle). - README badges: CI build status (from PR #182's build.yml workflow), supported Python versions (3.11/3.12/3.13 matching CI matrix), license. PyPI badge intentionally omitted until the package is published. No code change. No CHANGELOG.md (parallel PR delivers that). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(prompts): rename "seed" -> "test case" in internal_pipeline_prompts/ (#188) Customer-facing terminology cleanup (M-3 from prior audit). The renames in PR #181 originally targeted `internal-pipeline-prompts/` but those edits had to be dropped during rebase since PR #182 git mv-ed the directory under `assert_ai/internal_pipeline_prompts/`. Re-applying against the new path. Scope: prompt markdown files only. Python code comments retain internal terminology per project convention. One occurrence intentionally preserved ("seed config via --from" in init_system.md L436 — refers to a starter config, not a test case). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs+examples: post-rename cleanup, jargon scrub, broken nav fixes (#181) * tests: tighten p2m guard regex and skip binary/lockfile false positives The previous \\b p2m \b\ word-boundary regex missed env-var leaks like P2M_AZURE_DEPLOYMENT because the underscore broke the word boundary. Switched to a case-insensitive substring match so P2M_* env vars and other prose leaks are caught. To keep the test signal clean, also skip: - Binary file extensions (.svg, image formats) that may embed base64. - Lockfiles (package-lock.json, yarn.lock, pnpm-lock.yaml, uv.lock, poetry.lock) where sha512 hashes coincidentally contain 'p2m'. Added a sanity test that asserts the regex catches P2M_AZURE_DEPLOYMENT. Deleted an orphan website/public/icons/P2M Thumbnail.svg (no references). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/docs: replace internal model names with public substitutes Replaced internal-only model names with publicly available substitutes across examples, docs, and the website snippets so customers can run copy-paste configs without hitting unknown-model errors. - azure/gpt-5.4-mini -> azure/gpt-4o-mini - azure/gpt-5.4-nano -> azure/gpt-4o-mini - azure/gpt-5.4 -> azure/gpt-4o - GPT-5-nano (prose) -> gpt-4o-mini - GPT-5-railfree references removed (publishable substitute does not exist) Scope deliberately excludes assert_ai/ core code, tests/, .github/ workflows, and artifacts/results/** (frozen historical records). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * AGENTS.md: complete rename to ASSERT (branding + dead doc links) PR #178 fixed the GitHub URL but the prose, title, and doc cross-links still referenced 'Adaptive Eval' and pointed to renamed/moved doc paths. - AGENTS.md title and prose: 'Adaptive Eval' -> 'ASSERT' - Dead doc links updated: - docs/quickstart.md -> docs/getting-started.md - docs/writing-eval-specs.md -> docs/guides/create-evaluation.md - docs/reading-results.md -> docs/guides/results.md - Same branding fix applied to .cursorrules, .devcontainer/devcontainer.json, CONTRIBUTING.md, SUPPORT.md, and examples/azure_doc_qa/IMPROVEMENT_JOURNEY.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/change_control_agent: genericize internal Microsoft infra references The change_control_agent example referenced internal Microsoft deployment/ops systems (Safefly, Ev2, R2D, ICM, ADO, ChangeKeep) which are unfamiliar to external customers and leak internal context. Renamed to generic enterprise vocabulary so the example reads as a generic change-control pattern: - submit_to_safefly -> submit_to_deployment_gateway - submit_to_ev2 -> submit_to_rollout_service - submit_to_r2d -> submit_to_release_readiness - create_ado_change_request -> create_change_request - get_icm_incident -> get_incident - ChangeKeep -> ChangeFlow - SAFEFLY-<id> -> DEPLOYGATE-<id> - 'Safefly', 'Ev2', 'R2D' -> 'Deployment Gateway', 'Rollout Service', 'Release Readiness' - 'ADO', 'ICM' -> 'change-tracker', 'incident-tracker' - 'internal change-management assistant' -> 'change-management assistant' Both function names and the string-literal handles used as dict keys/ identifiers in tools.py were renamed so the example remains functional. README updated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/incident_triage_agent: rewrite README opening for customer context The README opening framed this example as an 'ACS efficacy demo' with an 'A -> C demo path', cross-linked an internal draft PR (#88), and referenced a bank-manager demo that does not ship in this repo. Rewrote the title, opening paragraph, and TL;DR to present the example as a generic incident-triage agent evaluation. Preserved the variant tables, eval-config matrix, and provenance section unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * examples/README: fix broken paths, add missing example entries - Quickstart YAML path: pipes/ -> prompt_agents/ (the pipes/ directory was renamed long ago; the README pointed at a 404). - Added table rows for four examples that exist on disk but were not listed: travel_planner_neurosan, change_control_agent, azure_doc_qa, benchmark. - Fixed dead doc links: docs/reference/cli.md -> docs/cli/commands.md. - Removed broken link to docs/case-study-incident-triage-joint.md (file does not exist). - Updated the layout block to match the current directory structure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * README: add inline Quick install block Surface a copy-pasteable install/run snippet above the 4-column 'Get started' table so first-time readers don't have to click through to find the bootstrap commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * scripts + pyproject: misc jargon and dead-reference cleanup - scripts/migrate_artifacts_to_pr23_vocab.py: docstring no longer references 'PR #23 vocabulary'; describes the migration in customer- neutral terms. - pyproject.toml: trimmed dspy pin comment to drop the bank-manager reference (that demo does not ship in this repo). - scripts/README.md, scripts/scenario_failure_prediction.py: replaced the relationship-entanglement-v1 suite name with the placeholder <your-suite-name> so examples are generic. - scripts/judge_stability_experiment.py: dropped internal model name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: clean-slate the public-facing surface — drop private-preview migration callouts (#189) The repo should read like a clean v1 launch, not a project mid-migration. Private-preview era transitions (assert_eval -> assert_ai, ASSERT_EVAL_* -> ASSERT_AI_*, prompts directory move) are noise in public docs because there were no public releases to break. Existing private-preview users are notified out-of-band. - README.md: drops the [!IMPORTANT] migration callout from PR #185. Added a bundled-viewer bullet to the capability list. - CHANGELOG.md: rewritten to empty [Unreleased] scaffold (Keep a Changelog 1.1.0), ready to populate when v0.1 ships. - .github templates: replaced stale `assert-eval` placeholders with `assert-ai` so contributor-facing examples match the current CLI. No code change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(tests): make test_runtime_safety subprocess inherit parent sys.path after assert_ai rename (#190) The subprocess invocation in tests/test_runtime_safety.py::test_run_stage_coro_does_not_block_subprocess_exit_when_worker_leaked spawned a fresh sys.executable but did not forward the parent interpreter's import paths. In CI, pip install -e . populates site-packages so import assert_ai works inside the subprocess; in any environment where pytest is the only thing putting the project root on sys.path (developer running pytest without first installing, or a leftover venv from before the rename), the subprocess hits ModuleNotFoundError: No module named 'assert_ai' and the test fails before it can even exercise the leaked-worker shutdown path. Fix: build PYTHONPATH from the parent's sys.path and pass it via env= to subprocess.run. Works whether assert_ai is editable-installed or only discovered through pytest's rootdir hook. Pre-existing failure since PR #177 (assert_eval -> assert_ai rename) — the import statement was correctly renamed but the underlying env-propagation gap was unmasked once the package name no longer matched any stale install left in dev venvs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: rename CLI/python refs and clean user-facing wording * docs: refine docs index and migration wording * Clean up of migration terminology and doc updates * docs: rename CLI/python refs and clean user-facing wording * docs: refine docs index and migration wording * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Restore init example context in getting started docs * examples(bank-manager): ACS-vs-unguarded comparison demo Self-contained example comparing an unguarded LangGraph bank-manager agent against the same agent guarded by the new Agent Control Specification (ACS) runtime. Two ASSERT callables, two eval configs, two frozen n=100 result snapshots for viewer playback. Contents (scoped to examples/bank_manager_agent_shield/): - agent.py: chat_unguarded + chat_guarded_acs - mcp_server.py: mock banking MCP server - acs/manifest.yaml + acs/policy/bank_manager.rego: stateless ACS policy (SSN input, sensitivity-scoped read/transfer gates, approval/admin-mode gates, prompt-injection scrubber) - eval_unguarded_v2.yaml: baseline (owns systematize+test_set) - eval_guarded_acs.yaml: ACS variant (reuses baseline test_set) - results/: frozen n=100 artifacts for both variants ACS integration uses the idiomatic SDK orchestration helpers: control.run() for input/output gating around the agent execution, control.run_tool() per MCP tool for pre/post tool-call gating. Per-turn state (transfer_approved, admin_mode_active, account_sensitivity) is tracked by the host wrapper and threaded into each snapshot, since ACS is stateless by design. Headline n=100 (same test set across both variants): unguarded: safety_violation 39%, unjustified_refusal 2% ACS: safety_violation 9%, unjustified_refusal 2% README documents both usage paths: seed the committed results into artifacts/results/ and view them, or run the full pipeline end-to-end (requires agent_control_specification SDK + opa binary on PATH). * examples(bank-manager): rename judge dims to policy_violation/overrefusal Re-judged the existing n=100 inference outputs against the renamed rubric (no inference re-run). Updated taxonomy.json behavior_categories to match the names referenced by the committed test_set.jsonl so the viewer renders. Regenerated .viewer/ caches. Results (n=100, byte-identical inference): unguarded: policy_violation 40%, overrefusal 4% ACS: policy_violation 5%, overrefusal 15% * chore(bank-manager-acs): rename eval_unguarded_v2.yaml to eval_unguarded.yaml The _v2 suffix is no longer meaningful; drop it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(bank-manager-acs): drop 4 target_error scenarios from unguarded variant The unguarded variant-a-unguarded-n100 artifacts contained 4 scenario rows that failed with stop_reason="target_error": test_case_000053, test_case_000063, test_case_000065, test_case_000090 Root cause: examples/bank_manager_agent_shield/agent.py:309 calls asyncio.run(_run_agent_async(...)) inside a worker thread. asyncio.run creates a new event loop and refuses to nest, so under concurrent scenario load the TaskGroup unwinds with an unhandled-errors crash. These are inference errors, NOT ACS policy blocks. The ACS-guarded variant ran clean at n=100. This commit drops the 4 affected rows from every downstream artifact in variant-a-unguarded-n100/ so the headline numbers reflect only the 96 cases that actually ran: - inference_set.jsonl: 100 -> 96 rows - scores.jsonl: 100 -> 96 rows - .viewer/viewer_audit_rows.json: 50 -> 46 (the 4 were scenarios) - .viewer/viewer_prompt_rows.json: 50 -> 50 (unaffected) - .viewer/viewer_score_index.json: byte offsets rebuilt - .viewer/viewer_transcript_index.json: byte offsets rebuilt - .viewer/viewer_run_manifest.json: source_files size_bytes patched - metrics.json: stages.judge.calls 100 -> 96 (skipped rows weren't actually judged) Recomputed unguarded headline from the surviving 96 scored rows: policy_violation 42% (was 39%), overrefusal 4% (was 2%). README updated to reflect n=96 for unguarded and n=100 for ACS-guarded. Follow-up suggestion: fix chat_unguarded / chat_guarded_acs to not call asyncio.run from within a thread. Either restructure as a coroutine and let the caller await it, or use a long-lived loop via asyncio.new_event_loop()/loop.run_until_complete() guarded by a lock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(bank-manager-acs): rename bank_manager_agent_shield to bank_manager_agent_control Matches the broader ACS = Agent Control Specification framing. Folder rename is via git mv to preserve blame. All in-repo references (Python imports, YAML targets, frozen artifact configs, viewer rows, README, scripts/render_trade_off.py) updated: - 712 underscore-form occurrences (bank_manager_agent_shield) - 12 dash-form occurrences (bank-manager-agent-shield) — suite IDs inside taxonomy.json, config.yaml, and README copy commands Internal artifact directory names (variant-a-unguarded-n100, variant-e-guarded-acs-n100) are unchanged — those are run names, not suite names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(bank-manager-acs): remap test_set behaviors to formal taxonomy names The systematize stage regenerated taxonomy.json with 4 formal category names but test_set.jsonl + downstream artifacts retained the older descriptive names, causing the viewer to 500 at viewer/src/lib/server/data.ts:184 (behaviorDefinition strict lookup). Maps every test case's dimensions.behavior + top-level behavior fields to the canonical formal name across: - examples/bank_manager_agent_control/results/test_set.jsonl - per-variant inference_set.jsonl, scores.jsonl, .viewer/*.json Mapping: authorized banking tool use -> Authorization-gated action handling accurate financial statements ... -> Correct tool use and customer-facing banking help confidentiality and social-engineering -> split per-case into Confidential data withholding OR Resistance to impersonation and prompt injection Each of the 4 formal categories now has >= 1 test case (the 4th was empty before). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(bank-manager-acs): rename assert-eval -> assert-ai in demo README Catches up the demo README to the post-rebrand CLI binary name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(bank-manager-acs): catch up demo to post-rebrand naming (assert-ai, ASSERT_AI_*) After rebasing onto main, sweeps the demo for stale terminology: - p2m run -> assert-ai run (in YAML comments + README) - p2m/stages/* -> assert_ai/stages/* (in YAML comments) - ASSERT_EVAL_* -> ASSERT_AI_* (env vars) - assert_eval -> assert_ai (module paths, if any) - assert-eval -> assert-ai (CLI binary) - drops references to eval_guarded_v2.yaml / eval_guarded_v3.yaml (now eval_guarded_acs.yaml only) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(bank-manager-acs): add Phoenix 2-line auto-instrumentation to agent.py Drops in the canonical Phoenix auto_instrument pattern at the top of agent.py so LangChain / OpenAI / MCP tool calls flow into Phoenix without any framework config. Optional via try/except — demo still runs without arize-phoenix-otel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(viewer/metrics): suppress permissibility-split cards when taxonomy is single-sided The "Permissible requests failed" / "Not-permissible requests failed" card pair only makes sense when the taxonomy has BOTH permissible and not-permissible behaviors. For single-sided taxonomies the not-aligned bucket would render an empty "no relevant judgments" tile that's noise. Now returns null/null when the permissibility index has < 2 distinct values, so the run-detail page's `policyViolationOnPermissible || policyViolationOnNotPermissible` gate naturally skips the section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(bank-manager-acs): add variant-c prompt-engineering intervention Adds chat_unguarded_prompted callable that runs the same raw LangGraph agent as chat_unguarded but with a defensive addendum appended to the system prompt (no tool gating). Reuses the suite-root frozen test_set + same judge dims for a fair 3-way comparison: bare baseline (a) vs prompt-engineering (c) vs ACS-guarded (e). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Minsoo Thigpen <mithigpe@microsoft.com> Co-authored-by: changliu2 <99364750+changliu2@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: tangym <tangym@users.noreply.github.com> Co-authored-by: sooyeonni <29706402+sooyeonni@users.noreply.github.com> Co-authored-by: sooyeonni <dusl1209@naver.com> Co-authored-by: Minsoo Thigpen <minsoo.thigpen@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Mike Shi <peichengshi@microsoft.com> Co-authored-by: Chang Liu <changliu2@microsoft.com>

tangym added 7 commits June 1, 2026 04:06

changliu2 approved these changes Jun 1, 2026

View reviewed changes

changliu2 merged commit 7827a9e into main Jun 1, 2026
12 checks passed

changliu2 deleted the yemingtang/wheel-build-hardening branch June 1, 2026 05:27

changliu2 mentioned this pull request Jun 1, 2026

docs(prompts): rename "seed" -> "test case" in internal_pipeline_prompts/ #188

Merged

changliu2 mentioned this pull request Jun 1, 2026

docs+examples: post-rename cleanup, jargon scrub, broken nav fixes #181

Merged

tangym mentioned this pull request Jun 1, 2026

ci: add release-triggered PyPI publish workflow (OIDC, reviewer-gated) #191

Merged

Copilot AI mentioned this pull request Jun 1, 2026

docs: metrics.json is token-usage telemetry, not aggregate eval rates — no rename #192

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

build: wheel-safe packaging, CI build workflow, env-var rename - #182

build: wheel-safe packaging, CI build workflow, env-var rename#182
changliu2 merged 7 commits into
mainfrom
yemingtang/wheel-build-hardening

tangym commented Jun 1, 2026

Uh oh!

changliu2 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

tangym commented Jun 1, 2026

Summary

Commits (small + ordered)

Verification

Follow-up

Uh oh!

changliu2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants