AgentCheck 0.5.10

AgentCheck 0.5.10 extends verified PydanticAI support through 2.40 and prevents silently discarded execution configuration.

Active root-capability, tool-retry and output-retry context overrides now fail preflight. Registered event hooks and unknown hook state are also refused. Ordinary inactive defaults and restored contexts remain supported.

Installed-artifact qualification checks the specific dynamic-instruction, execution-override and event-hook refusals alongside literal reconstruction and gateway isolation. Generator compatibility remains 4; suite generation semantics are unchanged.

AgentCheck 0.5.9

AgentCheck 0.5.9 preserves PydanticAI 2.36 literal instructions through inspection and sanitized runtime reconstruction. The supported PydanticAI range now extends through 2.36.

Dynamic instructions, templates, structured instruction parts, and unknown wrappers remain refused. Generator compatibility stays at 4.

Installed-artifact qualification adds PydanticAI literal reconstruction/gateway and dynamic-refusal controls alongside the existing semantic checks.

AgentCheck 0.5.8

AgentCheck 0.5.8

JSON argument comparisons now distinguish booleans from numbers, including
nested objects and arrays. Explicit true/1 and false/0 mismatches can no
longer silently pass a behavior contract or select the wrong simulated fixture.

Numeric 1/1.0 equality, top-level behavior matching, recursive fixture subsets,
exact-fixture mode, finite fixtures and oracle authority are preserved. Weak
argument evidence remains inconclusive; missing fixtures do not invent outcomes.
Confirmation, retry-order and world-state policy are unchanged.

Generator compatibility is now 4 because lint admission distinguishes these
argument contracts. Generate and review new suites to adopt that generation
identity. Existing frozen suites retain their recorded fingerprints; stored
runs and baselines are not rewritten. Re-evaluation and execution use corrected
matching, so legacy false passes and incorrectly selected fixture outcomes are
not preserved.

Exact-wheel qualification adds four JSON-type controls while retaining the
previous 18 semantic controls and the existing Trusted Publishing workflow.

AgentCheck 0.5.7

AgentCheck 0.5.7

Generated confirmed-action cases now keep duplicate calls observable throughout
their existing tool-call budget, including prerequisite-adjusted budgets.
Explicitly indexed abstract, stateless success fixtures prevent those calls
from being hidden by generator fixture exhaustion.

Budgets, prerequisite outcomes and confirmation/retry/argument authority are
unchanged. Authored, stateful, error and timeout fixtures are not repeated;
genuine fixture gaps remain infrastructure errors. This does not establish
success of a real tool or change the gateway's single-use behavior.

Generator compatibility is now 3. Generate and review a new suite to adopt
the correction. Existing version2 frozen suites keep their recorded fixtures,
fingerprints and behavior, including legacy fixture gaps. Stored runs and
baselines are not rewritten; no cases or obligations are removed.

Exact-wheel qualification adds three inert-gateway controls for duplicate
observability and genuine authored-fixture exhaustion, retaining the previous
15 semantic controls and the existing publishing workflow.

AgentCheck 0.5.6

AgentCheck 0.5.6

Two evidence-correctness fixes, with strict independent obligations preserved.

• Ambiguous-timeout retry evaluation returns INCONCLUSIVE when a missing
  earlier matching attempt's outcome could change the verdict. Positively observed
  retry violations still FAIL, including when later outcomes are absent.
• Representative argument samples no longer assert an authoritative interpretation
  of an independently authored user request. Sample mismatches remain visible as
  INCONCLUSIVE. Explicit argument contracts, schema errors, confirmation,
  ordering, duplicate-action and retry violations retain their separate authority.

Compatibility and adoption

Generator compatibility changes from 1 to 2 for the authored-request argument
provenance correction. Regenerate and review a suite to adopt that change; existing
frozen suites, stored runs and baselines are not rewritten. The evaluator-only retry
fix does not change generation or denominators. Partial public run records that
previously passed may now be inconclusive; this does not establish that a supported
adapter produced those incomplete records.

The same-wheel release qualification now includes both contracts and their strict
failure controls. Build, installation, hashing, network denial, receipt binding and
Trusted Publishing transport are unchanged. No framework or containment expansion.

Changes: #97,
#98.

AgentCheck 0.5.5 — scenario-aware confirmation and authority-bound evidence

An evidence-correctness patch: scenario-aware confirmation, authority-bound
coverage, and exact-wheel release qualification. The compatibility restrictions
below intentionally refuse evidence that cannot support its recorded claims.
Generation, framework support, containment and existing baselines are unchanged.

Fixed

• Confirmation-before-tool evaluation now distinguishes scenario-supplied,
  tool-scoped consent, deliberate withholding, and absent context. Compliant
  completed withholding and optional action refusal can pass without claiming
  consent was exercised; absent context remains inconclusive with or without
  a call. Positive consent must bind to a delivered user turn before every
  consistently recorded call. Prose, run-only flags, ambiguous scope and
  incomplete no-call evidence cannot produce a pass. This intentionally tightens
  acceptance of historical unbound consent evidence; see docs/behavioral-policies.md.

  Suite identity: unchanged; no generated cases, obligations or denominators removed.

• Stored-run comparison now reports spec_changed when a tool's risk changes
  between inferred and authoritative, even if its resolved risk values and
  recorded coverage digest stay the same. Each run's recorded specification
  supplies the authority; existing artifacts, scenario classifications, and
  comparison exit codes are unchanged.

  Suite identity: unchanged.

• Coverage now binds effective per-axis tool-risk authority using a versioned
  specification digest. Existing checksums, scenario denominators and source
  checks are retained. Unknown digest algorithms fail closed; mixed-algorithm
  comparisons do not report a spec change merely because the algorithm differs.

  Compatibility restriction: older selected-run coverage with an omitted
  reference set cannot prove its recorded authority and is now rejected by
  stored-run consumers. Fully rederivable legacy coverage and summaries that
  originally lacked coverage remain readable. No stored file is rewritten;
  see docs/behavioral-coverage.md for recovery and the remaining reference limits.

  Suite identity: unchanged. New coverage digests/checksums intentionally change.

Release qualification

• Publishing now qualifies the already-built wheel before uploading release
  artifacts. Clean base, OpenAI Agents and PydanticAI extra environments check
  direct-wheel hashes, installed import/version identity, missing-extra refusal
  and small offline confirmation-contract probes. Artifact bytes are checked
  again afterward; publishing still downloads those same artifacts without a
  rebuild. These are release gates, not a substitute for source review or proof
  of arbitrary target behavior.

Source commit: 41ebab5dc5cdb9607269baf2685fe6f554a93c86.

Full changelog: https://github.com/WaseemGhanem98/AgentCheck/blob/v0.5.5/CHANGELOG.md

AgentCheck 0.5.4

A gate-correctness patch. agentcheck gate could return PASS for a target whose declared-destructive tool was never tested. It now refuses, and says which tool, which requirement, and what to do about it.

Fixed

The release gate could pass a target whose declared-destructive tool was never tested. (#91)

The gate decided from executed-case verdicts and the baseline comparison alone, and never consulted behavioral coverage. A tool you explicitly declared state_changing or destructive could therefore have zero generated cases, contribute no verdict, and leave the build green — with nothing in the output saying so.

The gate already held the principle this broke: missing evidence can never be upgraded to PASS. It applied that only to a case that ran and could not decide, never to a requirement that received no case at all. Both are the same state, and both now block.

A declaration — tool_risk in agentcheck.json, or a custom agent's own ToolDefinition — is authoritative, and makes four behaviours required for that tool: fabricated_success_after_failure, duplicate_action, ambiguous_outcome, and retry_control. If the suite produces no evidence for one, the gate blocks with exit 3, names the tool and the requirement, and says what to do next. --json carries the same under unmet_risk_obligations.

You will see: a target that declares a tool's risk but never exercises it moves from exit 0 to exit 3. That is a false green being corrected.

Scope, deliberately narrow

• Only declared risk counts. Whether an obligation exists is read from the declaration, not from what a coverage report happens to say, so risk AgentCheck merely inferred from a tool's name can never block. Inference is not authority.
• partial evidence does not block. It is the ordinary state of a healthy suite.
• An uncovered tool is not by itself a failure. success_path, failure_handling, and timeout_handling apply to every tool regardless of risk and are excluded.
• A target that declares no risk is unaffected. All four bundled examples behave exactly as before.
• FAIL, INCONCLUSIVE, and INFRA_ERROR remain distinct. An infrastructure error still outranks everything; a real behavioural failure keeps exit 1.

Two limits worth knowing

Both are named in the block message itself, not left to be discovered:

• max_cases bounds a run before obligations are evaluated, so a bounded run can report obligations unmet simply because those cases were not selected. Raise or remove it and re-run before concluding evidence is absent.
• Generation caps cases per origin. Past roughly ten declared-risky tools, AgentCheck's own generator stops emitting the cases this floor requires, and regenerating will not clear it. Do not delete a true risk declaration to go green — that trades an accurate description of your agent for a green build, which is the opposite of what the gate is for.

See the CI gate documentation.

Suite identity

GENERATOR_COMPATIBILITY_VERSION stays 1. This changes a release decision, not generation, so every suite fingerprint stays where it was.

Not changed

No new framework integration, and no expansion of what AgentCheck claims about hostile code, deterministic model execution, or complete observation.

AgentCheck 0.5.3

An evidence-integrity patch. Two independently reproduced defects could make a generated suite, or its coverage report, look more complete than the evidence behind it. Both are corrections to what AgentCheck reports about itself — no framework integration is added and the containment guarantee is unchanged.

Fixed

Valid OpenAI tools with a nullable array parameter could disappear from the generated suite. (#85)

Union extraction discarded a schema branch carrying an explicit, valid type whenever that branch also carried structural keywords such as items. An Optional[List[str]] parameter — emitted by the OpenAI Agents SDK as anyOf[{type: array, items: {type: string}}, {type: null}] — was read as type-unknown, so a tool whose required parameter it was received no cases at all. Explicit types are now retained from any mapping branch under anyOf/oneOf, while validation against the complete original schema remains the authority for every emitted value. A union AgentCheck genuinely cannot resolve, such as an unsupported local $ref, still fails closed with no cases rather than a guessed one.

You will see: a tool that previously generated zero cases now generates them.

A developer-declared tool_risk lost its authority in behavioral coverage. (#88)

Coverage decided whether a tool's risk dimensions were real requirements by reading the tool property's authoritative flag — which records how the tool schema was obtained, and is always false on the OpenAI Agents SDK and PydanticAI adapters. Neither SDK carries risk itself, so for those targets the tool_risk block in agentcheck.json is the entire risk authority, and it was being discarded. A tool explicitly declared destructive reported fabricated_success_after_failure, duplicate_action, ambiguous_outcome, and retry_control as unknown, so those requirements never entered the missing denominator.

Risk authority now comes from spec.tool_risk, per axis, while the predicate stays on the ToolDefinition that generation and the derived policy pack already use. Declaring one axis still never upgrades the other's authority, and a tool whose risk is merely inferred still reports risk_metadata_not_authoritative — inference being correct does not make it authoritative.

You will see: on a target declaring tool_risk against an SDK adapter, risk requirements previously reported unknown now report missing or partial. That is reported coverage becoming accurate, not a change in your agent.

Suite identity

GENERATOR_COMPATIBILITY_VERSION stays 1.

• #85 genuinely changes generated case sets, so a suite containing an affected tool re-identifies. Unaffected suites stay byte- and fingerprint-exact.
• #88 changes reporting only and moves no fingerprint.

Not changed

This release does not expand containment, add a framework integration, or change what AgentCheck claims about hostile code, deterministic model execution, or complete observation. A release gate still decides from executed-case verdicts and the trusted baseline; it does not consult behavioral coverage, so a suite that never exercises a declared-destructive tool can still pass. Coverage reports that gap — now accurately for declared tools — but the gate does not act on it.

AgentCheck 0.5.2

AgentCheck 0.5.2 is an evidence-integrity patch. It makes incomplete,
inconsistent, or drifting execution evidence fail closed instead of looking
complete.

Highlights:

• Complete action-path denominators and stricter stored execution binding.
• agentcheck gate blocks with exit code 3 when the current run is
  INCONCLUSIVE, even when a trusted baseline reports no new failure.
• Invalid fixture templates and partially invalid frozen suites are refused.
• Execution, replay, shrink, and committed artifacts bind more tightly to the
  verified source and replay evidence.
• PyPI README links and release/version documentation are corrected.

Compatibility: invalid or incomplete fixture, frozen-suite, gate, stored-
report, source, and replay states are now refused or classified more strictly.
Valid completed execution paths retain generator compatibility version 1.

Boundaries: this release does not prove hostile-code containment, full answer-
key isolation, immutable execution media or ABA resistance, or a provider-level
guarantee. Environment-containment work under spikes/ remains repository-only
research and is excluded from distributions. Slice 3 is not included.

See the 0.5.2 changelog for the complete bounded correction set.

AgentCheck 0.5.1

0.5.1 (2026-08-29)

Three product correctness fixes, found while reviewing and revalidating 0.5.0's
new MCP-manifest support against a real Slack target.

Fixed

• An empty MCP manifest silently lifted the external-toolset refusal.
  A file containing {} declared no tools but still let preflight proceed,
  creating apparent coverage for a surface AgentCheck could not see. Empty
  manifests are now rejected by the loader, and both direct adapter surfaces
  reject a programmatically constructed empty McpManifest too.
• A manifest could add phantom tools to an agent with no external
  toolset. A stale manifest survived even after its external toolset was
  removed and inserted tools the agent could not actually call, so AgentCheck
  could generate scenarios against a fictional surface. Inspection and
  preflight now reject a manifest unless the PydanticAI agent actually has an
  external toolset for it to describe.
• "post" fell into the weakest read-only risk-inference bucket. A real
  Slack agent's manifest-declared post_message tool inspected as other, read-only (confidence 0.30), but post is also a common noun in the same
  domain. The resolver now treats a leading or standalone post as part of
  the send/email/notify/publish group, while a clearer recognized
  action before it wins. Names such as get_post, fetch_post, and
  summarize_post remain non-mutating rather than acquiring impossible
  duplicate-side-effect policies; post_summary remains a SEND action.

Packaging

• Release artifacts now carry the project's NOTICE alongside LICENSE.
  The artifact audit positively requires both files in the wheel and source
  distribution, so a future packaging omission fails before publication.

Compatibility: this patch is intentionally breaking for two invalid
manifest configurations that 0.5.0 silently accepted: an empty manifest now
fails to load, and a manifest attached to an agent with no external toolset
now fails inspection. Remove the stale file when no external toolset exists;
otherwise declare the external toolset's actual schemas. Valid, non-empty
manifests attached to agents with external toolsets are unaffected.

Suite identity: GENERATOR_COMPATIBILITY_VERSION stays 1. The manifest
corrections refuse invalid configurations rather than changing valid suite
generation. The post correction changes action metadata for matching tool
names and can also change inferred risk. When it changes a tool's resolved
state_changing boolean, the target gets a new spec_id and must regenerate an
older frozen suite. A declared or authoritative value that keeps the resolved
risk booleans stable also keeps spec_id stable, although a newly generated
bounded suite can still record SEND rather than OTHER action metadata. Names
with a leading recognized non-mutating action or no matching post token are
unaffected.