test(a2a): conformance + cross-product + authZ-negative suite - #89
Merged
Conversation
Independent test-engineer slice: static conformance of the frozen A2A v1 contract (schemas, agent cards, wire fixtures, JSON-RPC binding, error taxonomy, state mapping, SSE framing, caller-blind encapsulation). 68 green. Live server + cross-product + authZ-negative tests follow (RED until server). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session-Id: 17fb89fd-3579-433b-a6c4-9c9e7f3ec549
…+ CI runner Adds the RED-until-server integration tier: live conformance, the motivating cross-product acceptance (caller with no Jira skill/MCP gets tickets from FuzePlan), authZ-negative (non-allowlisted caller refused BY THE CALLEE; absent providesTo = DENY; body identity not trusted), and the always_ask/reach_human state machine. Live tests FAIL (not skip) with no server configured — the honest 'server slice not delivered' signal. Adds a scoped CI runner (conformance green / live-integration red until server). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session-Id: 17fb89fd-3579-433b-a6c4-9c9e7f3ec549
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Pin actions/checkout and actions/setup-python to 40-char commit SHAs instead of mutable @vn tags in a2a-contract-tests.yml, addressing 4 GHAS/Semgrep findings. Human-readable version kept as trailing comment. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session-Id: 17fb89fd-3579-433b-a6c4-9c9e7f3ec549
… set Phase 3 acceptance gate: the live integration/acceptance/authZ/state-machine tests now SKIP (module-level skipif on A2A_SERVER_BASE_URL) when no server is deployed, keeping main/PRs green and avoiding claude-ci-autofix churn. When the env is set they RUN and FAIL LOUDLY on any deviation — the honest grade lands in the with-server (Phase 3 rollout/CI) context. Conformance tier unchanged (68 green, no server needed). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session-Id: 17fb89fd-3579-433b-a6c4-9c9e7f3ec549
Per USER decision: the live acceptance/authZ-negative/state-machine tests are a required Phase-3 rollout gate against a DEPLOYED server, not skip-as-success in per-PR CI. - New .github/workflows/a2a-acceptance.yml: workflow_dispatch + workflow_call only (never pull_request/push). Runs 'pytest -m live' with A2A_REQUIRE_LIVE=1 so the suite FAILS LOUDLY (never skips) if the server/tokens are unset or the server misbehaves — a missing server at acceptance time IS a failure. Preflight step + SHA-pinned actions. TODO(devops) marker for server stand-up. - a2a-contract-tests.yml: dropped the live-integration job; per-PR now runs the conformance tier only (no server, stays green every PR). - Added 'live' marker to the four live modules; conftest gate now enforces under A2A_REQUIRE_LIVE and skips otherwise (library-level safety). Verified: conformance 68 green; 'pytest -m live' skips w/o server (exit 0) and fails loudly under A2A_REQUIRE_LIVE (exit 1). Honest grade preserved, made a required rollout gate rather than weakened. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session-Id: 17fb89fd-3579-433b-a6c4-9c9e7f3ec549
izzywdev
marked this pull request as ready for review
July 22, 2026 17:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Independent test-engineer slice for A2A Phase 2. Written against the FROZEN contract at `agent-templates/contracts/a2a/v1/`, blind to the server implementation (`agent-templates/a2a/`, backend-engineer). Does NOT touch the contract.
What is here (`tests/a2a/`)
Two tiers, marked so results are legible:
conformance (expected GREEN now — grades the frozen artifacts + client harness):
integration (expected RED until the server slice lands — env `A2A_SERVER_BASE_URL`):
These FAIL (not skip) when no server is configured — the honest-grader signal that the backend slice is not yet verified.
Status
Conformance: 68 passing locally. Live: RED by design until backend-engineer's server is reachable. A red integration test against a genuine gap is a valid deliverable.
Out of scope (owned by siblings)
A2A server / card generator / adapter (backend-engineer) · image/chart/Argo (devops-engineer) · handoff-MCP client (mcp-engineer) · docs (docs-maintainer). The contract is FROZEN.
🤖 Generated with Claude Code
Test tiers & where they run (per USER decision)
a2a-contract-tests.yml, jobconformance,pytest -m conformance): grades the FROZEN contract, needs no server, stays green on every PR. The live modules are not collected here.a2a-acceptance.yml,pytest -m live): the live cross-product acceptance + authZ-negative + state-machine + live-conformance tests.workflow_dispatch/workflow_callonly — never on PR/push, because they require a deployed A2A server. Runs withA2A_REQUIRE_LIVE=1so they FAIL LOUDLY (never skip) if the server/tokens are unset or the server misbehaves — a missing server at acceptance time IS a failure. Required to pass against a deployed A2A server before rollout — tracked in Phase-3 A2A acceptance gate: run live cross-product + authZ-negative tests against a deployed server #90.a2a-acceptance.yml(compose / kind+helm exportingA2A_SERVER_BASE_URL+ tokens) is devops-engineer's Phase-3 wiring (# TODO(devops)left in the workflow).This relocates an integration gate to where its dependency exists — the honest-grading signal is preserved and made a required rollout gate, not weakened.