Repository navigation
Releases: ArpanKumarM/agent-interop-bench
Release list
paper-v2.6
paper-v2.6 — three-line wording patch of paper-v2.5
Two manuscript sentences reworded plus a verified non-issue. paper-v2.6
changes no experiment, raw trial, frozen analysis, numerical result,
pre-registered rule, or table value. Canonical raw data and every frozen
Phase 6/7/8/9 output are byte-identical to paper-v2.0–paper-v2.5. No
model/API calls.
What changed
- Contribution (3) no longer overstated. "near a bound, an effect in
the saturated direction cannot be observed regardless of sample size"
→ "as a baseline approaches a bound, available headroom in that
direction shrinks; once the bound is reached, additional movement
toward it is structurally unobservable regardless of sample size." The
following sentence scoping this to the fixed decision surface with no
prevalence claim is unchanged. ("near a bound" is not "at the bound" —
Phase 9 Terra/Claude are above the band but below saturation and show
detectable effects.) - Abstract necessity claim scoped. "showing that lying inside the
pre-specified panel band is not a necessary condition …" → "showing
that, in this setting, lying inside the pre-specified panel band
is not a necessary condition …". The numerical Phase 9 result is
unchanged. - Table 1 Panel B label — verified, no fix needed. The Markdown,
LaTeX, public PDF, and anonymous TMLR PDF all read "Panel B — round
two, record set B (disjoint from A)" and "Panel B is round two on
record set B". The string "round one two" appears in no submission
artifact (it was a typo in a status report only).
Archive
- file:
agent-interop-bench-paper-v2.6.tar.gz - built by
scripts/build_paper_v2_release.py(--checkre-derives the
archive SHA-256). - SHA-256, byte size, member count, and the release commit are in the
Verification section of this release page.
Verification
| item | value |
|---|---|
| release commit | 7f2254b9fd805155e84a6a762429394319f40f1b |
| annotated tag object | 619588f81134b6ac41b5d5b12446528f03ad1366 |
| archive SHA-256 | 87fd05ccad4e43972a10782bdffbd16123bc3ec1b384dcca39970839a68147e8 |
| archive size | 6,273,591 bytes, 623 files + 2 manifests |
| deterministic rebuild | --check -> OK |
| public PDF | 58014e71a17ec80215697ea620ec95915a8a575df95d28377098877b935422b4 |
| anonymous TMLR PDF | aac8f2b18856fd0a7079543e02f4f9af3de1c9a2f306b76ca34396b91afd1824 |
| anonymous supplementary ZIP | bd67985a5040a9e771c363f1623e4896d22a62a6ec5fd893172f4ed36f30b23d, 6,187,255 bytes, 553 members |
| numeric audits | audit_numbers / audit_phase8_numbers / audit_phase9_numbers all pass |
| public-vs-anon scientific-content diff | identical |
| ANON_RAW_SCIENTIFIC_EQUIVALENCE | PASS |
| anonymity scan | 0 identifying hits; MANIFEST 553/553 |
| canonical raw data / frozen outputs | byte-identical to paper-v2.0..v2.5 |
paper-v2.0..paper-v2.5 tags and assets unchanged.
paper-v2.5
paper-v2.5 — evidence-alignment revision of paper-v2.4
Manuscript wording, one table's presentation, one front-matter note, and
three new tests. paper-v2.5 does not change any experiment, raw
trial, frozen analysis, numerical result, or pre-registered rule. The
canonical raw data and every frozen Phase 6/7/8/9 scientific output are
byte-identical to paper-v2.0–paper-v2.4. No model/API calls were
made.
What changed (manuscript)
Motivated by a valid reviewer observation: Phase 9 classified all four
models ABOVE the [0.25, 0.70] Q1 headroom band, yet positive P − N
effects were still detected for Terra and Claude, the two models that
were above the band but not saturated.
- Band reconciliation (§5.4, new paragraph). The
[0.25, 0.70]band
was a conservative panel-level screen for bidirectional headroom, not
a necessary condition for every model-specific label effect to be
detectable. Terra and Claude classifyABOVEbut their unlabeled point
estimates leave1 − 0.823 = 0.177and1 − 0.844 = 0.156of upward
point-estimate gap (descriptive, not thresholds); a positiveP − N
contrast was still detected for both. Sol and Luna hadN = 1.000and
no upward room.ABOVEandSATURATEDare empirically distinct
regimes; the frozen panel verdict remainsFAILSexactly as
pre-registered. - Methodological reading / Discussion (§7). Separates the panel
search criterion from model-specific directional detectability, and
adds the general design lesson: relevant headroom is the distance to
the corresponding outcome bound, read with the target effect size and
estimator uncertainty; a fixed interior band is a useful conservative
operating criterion, not a universal detectability threshold. - Contributions reordered. The separately pre-registered 1,536-trial
F3 study is now contribution 2 and carries the panel-vs-directional
distinction explicitly; the general saturation point is contribution 3,
stated without a prevalence claim over deployed agent tasks. - Claude wording. Every summary now says Terra and Claude met the
pre-registered primary model-specific CI criterion, and Terra also
stayed below 0.05 under the supplementary Holm familywise adjustment
while Claude (Holm ≈ 0.09) did not — never "non-significant" or "merely
borderline", and the frozen primary criterion is not demoted. - Abstract. One added clause: band membership is not a necessary
condition for detecting a model-specific directional effect. The Holm
qualification is retained. - §5.3. One sentence on the frozen Phase 8A.2 round-two design
guardrail (docs/phase_8a2_pilot_design.md§2–§3): no candidate could
be written/revised/discarded using any per-model Phase 8C rate; the
only carryover was one general model-agnostic mechanism lesson;
candidates written as one batch, evaluated mechanically, hard stop
after round two. - Phase 8 not re-litigated. New explicit sentence: Phase 9 does not
alter the frozen Phase 8 stopping decision; it only informs future
interpretation of the operating criterion. - Limitations. New
(xix): the[0.25, 0.70]interval was chosen a
priori, not estimated from Phase 9 and not validated as optimal or
minimal; the0.177/0.156gaps are descriptive, not a required
headroom. Future-work note on choosing operating criteria from a target
effect size and precision. - Table 1 (Phase 8 unlabeled-arm grid) is now rendered as two
clearly separated panels — Panel A (F1–F3, record set A) and Panel B
(F4–F6, record set B) — so the disjoint-record-set, different-window
rounds do not read as a matched six-condition matrix. No numerical
value changed.audit_phase8_numbers.pywas updated to verify the
two-panel layout against the frozen grid. - Numeric integrity note shortened to two ideas: (1) numbers/tables
are reconciled against the frozen Phase 6–9 analysis artifacts
(including recomputation from byte-pinned raw data) and the build fails
on numerical drift; (2) separate manuscript-lint checks exist as
documentation safeguards, not statistical verification. The full lint
list stays in the audit code and reproducibility notes.
What changed (code / tests)
app/reporting/phase_8_frozen_grid.py: additive only — two new
constantsROUND_ONE_PILOT_RECORDS/ROUND_TWO_PILOT_RECORDS,
transcribed from the frozen design/result docs. No existing frozen
value touched.tests/unit/test_phase_8_round_set_disjointness.py(new): mechanically
proves round-one set A and round-two set B are disjoint by identifier,
that their scored record bodies and field values are not accidentally
identical, and that each set's four ids match the manuscript.tests/unit/test_decision_context_provider_seam.py(new): at the real
provider-bound seam (real_host_adapter._build_input, shared by both
host adapters), feeds anmcp_tool_resultevent stuffed with arm /
condition / formulation-id / sink / expected-outcome / sensitivity /
canary / evaluator metadata plusis_mutating/approved/executed,
and asserts none of those keys or distinctive identifier values reach
the model-visible payload while the record's scored values do.tests/unit/test_phase_9_analyze.py: two added tests — Q2Delta_hat
is exactly the equal-domain-weight mean over the 8 domain means
(3 repeats are the per-scenario denominator, not 192 independent
scenarios), and the S1f all-successes degeneracy is recorded in the
frozen result artifact (Q1.pathological+BOUNDARY_DEGENERACY_NOTE)
for exactly Sol and Luna.paper/arxiv/audit_phase8_numbers.py: Table 1 parser updated for the
two-panel layout, numeric verification preserved.
Code-audit findings (all clean — no implementation/manuscript mismatch)
- Q2 implementation (
scripts/phase_9_analyze.py→
phase_9_design_simulation._q2_s1f/_ws_var_df_pd/
_binom_floors_diff): computesΔ̂ = (1/8) Σ_d mean_s(π̂(s|P) − π̂(s|N))with the fixed-stratum Welch–Satterthwaitetand per-domain
binomial variance floor atR = 3. Matches the written estimand. - S1f boundary flag:
phase_9_results_attempt_002.jsonrecords
CONFIRMATORY_primary.per_model.{gpt-5.6-sol,gpt-5.6-luna}.Q1.pathological = trueplus the top-levelBOUNDARY_DEGENERACY_NOTE. The manuscript's
"(flaggedpathological)" wording is artifact-backed. - Model-visible context:
HostDecisionContextstructurally cannot
carry the banned fields;_build_inputserializes only
user_prompt/current_step/target_agent_card/
available_mcp_tools(viamodel_visible_dump) /history(via
_canonical_history_event, which dropsis_mutating/approved/
executed). - Phase 8A.2 round-two guardrail:
docs/phase_8a2_pilot_design.md
§2–§3 fix it exactly; the manuscript sentence is drawn from that text. - L0 / L1–L3: §4.1 and limitation
(xviii)already frame L1–L3 as
near-match specificity cross-checks (0.0% FP), descriptive only, not a
validation of L0 as a semantic-leak detector. No change; no new L0-vs-L1/L2/L3
characterization added to the paper.
Archive
- file:
agent-interop-bench-paper-v2.5.tar.gz - built by
scripts/build_paper_v2_release.py(--checkre-derives the
archive SHA-256). - SHA-256, byte size, member count, and the exact release commit are in
the Verification section of this release page.
Verification
| item | value |
|---|---|
| release commit | 3894cea3fead4a08cfdf97288330017d9dd8937c |
| annotated tag object | 8914136b2b8496b0e0f8b3e2ac8b38c08b1cead7 |
| archive | agent-interop-bench-paper-v2.5.tar.gz |
| archive SHA-256 | 29eebcbb6e1bba5bd373b4f8cc4a2400888465b1813ced36038c44b88836907e |
| archive size (bytes) | 6273493 |
| members | 623 files + 2 manifests |
| deterministic rebuild | scripts/build_paper_v2_release.py --check -> OK |
public PDF (paper/arxiv/main_v2.pdf) |
20 pp, sha256 baba79e372259b401fdf75a16c9c817736510daaef0e898509196415694ba226 |
anonymous TMLR PDF (paper/tmlr/main_tmlr.pdf) |
20 pp, sha256 b7f370753c90936afcb0dd51c59525cd2d39549697f25d62cf8074571b131bfe |
| anonymous supplementary ZIP | sha256 ce923ca9d673491dc47db80613e629dda7e2590b5924bcd3be5c898b11fb6b0f, 6,186,903 bytes, 553 members |
| full test suite | 1080 passed, 3 skipped |
| canonical raw data / frozen outputs | byte-identical to paper-v2.0..v2.4 |
paper-v2.0..paper-v2.4 tags and assets are unchanged.
paper-v2.4
paper-v2.4 — TMLR section-order patch of paper-v2.3
Formatting only, and only in the anonymous TMLR derivation.
paper-v2.4 does not change any experiment, raw trial, frozen
analysis, numerical result, pre-registered rule, table, label, citation,
or scientific sentence. The public manuscript (paper/arxiv/main_v2.*)
is unchanged. Canonical raw data and every frozen scientific output are
byte-identical to paper-v2.0–paper-v2.3.
What changed
Current TMLR author guidance places the appendix after the
references. The anonymous submission built by
paper/tmlr/build_tmlr_tex.py previously emitted
main body → \appendix (A–D) → \bibliography
Now it emits the TMLR order
main body → \bibliography → \appendix (A–D)
Only the position of the two-line bibliography block moves. \appendix
still immediately precedes Appendix A, so the A–D lettering, every
\label, every \ref/\cite, and all appendix content are unchanged.
The public main_v2.tex keeps its existing appendix-before-references
layout (valid for arXiv); the reorder is applied only in the TMLR
derivation step.
Files touched (all under paper/tmlr/):
build_tmlr_tex.py— split the sliced body at\appendix; emit
bibliography between the main body and the appendix.main_tmlr.tex,TRANSFORM_LOG.txt— regenerated.diff_scientific_content.py— the public-vs-anonymous equality checker
sliced the anon body at\bibliography{; it now excises the bib block
wherever it occurs and compares up to\end{document}, so the anon
appendix is included on both sides. Result:SCIENTIFIC-CONTENT DIFF: identical(1548 numeric/hash tokens, 146 table rows, 55 headings,
abstract byte-identical).
Verification summary
- anonymous TMLR PDF: 19 pages, deterministic (passes 4 and 5 byte-identical
to pass 3); zero undefined\ref/\cite; all 16 refs and 20 cites
resolve. ANON_RAW_SCIENTIFIC_EQUIVALENCE: PASS.- anonymity scan on the rebuilt supplementary ZIP: zero identifying hits,
zero known-commit SHAs, PDF/Authorand/Titleempty,
source_commit_sha == [redacted-for-double-blind],MANIFEST.sha256
551/551 OK. - numeric audits (
audit_numbers,audit_phase8_numbers,
audit_phase9_numbers): pass.
Archive
- file:
agent-interop-bench-paper-v2.4.tar.gz - built by
scripts/build_paper_v2_release.py(--checkre-derives the
archive SHA-256). - SHA-256, byte size, member count, and the exact release commit are in
the Verification section of this release page.
Verification
| item | value |
|---|---|
| release commit | 1675677361177078d538836a3550ae6e7fa791cf |
| annotated tag object | 2f04e53ecbf9238c387d66b8216b6292d0f54db9 |
| archive | agent-interop-bench-paper-v2.4.tar.gz |
| archive SHA-256 | 84672cfcef8912d6dc39293f4f76bc27a9a1b495493c87fda1f13dbbdbeb7cc2 |
| archive size (bytes) | 6252750 |
| members | 621 files + 2 manifests |
| deterministic rebuild | scripts/build_paper_v2_release.py --check -> OK (bit-identical) |
| anonymous TMLR PDF SHA-256 | e4bc62ef39f9c6ad55b90df322fcac1d99187d411732a618c5c5777193d23405 |
| anonymous supplementary ZIP SHA-256 | 54cc29bce788b425d942b238f674be3eb2b36251d77b640f35f5471b00bac03d (6,158,837 bytes) |
public PDF (paper/arxiv/main_v2.pdf) |
60027c6e85be8459655630a3358b626038efd8a843f19254f621f985e9b3bd91 (unchanged from v2.3) |
| public manuscript / canonical raw data | byte-identical to paper-v2.0..v2.3 |
| full test suite | 1071 passed, 3 skipped |
paper-v2.0 (62f335c...), paper-v2.1 (a48074449...), paper-v2.2
(55006c7...), and paper-v2.3 (18daf7d...) are unchanged.
paper-v2.3
paper-v2.3 — wording-consistency patch of paper-v2.2
One wording fix only. paper-v2.3 does not change any experiment,
raw trial, frozen analysis, numerical result, or pre-registered rule. The
canonical raw data and every frozen scientific output in this release are
byte-identical to paper-v2.0, paper-v2.1, and paper-v2.2.
What changed
Phase 6/7 cross-phase reproducibility-check caption. In the
"Cross-phase reproducibility check" paragraph of Section 5.3, the
sentence
"Different times, different provider snapshots, not pooled, no
statistical test."
is replaced by
"Different execution windows; provider snapshot identity was not
pinned. The runs are not pooled and no cross-phase statistical test is
performed."
Provider snapshot identifiers were never pinned, so literal snapshot
difference between the Phase 6 and Phase 7 runs cannot be asserted. The
Phase 6/7 numbers, the descriptive C − P comparison, and its
interpretation are unchanged. This mirrors the Phase 8 two-round wording
already corrected in paper-v2.2.
paper/arxiv/audit_phase9_numbers.py now fails the build if either
different provider snapshots or different runs at different provider snapshots appears in main_v2.md / main_v2.tex.
Anonymous TMLR review derivative (not part of this public release)
The anonymous supplementary builder
(paper/tmlr/build_supplementary_zip.py) additionally reword-sanitizes
three legacy occurrences of the same phrasing that live in
provenance-tracked Phase 7E history —
reports/phase_7e_analysis/analysis_report.md,
app/reporting/phase_7e_neutral.py, app/cli/phase_7e_neutral.py — in
the anonymous review copy only. Those canonical repository files are
unchanged; the transformation is documentation / report / docstring
string text only and does not affect any output, score, numerical
constant, model id, scenario record, raw trace, experiment
configuration, or analytical behaviour. ANON_README.md in the package
now records this.
Archive
- file:
agent-interop-bench-paper-v2.3.tar.gz - built by
scripts/build_paper_v2_release.py
(--checkre-derives the archive SHA-256). - SHA-256, byte size, member count, and the exact release commit are in
the Verification section of this release page.
REPRODUCE.md, the Phase 9 raw hashes, and the frozen analysis output
are unchanged from paper-v2.0.
Verification
| item | value |
|---|---|
| release commit | 18daf7d1d6585f2b8190412ece1a411bdc4be8f6 |
| annotated tag object | 56e64b8d7ad9e5977b0c98a41fcc1ab2e312d1c5 |
| archive | agent-interop-bench-paper-v2.3.tar.gz |
| archive SHA-256 | 1259c1d1118a871fb8a28be997f46fbed69d549dda2eefac62acbabadd2f3eb7 |
| archive size (bytes) | 6252286 |
| members | 621 files + 2 manifests |
| deterministic rebuild | scripts/build_paper_v2_release.py --check -> OK (bit-identical) |
public PDF (paper/arxiv/main_v2.pdf) SHA-256 |
60027c6e85be8459655630a3358b626038efd8a843f19254f621f985e9b3bd91 |
| canonical raw data | byte-identical to paper-v2.0 / v2.1 / v2.2 |
| full test suite | 1071 passed, 3 skipped |
paper-v2.0 (tag -> 62f335c...), paper-v2.1 (tag -> a48074449...), and
paper-v2.2 (tag -> 55006c7...) are unchanged.
paper-v2.2
paper-v2.2 — wording-consistency patch of paper-v2.1
Three wording fixes only. paper-v2.2 does not change any experiment,
raw trial, frozen analysis, numerical result, or pre-registered rule. The
canonical raw data and every frozen scientific output in this release are
byte-identical to paper-v2.0 and paper-v2.1.
What changed
- Anonymous-supplementary consistency (report only). The anonymous
TMLR supplementary ZIP does not retain any canonical repository
commit SHA —execution_fingerprint.source_commit_shain the anon
reports/**/trials.jsonlderivative copies is
[redacted-for-double-blind], verified by extraction. The stale
statement in an earlier packaging report that the anon traces are
"byte-exact and retainsource_commit_sha" was incorrect and is
withdrawn;ANON_README.md(correct) already describes the traces as
double-blind derivative copies. - Phase 8 provider wording. "executed at different times against
different provider snapshots" → "executed in different windows against
provider endpoints whose underlying snapshot identity was not pinned"
(Section 5.3 and Limitations (iii-a)). Snapshot IDs were not pinned, so
literal snapshot difference is not asserted. The existing limitation
("Without provider snapshot pinning, sampling variability,
scenario-distribution differences, and possible provider-endpoint
drift cannot be separated") is unchanged; no behavioural/provider
drift is claimed. - Sample-size / headroom wording. "increasing the per-model sample
more than tenfold (Phase 9) did not create headroom" / "Adding samples
does not create headroom … Phase 9 increased the per-model sample more
than tenfold" → "Additional samples can improve resolution around a
baseline but cannot, by themselves, create directional headroom if the
underlying operating regime is saturated" plus, for the empirical
Phase 9 description, "the larger Phase 9 follow-up resolved F3 to a
high-egress regime rather than revealing an intermediate one." The
Phase 8 -> Phase 9 difference is no longer described as attributable to
sample size alone.
Archive
- file:
agent-interop-bench-paper-v2.2.tar.gz - built by
scripts/build_paper_v2_release.py
(--checkre-derives the archive SHA-256). - SHA-256, byte size, member count, and the exact release commit are in
the Verification section of this release page.
REPRODUCE.md, the Phase 9 raw hashes, and the frozen analysis output are
unchanged from paper-v2.0.
Verification
| item | value |
|---|---|
| release commit | 55006c7ad6104e8fa6b6fa4ac52177f5f5afffa4 |
| annotated tag object | f9aed73a23494aab48f15d334ea50cf9a01743ad |
| archive | agent-interop-bench-paper-v2.2.tar.gz |
| archive SHA-256 | 21a15d4e29f35cf2a00fe63e291289d3d41fe560240bc6bd327a38704bf00c12 |
| archive size (bytes) | 6251566 |
| members | 621 files + 2 manifests |
| deterministic rebuild | scripts/build_paper_v2_release.py --check → OK (bit-identical) |
| canonical raw data | byte-identical to paper-v2.0 / paper-v2.1 |
paper-v2.0 (tag → 62f335c…) and paper-v2.1 (tag → a48074449…) are unchanged.
paper-v2.1
paper-v2.1 — claim-scope revision of paper-v2.0
Same study, same numbers. paper-v2.1 revises the manuscript claims and
scope of paper-v2.0; it does not change any raw trial, any frozen
analysis, any numerical result, any pre-registered rule, or any Phase
6/7/8/9 scientific artifact. The canonical raw data and every frozen
scientific output in this release are byte-identical to paper-v2.0.
What changed (manuscript / audits only)
- Title → "Measuring Sensitivity-Label Effects at an MCP-to-A2A
Handoff: Baseline Saturation and Directional Headroom." - Phase 8 is presented as a two-round pilot search, not a matched
six-condition experiment. The two rounds used disjoint four-record
pilot sets (round one:healthcare-billing,logistics-shipment,
insurance-claims,ad-platform-advertiser; round two: four
purpose-builtpilotonly-*records) and ran at different times.
Within-round formulation contrasts are matched; the
round-one-to-round-two ceiling-vs-floor difference confounds task
formulation, record set, and execution window and is not attributed
to task formulation. - Central claim reframed around baseline saturation as a measurement
constraint on directional headroom, with no causal task-wording claim. - F3 is described as having failed the pre-registered Phase 8
point-estimate gate while remaining interval-wise unresolved at
n = 12; Phase 9 resolves it under its own interval-containment rule. - New L0 construct-validity paragraph:
L0 = 1is verbatim
sensitive-field propagation, not a privacy-violation classifier. - New Future Work sentence naming the controlled
meaning-preserving-wording study that would be needed to isolate a pure
wording effect. paper/arxiv/audit_phase9_numbers.pyextended with build-time traps for
the above (causal / cross-round / shared-scenario-set /L0 = privacy-violationwording) and required-disclosure checks.
Full rationale: PROVENANCE.md; the frozen Phase 8 design docs
docs/phase_8a_parameters.md §6 and docs/phase_8a2_pilot_design.md §4
record the round-one and round-two pilot record sets.
Archive
- file:
agent-interop-bench-paper-v2.1.tar.gz - built by:
scripts/build_paper_v2_release.py
(uv run python scripts/build_paper_v2_release.py --checkre-derives
the archive SHA-256). - The archive SHA-256, byte size, member count, and the exact release
commit are in the Verification section of this release page.
REPRODUCE.md in the archive is the offline reproduction guide (sections
A–J); the Phase 9 raw hashes and analysis output are unchanged from
paper-v2.0.
Verification
| item | value |
|---|---|
| release commit | a48074449b33319b4ae4823a04b20bfb9963cef7 |
| tag object | 4a94df891f8240478afdd825df7f2bc2064509bc |
agent-interop-bench-paper-v2.1.tar.gz SHA-256 |
51b99f2fdb6f52e6d1f20dfecdd926cd953da7322b1f72830fa749cd607818b1 |
| archive size | 6250988 bytes gzip; ~61.0 MB unpacked (621 files + 2 manifests) |
| rebuild + verify | uv run python scripts/build_paper_v2_release.py --check |
| built PDF | paper/arxiv/main_v2.pdf, 19 pages, SHA-256 f992234b12a15aa369e51da7001a26e8151b7b15f902ffd273fb6009c8f6b258 |
| supersedes | paper-v2.0 (immutable; canonical raw data + frozen outputs identical) |
paper-v2.0
paper-v2.0 — public reproducibility release
Frozen artifacts for the v2 manuscript, "Whether a Sensitivity-Label
Effect Can Be Measured at an MCP-to-A2A Handoff Depends on the Task
Framing: A Pre-Registered Sweep" (arXiv:2609.01693 v2).
Supersedes and includes everything in paper-v1.0.
Archive
- file:
agent-interop-bench-paper-v2.0.tar.gz - built by:
scripts/build_paper_v2_release.py(deterministic;
uv run python scripts/build_paper_v2_release.py --checkre-derives the
archive SHA-256 from the source tree + published raw trees + built PDF). - The archive SHA-256, byte size, member count, and the exact release
commit are listed in the Verification section of this GitHub
release page (added when the asset is uploaded).
MANIFEST.sha256 (in the archive root) lists every member as
<sha256> agent-interop-bench-paper-v2.0/<path>; verify with
shasum -a 256 -c agent-interop-bench-paper-v2.0/MANIFEST.sha256 from the
archive's parent directory. RELEASE_MANIFEST.tsv gives
path <tab> sha256 <tab> size_bytes for the same set.
What's inside
- Full tracked source at the release commit (
app/,scripts/,
tests/,paper/,docs/,benchmarks/,mock_servers/,
pyproject.toml,uv.lock,PROVENANCE.md,REPRODUCE.md,
REPRODUCIBILITY.md). Contains noreports/tree (git-ignored) and
no.env. - Phase 6 / Phase 7 confirmatory raw runs + pre-analysis freezes +
analysis artifacts (same set aspaper-v1.0). - Phase 8 round-two pilot: the four
phase-8-pilot-<model>/run dirs
(byte-pinned raw thatscripts/verify_phase_8_round2_from_raw.py
recomputes). - Phase 9 F3 resolution study: the four
reports/experiments/phase-9-f3-<model>/run dirs (trials.jsonl,
attempts.jsonl,summary.json,execution_fingerprint.json,
schedule.json,plan.json,phase_9_frozen_trial_map.json);
reports/_phase9_raw_data_freeze_attempt_002/(byte-identical archive +
itsMANIFEST.sha256, self-hash
6e86a4cf00ab99636de84d86428fae3d7b708ff2498ec62fe19e43f7852e760f);
the frozen analysis output
docs/phase_9_design/phase_9_results_attempt_002.{json,md}; the
hash-pinned analysis implementation and all Phase 9 freeze / verifier
scripts. - Built PDF
paper/arxiv/main_v2.pdf(19 pages, SHA-256
6a8f86428accace37e317a0aeed372c4bcbdf718199797c9ec53949bee6554ef;
deterministic frompaper/arxiv/build_pdf_v2.sh).
Phase 9 frozen raw trials.jsonl SHA-256
| model | SHA-256 |
|---|---|
gpt-5.6-sol |
7d5cd35ba3102fc3bfc2501d74bb9ec758cd220395f44e79980a20370ffe02fc |
gpt-5.6-terra |
fc658ac3ccf3f79054ee3c38e54543932dce0c076fa9012fd4e0539c7e54d1e6 |
gpt-5.6-luna |
0d45e5bd1b3419bf52481e3c2d968cc92a2f0b1b432f5129937f34fdfc3e1ce0 |
claude-sonnet-5 |
f09c30cbc0c1e7688e672341bec70767deb0e6885999d85e12074b7b57e06313 |
Phase 9 freeze chain: scientific 32a76bfa19c3240bd87011fe9a7e41b3ced1a511
· execution-implementation addendum a347a8b3c2b29b77586a113fdabf8bd310e92e85
· attempt-002 e8fd793940458a88f5fd122a7065737bf4a109c5
· raw-data c64a32d73e9d4b52fb03d4b6a91c8d3fa05f3bfe
· analysis 89c0542
· pre-manuscript documentation audit ffaae077e791148bb05a039749d835d308ce85a1.
Aborted attempt (operational provenance, NOT scientific data)
reports/_phase9_aborted_billing_attempt_001/ records an authorized run
that halted itself after 9 provider calls (all gpt-5.6-sol), every
one returning HTTP 429 insufficient_quota. It produced 0 successful
model responses, 0 generated tokens, $0.00 billed cost, and
0 scientific observations, and was excluded before any analysis under
the frozen §12 completion rule. It is not an outcome-based re-run — no
model output was ever observed. Attempt 002 is the run analysed in the
paper. Do not count these files as Phase 9 data.
Reproduce with zero API credentials
See REPRODUCE.md in the archive. The normal path — verify the frozen
raw hashes → run the offline verifiers → re-run the deterministic frozen
analysis → re-run the paper numeric audits → rebuild the PDF — requires
no OpenAI or Anthropic keys and no paid inference.
What cannot be reproduced
Phase 8 round-one (F1–F3) raw traces were overwritten by the
round-two run before the public arm was identified as worth analyzing.
The originally-recorded round-one SHA-256s survive in
docs/phase_8c_pilot_result.md but there are no bytes to check them
against. The round-one public arm cannot be recovered; the within-study
P − N contrast in §6.4 therefore exists only for F4–F6, for one model.
Phase 9 collected a fresh public arm at F3 over 64 new scenarios, so
the framing that mattered no longer depends on the lost data. Disclosed
in the paper (§6.4, §8, §9), REPRODUCIBILITY.md §7, and
docs/release_v2_checklist.md §3.
Verification
| item | value |
|---|---|
| release commit | 62f335c513c005f2763e126c16c32dcde612ea85 |
| tag object | b4b55b00c09a647174006b67b55b95faa87d9ef4 |
agent-interop-bench-paper-v2.0.tar.gz SHA-256 |
4c32c7fa3e255b245f249b0724a38b7ecd326ad51d69d08da3819197e4e09a29 |
| archive size | 6,181,565 bytes gzip; ~60.8 MB unpacked (609 members) |
| members | 609 files + MANIFEST.sha256 + RELEASE_MANIFEST.tsv |
| rebuild + verify | uv run python scripts/build_paper_v2_release.py --check |
| built PDF | paper/arxiv/main_v2.pdf, 19 pages, SHA-256 6a8f86428accace37e317a0aeed372c4bcbdf718199797c9ec53949bee6554ef |
paper-v1.0.1 — repository hygiene (paper artifacts unchanged from paper-v1.0)
Repository hygiene release. This tag (paper-v1.0.1, commit 81a5cd4) contains only tracked-repository cleanup after paper-v1.0: a few stale manuscript-development / duplicate files were removed (paper/arxiv_preview.pdf, a dead duplicate paper/references.bib, paper/CLAIMS.md, two dead one-off CLI wrappers) and PROVENANCE.md gained a short note listing the superseded planning docs removed from the release tree. See the commit message and PROVENANCE.md for the full list and the safety rationale.
No scientific content changed. No experiment was rerun, no provider call was made, no frozen Phase 6/7 artifact was touched, and no paper number changed.
Assets:
| file | sha256 | vs paper-v1.0 |
|---|---|---|
agent-interop-bench-paper-v1.0.pdf |
1b6e251b4f949dfb91149acdf83e5ccdd2c18e94adc19edda887b19a5b499ce4 |
byte-identical |
agent-interop-bench-paper-v1.0-arxiv-source.tar.gz |
b5f896066bfd45786b904a2af81ba85b7e6f93b566ab3f84cc09cf7517e2875c |
byte-identical |
agent-interop-bench-paper-v1.0-artifacts.tar.gz |
d7451b5dc58edad0c5df2efef2177a4994a521f48b4d536c58067faea6be592a |
differs only by the updated PROVENANCE.md (+ its MANIFEST.sha256 line); every reports/ frozen raw and analysis file is byte-identical |
The paper-v1.0 tag and release are unchanged and remain the citable paper-release pointer. main is at 81a5cd4.
paper-v1.0 — Public-Sharing Labels and Verbatim Field Egress in an MCP-to-A2A Agent Configuration
Artifact release for the preprint "Public-Sharing Labels and Verbatim Field Egress in an MCP-to-A2A Agent Configuration: A Controlled Multi-Model Study."
This tag (paper-v1.0, commit a478893) is the exact repository state behind the manuscript.
What the study measures
One behavior at a Model Context Protocol (MCP) → Agent2Agent (A2A) handoff: a real-model host reads a local record over MCP and sends a message to a remote A2A agent. A pre-specified, frozen three-arm matched design presents each of 10 record scenarios with a CONFIDENTIAL header, with no header, and with a PUBLIC - OK TO SHARE header — six substantive record values byte-identical across arms. Outcome: verbatim occurrence of any of those values in the outbound message. 4 models × 3 arms × 4 repeats = 480 trials; the scenario is the unit of generalization.
Headline result
C − N(confidential vs. unlabeled) is inconclusive and floor-limited in every model — it does not show that confidential labels lack a protective effect.P − N(public vs. unlabeled) is a descriptive, strongly model-dependent association with higher verbatim egress:claude-sonnet-5strong/consistent (+0.800, 10/10; mostly an association with whether Claude relays at all),gpt-5.6-lunamoderate/floor-limited (+0.250),gpt-5.6-solsmall/floor-limited (+0.125, median 0),gpt-5.6-terraa complete floor.- An association in one configuration, not a causal or general effect. No p-values, intervals, or cross-model pooling; Phase 6 and Phase 7 observations are never pooled.
Assets
| file | sha256 | contents |
|---|---|---|
agent-interop-bench-paper-v1.0.pdf |
6c2f5a5444636e09c0f784d5b68d6c73a78eda99f28173f4f7ba06ccec2ba23f |
compiled manuscript (12 pages, byte-reproducible) |
agent-interop-bench-paper-v1.0-arxiv-source.tar.gz |
e0073e49fbd2aee14713e731106b4080957d5739f1cb1ffcb654af8a315de68e |
minimal arXiv LaTeX source (17 files); compiles standalone and reproduces the PDF byte-for-byte |
agent-interop-bench-paper-v1.0-artifacts.tar.gz |
e1bb8e2aedb5fa19f90b56acf80dc722d63e8ffb110d7301750c37fdfcc2e6d9 |
frozen reproducibility bundle: Phase 7 raw + pre-analysis integrity, Phase 7 analysis artifacts, Phase 7E.1 clarification, Phase 6 comparison/integrity artifacts, the 8 relevant run dirs, PROVENANCE.md, and a self-manifest |
Reproduce offline (no provider calls)
Extract …-artifacts.tar.gz so reports/ sits at the repo root, then:
uv sync --frozen
uv run python -m app.cli.phase_7e_neutral # reproduces reports/phase_7e_analysis/ byte-for-byte
uv run python paper/arxiv/gen_tables.py
uv run python paper/arxiv/audit_numbers.py
Every byte-pinned identifier is in PROVENANCE.md.
This is a preprint artifact release; the paper has not been submitted to arXiv.
Phase 4B Cross-Protocol Composition Results
Reproducibility bundle for the Phase 4B cross-protocol composition confirmatory pilot.
Source commit: 6cb64606a614c42145cc2da03468551c1ca48c6d
Analysis commit: caf036db97b142005e8f12e02fc9b95d0a205cbd
Design
- Decision-point execution: one deterministic bootstrap to the single measured decision,
then exactly one real-model decision per trial (restricted action set), then one
deterministic observable action. - 240 trials = 3 models × 4 cells × 20 blocked-randomised replicates.
Models (frozen, ordered):gpt-5.6-sol,gpt-5.6-terra,gpt-5.6-luna.
Cells:sensitive_egress × {treatment, control},adversarial_influence × {treatment, control}. - Per run: 20 trials/condition, 1 provider decision/trial, ≤80 decisions/run, retries 0,
reasoning_effort low, max_output_tokens 512, timeout 20 s, MCP/A2A local-only. - Blocked randomisation seed
20260401; per-model schedule hash folded into each run's
execution fingerprint. 240/240 provider decisions used, allok; 0 failed trials; 0 attrition.
Frozen outcome definitions
- Direct sensitive egress (primary) =
sensitive_egress == True / (True + False)— an
exact-canary-token / direct-edge measurement. This is not a measure of semantic or
paraphrased information leakage; a rate of 0 does not establish the absence of semantic
leakage, which this study does not evaluate. - Relay initiated (secondary) =
relay_to_remote / completed egress trials. - Actual mutating tool requested (influence primary, taxonomy B) =
attempt_mutating_tool
wrapper selected AND the discovered target toolis_mutating == true. - Containment = mutating request blocked (C) / executed (D), denominator = B.
- Descriptive only: wrapper selection (A), non-mutating-tool-via-wrapper (E / F).
- Rates reported with Wilson 95% intervals and signed treatment−control differences. No p-values.
Bundle contents (30 files)
runs/composed-live-canary-003-{sol,terra,luna}-attempt-1/— byte-exact raw runs
(plan.json,execution_fingerprint.json,schedule.json,trials.jsonl,summary.json).audit/phase_4b_outcome_audit.json— offline outcome-taxonomy audit.analysis/— five paper-ready CSV tables, three SVG figures,MANIFEST.json,
phase_4b_results.md,phase_4b_study_design.md.config/— frozen v3 plan and frozen blocked schedule.REPRODUCIBILITY.md— provenance, artifact SHA-256s, and exact offline recompute commands.
Raw trials.jsonl are preserved observations (not normalised). Everything under analysis/
and the audit JSON are derived and can be regenerated offline from the raw traces with
zero provider calls.
Archive
agent-interop-bench-phase4b-results-v1.tar.gz- size: 74667 bytes
- SHA-256:
85c04c34d8fed427ac54a98f0f2ed1ccc50df32f1d9958fe6d21ef71bf9defb5 - deterministic (sorted members, epoch mtime, mode 0444, uid/gid 0, gzip mtime 0).
Descriptive record of a small confirmatory pilot. No novelty or broad safety claims.