Skip to content

Releases: ArpanKumarM/agent-interop-bench

paper-v2.6

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 09 Sep 23:13

paper-v2.6 — three-line wording patch of paper-v2.5

Two manuscript sentences reworded plus a verified non-issue. paper-v2.6
changes no experiment, raw trial, frozen analysis, numerical result,
pre-registered rule, or table value. Canonical raw data and every frozen
Phase 6/7/8/9 output are byte-identical to paper-v2.0–paper-v2.5. No
model/API calls.

What changed

  1. Contribution (3) no longer overstated. "near a bound, an effect in
    the saturated direction cannot be observed regardless of sample size"
    → "as a baseline approaches a bound, available headroom in that
    direction shrinks; once the bound is reached, additional movement
    toward it is structurally unobservable regardless of sample size." The
    following sentence scoping this to the fixed decision surface with no
    prevalence claim is unchanged. ("near a bound" is not "at the bound" —
    Phase 9 Terra/Claude are above the band but below saturation and show
    detectable effects.)
  2. Abstract necessity claim scoped. "showing that lying inside the
    pre-specified panel band is not a necessary condition …" → "showing
    that, in this setting, lying inside the pre-specified panel band
    is not a necessary condition …". The numerical Phase 9 result is
    unchanged.
  3. Table 1 Panel B label — verified, no fix needed. The Markdown,
    LaTeX, public PDF, and anonymous TMLR PDF all read "Panel B — round
    two, record set B (disjoint from A)" and "Panel B is round two on
    record set B". The string "round one two" appears in no submission
    artifact (it was a typo in a status report only).

Archive

  • file: agent-interop-bench-paper-v2.6.tar.gz
  • built by scripts/build_paper_v2_release.py (--check re-derives the
    archive SHA-256).
  • SHA-256, byte size, member count, and the release commit are in the
    Verification section of this release page.

Verification

item value
release commit 7f2254b9fd805155e84a6a762429394319f40f1b
annotated tag object 619588f81134b6ac41b5d5b12446528f03ad1366
archive SHA-256 87fd05ccad4e43972a10782bdffbd16123bc3ec1b384dcca39970839a68147e8
archive size 6,273,591 bytes, 623 files + 2 manifests
deterministic rebuild --check -> OK
public PDF 58014e71a17ec80215697ea620ec95915a8a575df95d28377098877b935422b4
anonymous TMLR PDF aac8f2b18856fd0a7079543e02f4f9af3de1c9a2f306b76ca34396b91afd1824
anonymous supplementary ZIP bd67985a5040a9e771c363f1623e4896d22a62a6ec5fd893172f4ed36f30b23d, 6,187,255 bytes, 553 members
numeric audits audit_numbers / audit_phase8_numbers / audit_phase9_numbers all pass
public-vs-anon scientific-content diff identical
ANON_RAW_SCIENTIFIC_EQUIVALENCE PASS
anonymity scan 0 identifying hits; MANIFEST 553/553
canonical raw data / frozen outputs byte-identical to paper-v2.0..v2.5

paper-v2.0..paper-v2.5 tags and assets unchanged.

paper-v2.5

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 09 Sep 22:19

paper-v2.5 — evidence-alignment revision of paper-v2.4

Manuscript wording, one table's presentation, one front-matter note, and
three new tests. paper-v2.5 does not change any experiment, raw
trial, frozen analysis, numerical result, or pre-registered rule. The
canonical raw data and every frozen Phase 6/7/8/9 scientific output are
byte-identical to paper-v2.0–paper-v2.4. No model/API calls were
made.

What changed (manuscript)

Motivated by a valid reviewer observation: Phase 9 classified all four
models ABOVE the [0.25, 0.70] Q1 headroom band, yet positive P − N
effects were still detected for Terra and Claude, the two models that
were above the band but not saturated.

  • Band reconciliation (§5.4, new paragraph). The [0.25, 0.70] band
    was a conservative panel-level screen for bidirectional headroom, not
    a necessary condition for every model-specific label effect to be
    detectable. Terra and Claude classify ABOVE but their unlabeled point
    estimates leave 1 − 0.823 = 0.177 and 1 − 0.844 = 0.156 of upward
    point-estimate gap (descriptive, not thresholds); a positive P − N
    contrast was still detected for both. Sol and Luna had N = 1.000 and
    no upward room. ABOVE and SATURATED are empirically distinct
    regimes; the frozen panel verdict remains FAILS exactly as
    pre-registered.
  • Methodological reading / Discussion (§7). Separates the panel
    search criterion from model-specific directional detectability, and
    adds the general design lesson: relevant headroom is the distance to
    the corresponding outcome bound, read with the target effect size and
    estimator uncertainty; a fixed interior band is a useful conservative
    operating criterion, not a universal detectability threshold.
  • Contributions reordered. The separately pre-registered 1,536-trial
    F3 study is now contribution 2 and carries the panel-vs-directional
    distinction explicitly; the general saturation point is contribution 3,
    stated without a prevalence claim over deployed agent tasks.
  • Claude wording. Every summary now says Terra and Claude met the
    pre-registered primary model-specific CI criterion, and Terra also
    stayed below 0.05 under the supplementary Holm familywise adjustment
    while Claude (Holm ≈ 0.09) did not — never "non-significant" or "merely
    borderline", and the frozen primary criterion is not demoted.
  • Abstract. One added clause: band membership is not a necessary
    condition for detecting a model-specific directional effect. The Holm
    qualification is retained.
  • §5.3. One sentence on the frozen Phase 8A.2 round-two design
    guardrail (docs/phase_8a2_pilot_design.md §2–§3): no candidate could
    be written/revised/discarded using any per-model Phase 8C rate; the
    only carryover was one general model-agnostic mechanism lesson;
    candidates written as one batch, evaluated mechanically, hard stop
    after round two.
  • Phase 8 not re-litigated. New explicit sentence: Phase 9 does not
    alter the frozen Phase 8 stopping decision; it only informs future
    interpretation of the operating criterion.
  • Limitations. New (xix): the [0.25, 0.70] interval was chosen a
    priori, not estimated from Phase 9 and not validated as optimal or
    minimal; the 0.177 / 0.156 gaps are descriptive, not a required
    headroom. Future-work note on choosing operating criteria from a target
    effect size and precision.
  • Table 1 (Phase 8 unlabeled-arm grid) is now rendered as two
    clearly separated panels
    — Panel A (F1–F3, record set A) and Panel B
    (F4–F6, record set B) — so the disjoint-record-set, different-window
    rounds do not read as a matched six-condition matrix. No numerical
    value changed. audit_phase8_numbers.py was updated to verify the
    two-panel layout against the frozen grid.
  • Numeric integrity note shortened to two ideas: (1) numbers/tables
    are reconciled against the frozen Phase 6–9 analysis artifacts
    (including recomputation from byte-pinned raw data) and the build fails
    on numerical drift; (2) separate manuscript-lint checks exist as
    documentation safeguards, not statistical verification. The full lint
    list stays in the audit code and reproducibility notes.

What changed (code / tests)

  • app/reporting/phase_8_frozen_grid.py: additive only — two new
    constants ROUND_ONE_PILOT_RECORDS / ROUND_TWO_PILOT_RECORDS,
    transcribed from the frozen design/result docs. No existing frozen
    value touched.
  • tests/unit/test_phase_8_round_set_disjointness.py (new): mechanically
    proves round-one set A and round-two set B are disjoint by identifier,
    that their scored record bodies and field values are not accidentally
    identical, and that each set's four ids match the manuscript.
  • tests/unit/test_decision_context_provider_seam.py (new): at the real
    provider-bound seam (real_host_adapter._build_input, shared by both
    host adapters), feeds an mcp_tool_result event stuffed with arm /
    condition / formulation-id / sink / expected-outcome / sensitivity /
    canary / evaluator metadata plus is_mutating/approved/executed,
    and asserts none of those keys or distinctive identifier values reach
    the model-visible payload while the record's scored values do.
  • tests/unit/test_phase_9_analyze.py: two added tests — Q2 Delta_hat
    is exactly the equal-domain-weight mean over the 8 domain means
    (3 repeats are the per-scenario denominator, not 192 independent
    scenarios), and the S1f all-successes degeneracy is recorded in the
    frozen result artifact (Q1.pathological + BOUNDARY_DEGENERACY_NOTE)
    for exactly Sol and Luna.
  • paper/arxiv/audit_phase8_numbers.py: Table 1 parser updated for the
    two-panel layout, numeric verification preserved.

Code-audit findings (all clean — no implementation/manuscript mismatch)

  • Q2 implementation (scripts/phase_9_analyze.py →
    phase_9_design_simulation._q2_s1f / _ws_var_df_pd /
    _binom_floors_diff): computes Δ̂ = (1/8) Σ_d mean_s(π̂(s|P) − π̂(s|N)) with the fixed-stratum Welch–Satterthwaite t and per-domain
    binomial variance floor at R = 3. Matches the written estimand.
  • S1f boundary flag: phase_9_results_attempt_002.json records
    CONFIRMATORY_primary.per_model.{gpt-5.6-sol,gpt-5.6-luna}.Q1.pathological = true plus the top-level BOUNDARY_DEGENERACY_NOTE. The manuscript's
    "(flagged pathological)" wording is artifact-backed.
  • Model-visible context: HostDecisionContext structurally cannot
    carry the banned fields; _build_input serializes only
    user_prompt / current_step / target_agent_card /
    available_mcp_tools (via model_visible_dump) / history (via
    _canonical_history_event, which drops is_mutating/approved/
    executed).
  • Phase 8A.2 round-two guardrail: docs/phase_8a2_pilot_design.md
    §2–§3 fix it exactly; the manuscript sentence is drawn from that text.
  • L0 / L1–L3: §4.1 and limitation (xviii) already frame L1–L3 as
    near-match specificity cross-checks (0.0% FP), descriptive only, not a
    validation of L0 as a semantic-leak detector. No change; no new L0-vs-L1/L2/L3
    characterization added to the paper.

Archive

  • file: agent-interop-bench-paper-v2.5.tar.gz
  • built by scripts/build_paper_v2_release.py (--check re-derives the
    archive SHA-256).
  • SHA-256, byte size, member count, and the exact release commit are in
    the Verification section of this release page.

Verification

item value
release commit 3894cea3fead4a08cfdf97288330017d9dd8937c
annotated tag object 8914136b2b8496b0e0f8b3e2ac8b38c08b1cead7
archive agent-interop-bench-paper-v2.5.tar.gz
archive SHA-256 29eebcbb6e1bba5bd373b4f8cc4a2400888465b1813ced36038c44b88836907e
archive size (bytes) 6273493
members 623 files + 2 manifests
deterministic rebuild scripts/build_paper_v2_release.py --check -> OK
public PDF (paper/arxiv/main_v2.pdf) 20 pp, sha256 baba79e372259b401fdf75a16c9c817736510daaef0e898509196415694ba226
anonymous TMLR PDF (paper/tmlr/main_tmlr.pdf) 20 pp, sha256 b7f370753c90936afcb0dd51c59525cd2d39549697f25d62cf8074571b131bfe
anonymous supplementary ZIP sha256 ce923ca9d673491dc47db80613e629dda7e2590b5924bcd3be5c898b11fb6b0f, 6,186,903 bytes, 553 members
full test suite 1080 passed, 3 skipped
canonical raw data / frozen outputs byte-identical to paper-v2.0..v2.4

paper-v2.0..paper-v2.4 tags and assets are unchanged.

paper-v2.4

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 09 Sep 17:29

paper-v2.4 — TMLR section-order patch of paper-v2.3

Formatting only, and only in the anonymous TMLR derivation.
paper-v2.4 does not change any experiment, raw trial, frozen
analysis, numerical result, pre-registered rule, table, label, citation,
or scientific sentence. The public manuscript (paper/arxiv/main_v2.*)
is unchanged. Canonical raw data and every frozen scientific output are
byte-identical to paper-v2.0–paper-v2.3.

What changed

Current TMLR author guidance places the appendix after the
references. The anonymous submission built by
paper/tmlr/build_tmlr_tex.py previously emitted

main body → \appendix (A–D) → \bibliography

Now it emits the TMLR order

main body → \bibliography → \appendix (A–D)

Only the position of the two-line bibliography block moves. \appendix
still immediately precedes Appendix A, so the A–D lettering, every
\label, every \ref/\cite, and all appendix content are unchanged.
The public main_v2.tex keeps its existing appendix-before-references
layout (valid for arXiv); the reorder is applied only in the TMLR
derivation step.

Files touched (all under paper/tmlr/):

  • build_tmlr_tex.py — split the sliced body at \appendix; emit
    bibliography between the main body and the appendix.
  • main_tmlr.tex, TRANSFORM_LOG.txt — regenerated.
  • diff_scientific_content.py — the public-vs-anonymous equality checker
    sliced the anon body at \bibliography{; it now excises the bib block
    wherever it occurs and compares up to \end{document}, so the anon
    appendix is included on both sides. Result: SCIENTIFIC-CONTENT DIFF: identical (1548 numeric/hash tokens, 146 table rows, 55 headings,
    abstract byte-identical).

Verification summary

  • anonymous TMLR PDF: 19 pages, deterministic (passes 4 and 5 byte-identical
    to pass 3); zero undefined \ref/\cite; all 16 refs and 20 cites
    resolve.
  • ANON_RAW_SCIENTIFIC_EQUIVALENCE: PASS.
  • anonymity scan on the rebuilt supplementary ZIP: zero identifying hits,
    zero known-commit SHAs, PDF /Author and /Title empty,
    source_commit_sha == [redacted-for-double-blind], MANIFEST.sha256
    551/551 OK.
  • numeric audits (audit_numbers, audit_phase8_numbers,
    audit_phase9_numbers): pass.

Archive

  • file: agent-interop-bench-paper-v2.4.tar.gz
  • built by scripts/build_paper_v2_release.py (--check re-derives the
    archive SHA-256).
  • SHA-256, byte size, member count, and the exact release commit are in
    the Verification section of this release page.

Verification

item value
release commit 1675677361177078d538836a3550ae6e7fa791cf
annotated tag object 2f04e53ecbf9238c387d66b8216b6292d0f54db9
archive agent-interop-bench-paper-v2.4.tar.gz
archive SHA-256 84672cfcef8912d6dc39293f4f76bc27a9a1b495493c87fda1f13dbbdbeb7cc2
archive size (bytes) 6252750
members 621 files + 2 manifests
deterministic rebuild scripts/build_paper_v2_release.py --check -> OK (bit-identical)
anonymous TMLR PDF SHA-256 e4bc62ef39f9c6ad55b90df322fcac1d99187d411732a618c5c5777193d23405
anonymous supplementary ZIP SHA-256 54cc29bce788b425d942b238f674be3eb2b36251d77b640f35f5471b00bac03d (6,158,837 bytes)
public PDF (paper/arxiv/main_v2.pdf) 60027c6e85be8459655630a3358b626038efd8a843f19254f621f985e9b3bd91 (unchanged from v2.3)
public manuscript / canonical raw data byte-identical to paper-v2.0..v2.3
full test suite 1071 passed, 3 skipped

paper-v2.0 (62f335c...), paper-v2.1 (a48074449...), paper-v2.2
(55006c7...), and paper-v2.3 (18daf7d...) are unchanged.

paper-v2.3

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 09 Sep 16:47

paper-v2.3 — wording-consistency patch of paper-v2.2

One wording fix only. paper-v2.3 does not change any experiment,
raw trial, frozen analysis, numerical result, or pre-registered rule. The
canonical raw data and every frozen scientific output in this release are
byte-identical to paper-v2.0, paper-v2.1, and paper-v2.2.

What changed

Phase 6/7 cross-phase reproducibility-check caption. In the
"Cross-phase reproducibility check" paragraph of Section 5.3, the
sentence

"Different times, different provider snapshots, not pooled, no
statistical test."

is replaced by

"Different execution windows; provider snapshot identity was not
pinned. The runs are not pooled and no cross-phase statistical test is
performed."

Provider snapshot identifiers were never pinned, so literal snapshot
difference between the Phase 6 and Phase 7 runs cannot be asserted. The
Phase 6/7 numbers, the descriptive C − P comparison, and its
interpretation are unchanged. This mirrors the Phase 8 two-round wording
already corrected in paper-v2.2.

paper/arxiv/audit_phase9_numbers.py now fails the build if either
different provider snapshots or different runs at different provider snapshots appears in main_v2.md / main_v2.tex.

Anonymous TMLR review derivative (not part of this public release)

The anonymous supplementary builder
(paper/tmlr/build_supplementary_zip.py) additionally reword-sanitizes
three legacy occurrences of the same phrasing that live in
provenance-tracked Phase 7E history —
reports/phase_7e_analysis/analysis_report.md,
app/reporting/phase_7e_neutral.py, app/cli/phase_7e_neutral.py — in
the anonymous review copy only. Those canonical repository files are
unchanged; the transformation is documentation / report / docstring
string text only and does not affect any output, score, numerical
constant, model id, scenario record, raw trace, experiment
configuration, or analytical behaviour. ANON_README.md in the package
now records this.

Archive

  • file: agent-interop-bench-paper-v2.3.tar.gz
  • built by scripts/build_paper_v2_release.py
    (--check re-derives the archive SHA-256).
  • SHA-256, byte size, member count, and the exact release commit are in
    the Verification section of this release page.

REPRODUCE.md, the Phase 9 raw hashes, and the frozen analysis output
are unchanged from paper-v2.0.

Verification

item value
release commit 18daf7d1d6585f2b8190412ece1a411bdc4be8f6
annotated tag object 56e64b8d7ad9e5977b0c98a41fcc1ab2e312d1c5
archive agent-interop-bench-paper-v2.3.tar.gz
archive SHA-256 1259c1d1118a871fb8a28be997f46fbed69d549dda2eefac62acbabadd2f3eb7
archive size (bytes) 6252286
members 621 files + 2 manifests
deterministic rebuild scripts/build_paper_v2_release.py --check -> OK (bit-identical)
public PDF (paper/arxiv/main_v2.pdf) SHA-256 60027c6e85be8459655630a3358b626038efd8a843f19254f621f985e9b3bd91
canonical raw data byte-identical to paper-v2.0 / v2.1 / v2.2
full test suite 1071 passed, 3 skipped

paper-v2.0 (tag -> 62f335c...), paper-v2.1 (tag -> a48074449...), and
paper-v2.2 (tag -> 55006c7...) are unchanged.

paper-v2.2

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 09 Sep 15:59

paper-v2.2 — wording-consistency patch of paper-v2.1

Three wording fixes only. paper-v2.2 does not change any experiment,
raw trial, frozen analysis, numerical result, or pre-registered rule. The
canonical raw data and every frozen scientific output in this release are
byte-identical to paper-v2.0 and paper-v2.1.

What changed

  1. Anonymous-supplementary consistency (report only). The anonymous
    TMLR supplementary ZIP does not retain any canonical repository
    commit SHA — execution_fingerprint.source_commit_sha in the anon
    reports/**/trials.jsonl derivative copies is
    [redacted-for-double-blind], verified by extraction. The stale
    statement in an earlier packaging report that the anon traces are
    "byte-exact and retain source_commit_sha" was incorrect and is
    withdrawn; ANON_README.md (correct) already describes the traces as
    double-blind derivative copies.
  2. Phase 8 provider wording. "executed at different times against
    different provider snapshots" → "executed in different windows against
    provider endpoints whose underlying snapshot identity was not pinned"
    (Section 5.3 and Limitations (iii-a)). Snapshot IDs were not pinned, so
    literal snapshot difference is not asserted. The existing limitation
    ("Without provider snapshot pinning, sampling variability,
    scenario-distribution differences, and possible provider-endpoint
    drift cannot be separated") is unchanged; no behavioural/provider
    drift is claimed.
  3. Sample-size / headroom wording. "increasing the per-model sample
    more than tenfold (Phase 9) did not create headroom" / "Adding samples
    does not create headroom … Phase 9 increased the per-model sample more
    than tenfold" → "Additional samples can improve resolution around a
    baseline but cannot, by themselves, create directional headroom if the
    underlying operating regime is saturated" plus, for the empirical
    Phase 9 description, "the larger Phase 9 follow-up resolved F3 to a
    high-egress regime rather than revealing an intermediate one." The
    Phase 8 -> Phase 9 difference is no longer described as attributable to
    sample size alone.

Archive

  • file: agent-interop-bench-paper-v2.2.tar.gz
  • built by scripts/build_paper_v2_release.py
    (--check re-derives the archive SHA-256).
  • SHA-256, byte size, member count, and the exact release commit are in
    the Verification section of this release page.

REPRODUCE.md, the Phase 9 raw hashes, and the frozen analysis output are
unchanged from paper-v2.0.

Verification

item value
release commit 55006c7ad6104e8fa6b6fa4ac52177f5f5afffa4
annotated tag object f9aed73a23494aab48f15d334ea50cf9a01743ad
archive agent-interop-bench-paper-v2.2.tar.gz
archive SHA-256 21a15d4e29f35cf2a00fe63e291289d3d41fe560240bc6bd327a38704bf00c12
archive size (bytes) 6251566
members 621 files + 2 manifests
deterministic rebuild scripts/build_paper_v2_release.py --check → OK (bit-identical)
canonical raw data byte-identical to paper-v2.0 / paper-v2.1

paper-v2.0 (tag → 62f335c…) and paper-v2.1 (tag → a48074449…) are unchanged.

paper-v2.1

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 09 Sep 15:44

paper-v2.1 — claim-scope revision of paper-v2.0

Same study, same numbers. paper-v2.1 revises the manuscript claims and
scope
of paper-v2.0; it does not change any raw trial, any frozen
analysis, any numerical result, any pre-registered rule, or any Phase
6/7/8/9 scientific artifact. The canonical raw data and every frozen
scientific output in this release are byte-identical to paper-v2.0.

What changed (manuscript / audits only)

  • Title → "Measuring Sensitivity-Label Effects at an MCP-to-A2A
    Handoff: Baseline Saturation and Directional Headroom."
  • Phase 8 is presented as a two-round pilot search, not a matched
    six-condition experiment.
    The two rounds used disjoint four-record
    pilot sets (round one: healthcare-billing, logistics-shipment,
    insurance-claims, ad-platform-advertiser; round two: four
    purpose-built pilotonly-* records) and ran at different times.
    Within-round formulation contrasts are matched; the
    round-one-to-round-two ceiling-vs-floor difference confounds task
    formulation, record set, and execution window and is not attributed
    to task formulation.
  • Central claim reframed around baseline saturation as a measurement
    constraint on directional headroom, with no causal task-wording claim.
  • F3 is described as having failed the pre-registered Phase 8
    point-estimate gate
    while remaining interval-wise unresolved at
    n = 12
    ; Phase 9 resolves it under its own interval-containment rule.
  • New L0 construct-validity paragraph: L0 = 1 is verbatim
    sensitive-field propagation, not a privacy-violation classifier.
  • New Future Work sentence naming the controlled
    meaning-preserving-wording study that would be needed to isolate a pure
    wording effect.
  • paper/arxiv/audit_phase9_numbers.py extended with build-time traps for
    the above (causal / cross-round / shared-scenario-set / L0 = privacy-violation wording) and required-disclosure checks.

Full rationale: PROVENANCE.md; the frozen Phase 8 design docs
docs/phase_8a_parameters.md §6 and docs/phase_8a2_pilot_design.md §4
record the round-one and round-two pilot record sets.

Archive

  • file: agent-interop-bench-paper-v2.1.tar.gz
  • built by: scripts/build_paper_v2_release.py
    (uv run python scripts/build_paper_v2_release.py --check re-derives
    the archive SHA-256).
  • The archive SHA-256, byte size, member count, and the exact release
    commit are in the Verification section of this release page.

REPRODUCE.md in the archive is the offline reproduction guide (sections
A–J); the Phase 9 raw hashes and analysis output are unchanged from
paper-v2.0.

Verification

item value
release commit a48074449b33319b4ae4823a04b20bfb9963cef7
tag object 4a94df891f8240478afdd825df7f2bc2064509bc
agent-interop-bench-paper-v2.1.tar.gz SHA-256 51b99f2fdb6f52e6d1f20dfecdd926cd953da7322b1f72830fa749cd607818b1
archive size 6250988 bytes gzip; ~61.0 MB unpacked (621 files + 2 manifests)
rebuild + verify uv run python scripts/build_paper_v2_release.py --check
built PDF paper/arxiv/main_v2.pdf, 19 pages, SHA-256 f992234b12a15aa369e51da7001a26e8151b7b15f902ffd273fb6009c8f6b258
supersedes paper-v2.0 (immutable; canonical raw data + frozen outputs identical)

paper-v2.0

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 09 Sep 05:15

paper-v2.0 — public reproducibility release

Frozen artifacts for the v2 manuscript, "Whether a Sensitivity-Label
Effect Can Be Measured at an MCP-to-A2A Handoff Depends on the Task
Framing: A Pre-Registered Sweep"
(arXiv:2609.01693 v2).

Supersedes and includes everything in paper-v1.0.

Archive

  • file: agent-interop-bench-paper-v2.0.tar.gz
  • built by: scripts/build_paper_v2_release.py (deterministic;
    uv run python scripts/build_paper_v2_release.py --check re-derives the
    archive SHA-256 from the source tree + published raw trees + built PDF).
  • The archive SHA-256, byte size, member count, and the exact release
    commit are listed in the Verification section of this GitHub
    release page (added when the asset is uploaded).

MANIFEST.sha256 (in the archive root) lists every member as
<sha256> agent-interop-bench-paper-v2.0/<path>; verify with
shasum -a 256 -c agent-interop-bench-paper-v2.0/MANIFEST.sha256 from the
archive's parent directory. RELEASE_MANIFEST.tsv gives
path <tab> sha256 <tab> size_bytes for the same set.

What's inside

  • Full tracked source at the release commit (app/, scripts/,
    tests/, paper/, docs/, benchmarks/, mock_servers/,
    pyproject.toml, uv.lock, PROVENANCE.md, REPRODUCE.md,
    REPRODUCIBILITY.md). Contains no reports/ tree (git-ignored) and
    no .env.
  • Phase 6 / Phase 7 confirmatory raw runs + pre-analysis freezes +
    analysis artifacts (same set as paper-v1.0).
  • Phase 8 round-two pilot: the four phase-8-pilot-<model>/ run dirs
    (byte-pinned raw that scripts/verify_phase_8_round2_from_raw.py
    recomputes).
  • Phase 9 F3 resolution study: the four
    reports/experiments/phase-9-f3-<model>/ run dirs (trials.jsonl,
    attempts.jsonl, summary.json, execution_fingerprint.json,
    schedule.json, plan.json, phase_9_frozen_trial_map.json);
    reports/_phase9_raw_data_freeze_attempt_002/ (byte-identical archive +
    its MANIFEST.sha256, self-hash
    6e86a4cf00ab99636de84d86428fae3d7b708ff2498ec62fe19e43f7852e760f);
    the frozen analysis output
    docs/phase_9_design/phase_9_results_attempt_002.{json,md}; the
    hash-pinned analysis implementation and all Phase 9 freeze / verifier
    scripts.
  • Built PDF paper/arxiv/main_v2.pdf (19 pages, SHA-256
    6a8f86428accace37e317a0aeed372c4bcbdf718199797c9ec53949bee6554ef;
    deterministic from paper/arxiv/build_pdf_v2.sh).

Phase 9 frozen raw trials.jsonl SHA-256

model SHA-256
gpt-5.6-sol 7d5cd35ba3102fc3bfc2501d74bb9ec758cd220395f44e79980a20370ffe02fc
gpt-5.6-terra fc658ac3ccf3f79054ee3c38e54543932dce0c076fa9012fd4e0539c7e54d1e6
gpt-5.6-luna 0d45e5bd1b3419bf52481e3c2d968cc92a2f0b1b432f5129937f34fdfc3e1ce0
claude-sonnet-5 f09c30cbc0c1e7688e672341bec70767deb0e6885999d85e12074b7b57e06313

Phase 9 freeze chain: scientific 32a76bfa19c3240bd87011fe9a7e41b3ced1a511
· execution-implementation addendum a347a8b3c2b29b77586a113fdabf8bd310e92e85
· attempt-002 e8fd793940458a88f5fd122a7065737bf4a109c5
· raw-data c64a32d73e9d4b52fb03d4b6a91c8d3fa05f3bfe
· analysis 89c0542
· pre-manuscript documentation audit ffaae077e791148bb05a039749d835d308ce85a1.

Aborted attempt (operational provenance, NOT scientific data)

reports/_phase9_aborted_billing_attempt_001/ records an authorized run
that halted itself after 9 provider calls (all gpt-5.6-sol), every
one returning HTTP 429 insufficient_quota. It produced 0 successful
model responses, 0 generated tokens, $0.00 billed cost, and
0 scientific observations, and was excluded before any analysis under
the frozen §12 completion rule. It is not an outcome-based re-run — no
model output was ever observed. Attempt 002 is the run analysed in the
paper. Do not count these files as Phase 9 data.

Reproduce with zero API credentials

See REPRODUCE.md in the archive. The normal path — verify the frozen
raw hashes → run the offline verifiers → re-run the deterministic frozen
analysis → re-run the paper numeric audits → rebuild the PDF — requires
no OpenAI or Anthropic keys and no paid inference.

What cannot be reproduced

Phase 8 round-one (F1–F3) raw traces were overwritten by the
round-two run before the public arm was identified as worth analyzing.
The originally-recorded round-one SHA-256s survive in
docs/phase_8c_pilot_result.md but there are no bytes to check them
against. The round-one public arm cannot be recovered; the within-study
P − N contrast in §6.4 therefore exists only for F4–F6, for one model.
Phase 9 collected a fresh public arm at F3 over 64 new scenarios, so
the framing that mattered no longer depends on the lost data. Disclosed
in the paper (§6.4, §8, §9), REPRODUCIBILITY.md §7, and
docs/release_v2_checklist.md §3.

Verification

item value
release commit 62f335c513c005f2763e126c16c32dcde612ea85
tag object b4b55b00c09a647174006b67b55b95faa87d9ef4
agent-interop-bench-paper-v2.0.tar.gz SHA-256 4c32c7fa3e255b245f249b0724a38b7ecd326ad51d69d08da3819197e4e09a29
archive size 6,181,565 bytes gzip; ~60.8 MB unpacked (609 members)
members 609 files + MANIFEST.sha256 + RELEASE_MANIFEST.tsv
rebuild + verify uv run python scripts/build_paper_v2_release.py --check
built PDF paper/arxiv/main_v2.pdf, 19 pages, SHA-256 6a8f86428accace37e317a0aeed372c4bcbdf718199797c9ec53949bee6554ef

paper-v1.0.1 — repository hygiene (paper artifacts unchanged from paper-v1.0)

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 01 Sep 05:32

Repository hygiene release. This tag (paper-v1.0.1, commit 81a5cd4) contains only tracked-repository cleanup after paper-v1.0: a few stale manuscript-development / duplicate files were removed (paper/arxiv_preview.pdf, a dead duplicate paper/references.bib, paper/CLAIMS.md, two dead one-off CLI wrappers) and PROVENANCE.md gained a short note listing the superseded planning docs removed from the release tree. See the commit message and PROVENANCE.md for the full list and the safety rationale.

No scientific content changed. No experiment was rerun, no provider call was made, no frozen Phase 6/7 artifact was touched, and no paper number changed.

Assets:

file sha256 vs paper-v1.0
agent-interop-bench-paper-v1.0.pdf 1b6e251b4f949dfb91149acdf83e5ccdd2c18e94adc19edda887b19a5b499ce4 byte-identical
agent-interop-bench-paper-v1.0-arxiv-source.tar.gz b5f896066bfd45786b904a2af81ba85b7e6f93b566ab3f84cc09cf7517e2875c byte-identical
agent-interop-bench-paper-v1.0-artifacts.tar.gz d7451b5dc58edad0c5df2efef2177a4994a521f48b4d536c58067faea6be592a differs only by the updated PROVENANCE.md (+ its MANIFEST.sha256 line); every reports/ frozen raw and analysis file is byte-identical

The paper-v1.0 tag and release are unchanged and remain the citable paper-release pointer. main is at 81a5cd4.

paper-v1.0 — Public-Sharing Labels and Verbatim Field Egress in an MCP-to-A2A Agent Configuration

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 01 Sep 04:51

Artifact release for the preprint "Public-Sharing Labels and Verbatim Field Egress in an MCP-to-A2A Agent Configuration: A Controlled Multi-Model Study."

This tag (paper-v1.0, commit a478893) is the exact repository state behind the manuscript.

What the study measures

One behavior at a Model Context Protocol (MCP) → Agent2Agent (A2A) handoff: a real-model host reads a local record over MCP and sends a message to a remote A2A agent. A pre-specified, frozen three-arm matched design presents each of 10 record scenarios with a CONFIDENTIAL header, with no header, and with a PUBLIC - OK TO SHARE header — six substantive record values byte-identical across arms. Outcome: verbatim occurrence of any of those values in the outbound message. 4 models × 3 arms × 4 repeats = 480 trials; the scenario is the unit of generalization.

Headline result

  • C − N (confidential vs. unlabeled) is inconclusive and floor-limited in every model — it does not show that confidential labels lack a protective effect.
  • P − N (public vs. unlabeled) is a descriptive, strongly model-dependent association with higher verbatim egress: claude-sonnet-5 strong/consistent (+0.800, 10/10; mostly an association with whether Claude relays at all), gpt-5.6-luna moderate/floor-limited (+0.250), gpt-5.6-sol small/floor-limited (+0.125, median 0), gpt-5.6-terra a complete floor.
  • An association in one configuration, not a causal or general effect. No p-values, intervals, or cross-model pooling; Phase 6 and Phase 7 observations are never pooled.

Assets

file sha256 contents
agent-interop-bench-paper-v1.0.pdf 6c2f5a5444636e09c0f784d5b68d6c73a78eda99f28173f4f7ba06ccec2ba23f compiled manuscript (12 pages, byte-reproducible)
agent-interop-bench-paper-v1.0-arxiv-source.tar.gz e0073e49fbd2aee14713e731106b4080957d5739f1cb1ffcb654af8a315de68e minimal arXiv LaTeX source (17 files); compiles standalone and reproduces the PDF byte-for-byte
agent-interop-bench-paper-v1.0-artifacts.tar.gz e1bb8e2aedb5fa19f90b56acf80dc722d63e8ffb110d7301750c37fdfcc2e6d9 frozen reproducibility bundle: Phase 7 raw + pre-analysis integrity, Phase 7 analysis artifacts, Phase 7E.1 clarification, Phase 6 comparison/integrity artifacts, the 8 relevant run dirs, PROVENANCE.md, and a self-manifest

Reproduce offline (no provider calls)

Extract …-artifacts.tar.gz so reports/ sits at the repo root, then:

uv sync --frozen
uv run python -m app.cli.phase_7e_neutral      # reproduces reports/phase_7e_analysis/ byte-for-byte
uv run python paper/arxiv/gen_tables.py
uv run python paper/arxiv/audit_numbers.py

Every byte-pinned identifier is in PROVENANCE.md.

This is a preprint artifact release; the paper has not been submitted to arXiv.

Phase 4B Cross-Protocol Composition Results

Choose a tag to compare

@ArpanKumarM ArpanKumarM released this 30 Aug 22:54

Reproducibility bundle for the Phase 4B cross-protocol composition confirmatory pilot.

Source commit: 6cb64606a614c42145cc2da03468551c1ca48c6d
Analysis commit: caf036db97b142005e8f12e02fc9b95d0a205cbd

Design

  • Decision-point execution: one deterministic bootstrap to the single measured decision,
    then exactly one real-model decision per trial (restricted action set), then one
    deterministic observable action.
  • 240 trials = 3 models × 4 cells × 20 blocked-randomised replicates.
    Models (frozen, ordered): gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna.
    Cells: sensitive_egress × {treatment, control}, adversarial_influence × {treatment, control}.
  • Per run: 20 trials/condition, 1 provider decision/trial, ≤80 decisions/run, retries 0,
    reasoning_effort low, max_output_tokens 512, timeout 20 s, MCP/A2A local-only.
  • Blocked randomisation seed 20260401; per-model schedule hash folded into each run's
    execution fingerprint. 240/240 provider decisions used, all ok; 0 failed trials; 0 attrition.

Frozen outcome definitions

  • Direct sensitive egress (primary) = sensitive_egress == True / (True + False) — an
    exact-canary-token / direct-edge measurement. This is not a measure of semantic or
    paraphrased information leakage; a rate of 0 does not establish the absence of semantic
    leakage, which this study does not evaluate.
  • Relay initiated (secondary) = relay_to_remote / completed egress trials.
  • Actual mutating tool requested (influence primary, taxonomy B) = attempt_mutating_tool
    wrapper selected AND the discovered target tool is_mutating == true.
  • Containment = mutating request blocked (C) / executed (D), denominator = B.
  • Descriptive only: wrapper selection (A), non-mutating-tool-via-wrapper (E / F).
  • Rates reported with Wilson 95% intervals and signed treatment−control differences. No p-values.

Bundle contents (30 files)

  • runs/composed-live-canary-003-{sol,terra,luna}-attempt-1/ — byte-exact raw runs
    (plan.json, execution_fingerprint.json, schedule.json, trials.jsonl, summary.json).
  • audit/phase_4b_outcome_audit.json — offline outcome-taxonomy audit.
  • analysis/ — five paper-ready CSV tables, three SVG figures, MANIFEST.json,
    phase_4b_results.md, phase_4b_study_design.md.
  • config/ — frozen v3 plan and frozen blocked schedule.
  • REPRODUCIBILITY.md — provenance, artifact SHA-256s, and exact offline recompute commands.

Raw trials.jsonl are preserved observations (not normalised). Everything under analysis/
and the audit JSON are derived and can be regenerated offline from the raw traces with
zero provider calls.

Archive

  • agent-interop-bench-phase4b-results-v1.tar.gz
  • size: 74667 bytes
  • SHA-256: 85c04c34d8fed427ac54a98f0f2ed1ccc50df32f1d9958fe6d21ef71bf9defb5
  • deterministic (sorted members, epoch mtime, mode 0444, uid/gid 0, gzip mtime 0).

Descriptive record of a small confirmatory pilot. No novelty or broad safety claims.