Repository navigation
Research 3 Reproduction
The study branch is 03-open-agent-self-preservation. Start with its reviewer quickstart. The lightweight replay below is available on the Research 3 study branch after PR #85. No model or Databricks account is needed for saved-evidence replay.
The primary release is immutable. Its original manifest pins every member. The CLI's default evidence scope downloads datasets.zip, evidence.zip and MANIFEST.json: 4,298,285 compressed bytes and 302 required members after extraction. Verification hashes every required member against the same complete manifest. It does not claim to verify adapters that were not downloaded. Use --scope full to download and verify the complete package.
python -m src.public_artifacts fetch --root ../research3-evidence
python -m src.public_artifacts verify --root ../research3-evidence
python -m src.public_artifacts replay --root ../research3-evidence --output ../research3-replay
python -m src.workflow_failure_summary --root ../research3-evidence --output ../research3-failure-summary.json
The evidence download is 4.3 MB; all primary assets total 618 MB. Outputs must be new and outside both the checkout and original evidence. Saved-score replay allocates no GPU, loads no model, contacts no annotation provider and starts no cloud job. Its saved summary is identical whether the evidence-only or complete package is supplied.
For model inference, use a separate complete download:
python -m src.public_artifacts fetch --scope full --root ../research3-model-artifacts
python -m src.public_artifacts verify --scope full --root ../research3-model-artifacts
Python calls retain the earlier full-package defaults: fetch(root) and verify(root). Pass scope="evidence" explicitly for a lightweight Python call. replay(root, output) checks its required replay evidence. Model gate and workflow inference retain full-package verification. The MCP replay tools verify evidence scope and disclose that scope.
The steering profile's evidence scope downloads diagnostic_outputs.zip and its original manifest, excluding fitting arrays and the executed source archive. Use --profile steering --scope full for the fitting audit. No experiment configuration or archive is rewritten.
The paper has a local Model Context Protocol (MCP) companion for compatible agents. It follows the tools/resources/prompts organization described in Paper2Agent, with direct wrappers around this repository's verified functions. We have not run Paper2Agent's automatic multi-agent construction pipeline or measured improved LLM answer accuracy.
Use Python 3.12 and install the optional companion dependencies:
python -m pip install -r requirements-mcp.txt
python -m src.paper_mcp
This starts a local stdio server, not a public web endpoint. Without artifact downloads, it exposes the complete accepted manuscript in Markdown (including four equations, six tables and one figure link), recorded results, cohort roles, claim-to-evidence pointers, source identities and reproduction instructions. It needs no Databricks credentials, model loading or GPU. Nothing starts on import.
| Tool | Purpose |
|---|---|
get_cohort |
Explain membership, units, split roles and completeness |
get_result |
Return scoped metrics with denominators and limitations |
get_claim |
Resolve a claim to exact recorded evidence |
list_artifacts |
List released adapters, download URLs and hashes |
verify_artifacts |
Verify a configured public release's file identities |
replay_results |
Rescore saved evidence on CPU into a new report |
Five resources and two prompts guide evidence explanation and saved-result replay. Replay reports have a separate resource template. The host configures artifact and output roots; tool callers cannot supply filesystem paths, code or commands. Original evidence stays immutable. The companion cannot launch neural inference, training, provider calls or Databricks jobs.
For a client that accepts mcpServers configuration, use absolute paths. Running
the server file directly works from any working directory:
{
"mcpServers": {
"research3": {
"command": "/absolute/path/to/venv/python",
"args": ["/absolute/path/to/sp_lense/src/paper_mcp.py"]
}
}
}On Windows the command can end in .venv/Scripts/python.exe; forward slashes are
accepted. This example is client-neutral and does not install or change a client.
To enable replay, fetch the public profiles using the commands below and add
--artifacts /absolute/path/to/primary and
--steering-artifacts /absolute/path/to/supplement to the server arguments.
--output-root optionally selects a separate report directory outside the checkout
and evidence. The default is ~/Research3/artifacts/mcp_replays.
Validate the actual stdio protocol from an unrelated working directory:
python -m src.paper_mcp_validate --output <new-validation-report.json>
python -m src.paper_mcp_validate --artifacts <primary-directory> --steering-artifacts <supplement-directory> --output <another-new-validation-report.json>
The 24 predeclared query fixtures cover denominators, historical split reuse,
incomplete screens, claim provenance and rejected out-of-scope requests. All registered
analysis views, cohorts and claims are also exercised. These are fixed
protocol/evidence checks, not a benchmark of an LLM selecting tools or answering
questions. The second command additionally verifies the required evidence from both releases before and after
replay and checks returned scores against the recorded paper results.
The checked report is in paper/mcp_validation.json. The research3-paper-mcp-v2 release preserves the earlier companion snapshot. Current clean Word, PDF and Markdown manuscripts are maintained on the Research 3 branch; earlier releases remain immutable.
The failure-localization counts are independently recomputed from the original public trajectories, with no model execution:
python -m src.workflow_failure_summary --root <primary-directory> --output <new-report.json>
paper/manuscript.md is exported from the clean accepted Word copy, with its
source SHA-256. python -m src.paper_export refreshes the export after authoring;
the source manifest must be refreshed before serving a changed paper. Manuscript,
catalog, figure and result identities are checked before the companion starts;
fixed-file drift is rejected while it is running. Restart after a checkout update.
Replay recalculates finite saved outcomes; it does not recreate model generation, retrain weights or establish a new holdout result. The diagnostic supplement supplies previous teacher tokens on three training responses. Prediction agreement is not autonomous success. See the manuscript and paper/results.json for denominators and limits.
The earlier private workspace audit is optional and separate from public replay. With already authorized workspace access and local configuration:
python -m src.reproduce --fetch-evidence --evidence-root ../private-evidence
python -m src.reproduce --execute --evidence-root ../private-evidence --output ../private-replay
Its original identities remain in config/reproduction.yaml. Shared helpers check paths, file identities, historical summary fields and the original stopped-screen proof. The scientific scorers and their original source pins remain unchanged.