reprobe — AutoUI open-data reproducibility harness

Give a URL, get a runnable check. reprobe fetches a research artifact (a git repo or an archival DOI), detects what it is, builds a sandboxed environment, runs the data analyses (Python / R / Jupyter / R Markdown), and writes a report that maps to ACM artifact badges and FAIR principles. Unity prototypes are supported at a structural tier. A small local LLM (Gemma 4 (e4b) via Ollama, fully offline) optionally assists — but only ever advises.

Built for the AutoUI Open Data chair to reuse year over year: the annual job is editing config/pins.yaml, not the code.

Full architecture and rationale: docs/DESIGN.md. Running it as chair each year: docs/chair-runbook.md.

The one idea: a trust boundary

The orchestrator is the only trusted process. Untrusted author code runs only inside ephemeral containers with --network none, a non-root user, all capabilities dropped, a read-only root filesystem, and cpu/memory/pid/time caps. The LLM returns data only — it has no shell, no socket, and cannot execute anything. reprobe --no-llm is fully functional and deterministic.

Install

Important

Docker must be running — this is a hard prerequisite. reprobe executes every analysis inside Docker, so a working, running Docker Engine is mandatory (reprobe doctor verifies it; only the code-free commands reprobe detect and reprobe run --no-run work without it).

Start Docker yourself first. Docker Desktop does not always launch automatically after login or a reboot — start it manually (open Docker Desktop) and wait for the daemon to be ready (docker info succeeds, or reprobe doctor shows docker … ok) before any reprobe pull/run/batch.

• Windows — install Docker Desktop, which needs either the WSL 2 backend (recommended: enable "Use the WSL 2 based engine" in Docker Desktop → Settings) or, if WSL 2 is unavailable to you, the Hyper-V backend. Both rely on hardware virtualization, so on a managed or locked-down machine you will need administrator / IT permission to install Docker Desktop and turn virtualization on. Without WSL 2 or that permission, Docker will not start and no runs are possible.
• Linux — the daemon must be running and your user must be allowed to use it: either be in the docker group, or use rootless Docker.
• macOS — Docker Desktop (or an equivalent daemon) must be running.

pip install -e .            # Python 3.11+; needs Docker available for runs
reprobe pull               # fetch the pinned base images (published, no login needed)
reprobe doctor --smoke     # self-check: config, Docker, base images, Ollama, sandbox

The pinned Python/R analysis environments are published images — ghcr.io/m-colley/reprobe-base-py:2026.1 and …-base-r:2026.1, built by images/build-images.sh. Authors can test their code against the exact same environment reviewers use (see docs/chair-runbook.md).

Use

# Single artifact (fetch + detect + sandboxed run + report)
reprobe run https://github.com/jorgpg5/PDRA_XAI_OS

# A Zenodo deposit by DOI
reprobe run https://doi.org/10.5281/zenodo.123456

# Detection only — no code executed (shows the run plan reprobe would use)
reprobe detect ./examples/example-python

# A whole review season -> sortable dashboard + badges.json + badges.csv
reprobe batch submissions.csv      # add --resume to continue an interrupted season

# Available badge only, never execute code:
reprobe run <url> --no-run

# Pull git-lfs data during fetch (opt-in, hardened; default keeps skip-smudge):
reprobe run <url> --allow-lfs

# Give long-running steps a bigger budget (notebooks default to 90 min;
# clamped to limits.yaml:max_timeout_s, and recorded in the report):
reprobe run <url> --timeout 10800

# Code in git, data on OSF/Zenodo/Dryad/figshare/Dataverse — merge both:
reprobe run https://github.com/ciao-group/PerceivedRisk --data https://osf.io/cwd6h

# ...into the directory the code expects, and repeat for several deposits:
reprobe run <code-url> --data <deposit-url>::dataset --data <other-url>::models

Artifacts split across two repositories are the normal case, not the exception: the code is in git and the data sits in a repository the README links to in prose. Fetching either half alone says nothing — the code half dies at the first read_csv, the data half has nothing to run. --data fetches the deposit with the same fetchers used for a submission and merges it into the artifact tree before detection. It never overwrites a file the code source already provides (collisions are reported), and it never strengthens the Available badge — that still follows the primary source's pin alone. Authors can declare the same thing permanently with data_sources: in the manifest; when neither is present and the README links a data repository, the report says so and prints the command.

Declared environments are built, not assumed. A requirements.txt is pip-installed and an environment.yml is built with micromamba (the pinned base is a micromamba image) during the sandboxed install phase; the analysis then runs with that interpreter, still under --network none. The report states what building it does not reproduce — conda activate hooks, environment variables, and exact versions, since channels resolve fresh with no lock file.

Missing R packages are handled automatically: reprobe detects the CRAN packages a repo needs (library()/require()/pkg:: and DESCRIPTION) and installs the CRAN-available ones — pinned to a dated snapshot (r.cran_snapshot in config/pins.yaml) — during the sandboxed install phase, so the author analysis still runs with --network none. Packages not on CRAN are reported, not faked.

Outputs land in work/<submission>/out/: report.json (machine-readable), report.md, and a single-file report.html.

What the badges mean here

Badge	reprobe behaviour
ACM Artifact Available	Granted automatically — but only on an archival persistent ID (Zenodo version DOI, Software Heritage SWHID). A bare GitHub commit is reproducibly pinned yet not archival, so it becomes a candidate with a "deposit this in Zenodo" note. No code is executed for this badge.
ACM Artifacts Evaluated — Functional	Proposed as a candidate for a human when declared steps pass and produce a declared output. Never auto-granted. Opt-in.
ACM Results Reproduced	Not auto-granted; reprobe surfaces produced artifacts to help the human reviewer.
FAIR	Scored from fetch metadata (persistent ID, license, formats, manifest).

The cardinal rule: the harness grants only Available automatically and proposes everything deeper. Over-claiming is the failure mode it avoids — every step reports both what it verified and what it explicitly did not.

Authors: make it effortless (optional)

Drop an autoui-repro.yml in your repo to remove all guesswork (existing CODECHECK codecheck.yml is also read). See src/reprobe/schemas/autoui-repro.schema.json and examples/example-python. Two knobs worth knowing:

• environment.r_packages: [pkg1, pkg2] — pin exactly the CRAN packages to install (reprobe also auto-detects them, so this is only for overriding).
• data: [{path, source, checksum}] — individual data files reprobe should download (from an http(s) source) into the run tree before your analysis runs.
• data_sources: [{source, into}] — whole external deposits (OSF, Zenodo, Dryad, figshare, Dataverse, or a direct URL) merged into your tree before the run. This is the fix if your data lives on OSF and your code on GitHub: without it the harness has only your README's prose to go on, and every step fails on a missing input. into is the subdirectory your code reads from (default: the repo root).
• paper: {doi, pdf} — the paper this artifact reproduces. reprobe compares your produced numbers against its claims and shows a reviewer the differences (advisory only — it never grants a badge). Committing the PDF is worth far more than the DOI: paywalled venues refuse automated downloads, leaving only the abstract, which states "significantly higher" rather than F(1,16)=11.12. Without a manifest reprobe still looks for a PDF in the repo and a DOI in CITATION.cff / the README.

Status

MVP + breadth: fetch (git / Zenodo / figshare / Dryad / OSF / Dataverse / Software Heritage / anonymous.4open.science / local / resolvable DOIs), detection (runnable code and non-code artifacts: video / audio / dataset / document / 3D — the AutoUI submission-form categories), sandboxed Python/R/Jupyter/Rmd execution with CLI args + a split-by-language dependency-install phase, Unity T0 structural, badges, reports with a copy-pasteable author-feedback block, batch dashboard (+ --resume, badges.csv), a golden-report regression (reprobe doctor --golden), and the advisory LLM. Base images build via images/build-images.sh and publish to ghcr.io/m-colley/reprobe-base-{py,r}:2026.1. Remaining: repo2docker fallback and Unity compile/build tiers — scoped in docs/DESIGN.md §11.