Turn research papers into verified, executable code.
Upload a PDF paper → retrieve relevant sections → generate implementation code → execute in a sandbox → self-debug on errors → verify faithfulness with an independent critic agent.
Unlike existing paper-to-code tools that generate plausible-looking code and judge quality with an LLM, RepliCode closes the loop: code that doesn't run gets fixed automatically, and an independent critic verifies every generation against the paper.
Tested on a well-known paper (Attention Is All You Need, 2017) and a paper published May 2025 that the model has never seen (TabReason, ICML 2025).
| Paper | Query | Iterations to success | Executes | Critic | Confidence |
|---|---|---|---|---|---|
| Transformer (2017) | Attention mechanism | 1 | ✅ | Faithful | 0.95 |
| TabReason (May 2025) | GRPO + reward functions | 1 | ✅ | Faithful | 0.90 (0.95 for Cached) |
2 queries produce executable, faithful code. Complex queries self-debug via real traceback feedback in up to 4 iterations.
| PaperCoder / Paper2Code | RepliCode | |
|---|---|---|
| Execution feedback | ❌ Generates code, never runs it | ✅ Runs code in sandbox, feeds tracebacks back to LLM |
| Self-debugging | ❌ Single-pass generation | ✅ Iterative refinement loop (up to N iterations) |
| Verification | LLM-as-judge | ✅ Independent critic agent checks code against paper |
| Retrieval | Reads full paper into context | ✅ Hybrid search (vector + BM25 + RRF) over paper chunks |
| Works on new papers | Relies on model memory | ✅ Retrieval-grounded — works on papers the model has never seen |
| Evaluation | LLM-judge score | ✅ Execution success + AST checks + critic report |
PDF ─→ Parse ─→ Chunk ─→ Index (Chroma + BM25)
│
User Query ────┤
▼
┌── Planner ──┐
│ (once/paper) │
└──────┬───────┘
▼
┌─── Coder Loop ───┐
│ generate code │
│ → sandbox.run() │
│ if fail → refine │
└──────┬────────────┘
▼
┌── Critic ──┐
│ code vs │
│ paper │
└──────┬─────┘
▼
┌── Eval ──┐
│ metrics │
└──────────┘
8 pipeline stages, 3 agents:
| Stage | What it does |
|---|---|
| 1. Parse | PDF → typed text blocks, tables, figures (PyMuPDF4LLM) |
| 2. Chunk | Three strategies: simple / hybrid / full-agentic |
| 3. Index | Chroma vector index + BM25 lexical index |
| 4. Retrieve | Reciprocal Rank Fusion over both indexes |
| 5. Generate | GPT-4o with strict prompting + citation tracking |
| 6. Execute | Sandbox execution with timeout |
| 7. Refine | Feed traceback → LLM → re-execute (up to N iterations) |
| 8. Evaluate | Lexical overlap, AST checks, execution success |
| Agent | Role |
|---|---|
| Planner | Once per paper: extracts file plan, claims, equations, dependencies |
| Coder | Per query: generate → execute → refine loop |
| Critic | Per generation: independently reviews code against retrieved chunks |
Stay tuned for the full technical walkthrough.
git clone https://github.com/YOUR_USERNAME/replicode.git
cd replicode
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
export OPENAI_API_KEY=sk-... # Windows: set OPENAI_API_KEY=sk-...
streamlit run app.pydocker build -t replicode .
docker run -p 8501:8501 -e OPENAI_API_KEY=sk-... replicodepython main.py paper.pdf "Implement the attention mechanism from Section 3.2"pytest tests/ -v # 36 tests
| Layer | Choice |
|---|---|
| PDF parsing | PyMuPDF + PyMuPDF4LLM |
| Embeddings | OpenAI text-embedding-3-large |
| Vector store | Chroma (local, persistent) |
| Lexical search | BM25 (rank-bm25) |
| LLM | GPT-4o (generator, planner, critic, refiner) |
| Sandbox | subprocess + tempdir (Docker for isolation) |
| Orchestration | LangGraph state machine |
| UI | Streamlit |
replicode/
├── app.py Streamlit UI
├── main.py CLI entrypoint
├── Dockerfile
├── requirements.txt
│
├── src/
│ ├── parser.py Stage 1 — PDF parsing
│ ├── chunking.py Stage 2 — chunking strategies
│ ├── retriever.py Stages 3+4 — hybrid retrieval
│ ├── generator.py Stage 5 — code generation
│ ├── sandbox_runner.py Stages 6+7 — execute + refine
│ ├── evaluation.py Stage 8 — metrics
│ ├── repro_evaluator.py Numerical reproduction scoring
│ ├── ui_helpers.py Streamlit renderers
│ ├── draft_refine_pipeline.py Optional: draft-refine alternative
│ │
│ ├── sandbox/
│ │ ├── base.py Sandbox protocol
│ │ └── local.py LocalSandbox implementation
│ │
│ ├── agents/
│ │ ├── planner.py Planner agent
│ │ ├── critic.py Critic agent
│ │ └── pipeline.py LangGraph wiring
│ │
│ └── spec/
│ └── schema.py ReproSpec contract
│
├── tests/ 36 tests, no API key needed
├── docs/
│ ├── screenshots Demo screenshots
│
Execution ≠ correctness. Code that runs isn't necessarily code that's right. The execution metric confirms the code doesn't crash — not that it reproduces the paper's results. Output assertions (shape checks, NaN checks) and the ReproEvaluator narrow this gap but don't close it entirely.
The critic is an LLM, not an oracle. It catches many hallucinations but can miss subtle ones (wrong activation, transposed dimensions) and can false-positive on legitimate choices the paper underspecifies. Zero findings means the critic didn't catch anything — not that nothing is wrong.
Scanned and image-heavy PDFs. The parser extracts text layers. Scanned PDFs produce empty text. Equations rendered as images come through garbled. Papers where the method lives primarily in figures or pseudocode images are underserved.
Retrieval caps generation. If the retriever ranks the wrong chunks highly, the generator can't produce faithful code for that section. This is the fundamental constraint of any RAG system.
No formal benchmarking. The comparison with Paper2Coder describes architectural differences. A rigorous head-to-head on the same papers with the same queries hasn't been done — the results table is a demonstration, not a benchmark.
MIT
Built by Dr. Lakshay Arora. Inspired by the Paper2Code architecture (KAIST, 2025), with execution grounding, independent verification, and retrieval augmentation as the core differentiators.