diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml new file mode 100644 index 0000000..f0587ba --- /dev/null +++ b/.github/workflows/pages.yml @@ -0,0 +1,31 @@ +name: Deploy landing page + +on: + push: + branches: [main] + paths: ["site/**", ".github/workflows/pages.yml"] + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: true + +jobs: + deploy: + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - uses: actions/checkout@v4 + - uses: actions/configure-pages@v5 + - uses: actions/upload-pages-artifact@v3 + with: + path: site + - id: deployment + uses: actions/deploy-pages@v4 diff --git a/README.md b/README.md index 140983a..a048756 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,8 @@ project still builds and passes its own test suite. [![Python 3.12](https://img.shields.io/badge/Python-3.12-3776AB.svg)](server/pyproject.toml) [![Next.js 15](https://img.shields.io/badge/Next.js-15-000000.svg)](web/package.json) +Accepted to **[EMNLP 2026 — System Demonstrations](https://pischool.github.io/refactor-platform/)**. + Run detail: agent steps on the left, live terminal and diff in the centre, evaluation checks and token accounting on the right @@ -176,6 +178,16 @@ Publishing it to a network requires a reverse proxy that provides TLS and authentication. To share results instead, export a run: the archive carries the evidence without the deployment. +## Contributors + +- [Aziz Ben Amor](https://www.linkedin.com/in/aziz-benamor/) — Pi School (equal contribution) +- [Drish Mali](https://www.linkedin.com/in/drish-mali/) — Pi School (equal contribution) +- [Mann Acharya](https://mannacharya.com) — Pi School (equal contribution) +- [Vijayasri Iyer](https://www.linkedin.com/in/vijayasriiyer/) — Pi School (corresponding author) +- [Sébastien Bratières](https://www.linkedin.com/in/sebastien-bratieres/) — Pi School, Translated + +[![Pi School](web/public/brand/pischool-wordmark.png)](https://picampus-school.com/) + ## License Platform source code is [MIT](LICENSE). RefactorBench, SWE-Refactor, diff --git a/docs/README.md b/docs/README.md index fae14b9..e53985b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,7 @@ # Documentation +[← Back to README](../README.md) + | Document | Scope | |---|---| | [Deployment](deployment.md) | Installing the stack, remote hosts, upgrades and backups | diff --git a/site/assets/chunking-frontier-light.png b/site/assets/chunking-frontier-light.png new file mode 100644 index 0000000..6595b70 Binary files /dev/null and b/site/assets/chunking-frontier-light.png differ diff --git a/site/assets/pischool-wordmark.png b/site/assets/pischool-wordmark.png new file mode 100644 index 0000000..fba1e0f Binary files /dev/null and b/site/assets/pischool-wordmark.png differ diff --git a/site/assets/refactorbench-models-light.png b/site/assets/refactorbench-models-light.png new file mode 100644 index 0000000..024633f Binary files /dev/null and b/site/assets/refactorbench-models-light.png differ diff --git a/site/assets/refactorbench-prompt-modes-light.png b/site/assets/refactorbench-prompt-modes-light.png new file mode 100644 index 0000000..3d9176b Binary files /dev/null and b/site/assets/refactorbench-prompt-modes-light.png differ diff --git a/site/assets/swe-verification-funnel-light.png b/site/assets/swe-verification-funnel-light.png new file mode 100644 index 0000000..a6cd970 Binary files /dev/null and b/site/assets/swe-verification-funnel-light.png differ diff --git a/site/assets/ui-evaluation-light.png b/site/assets/ui-evaluation-light.png new file mode 100644 index 0000000..fc2ad48 Binary files /dev/null and b/site/assets/ui-evaluation-light.png differ diff --git a/site/assets/ui-run-detail-light.png b/site/assets/ui-run-detail-light.png new file mode 100644 index 0000000..881585f Binary files /dev/null and b/site/assets/ui-run-detail-light.png differ diff --git a/site/assets/ui-terminal-light.png b/site/assets/ui-terminal-light.png new file mode 100644 index 0000000..a7f7b3e Binary files /dev/null and b/site/assets/ui-terminal-light.png differ diff --git a/site/assets/ui-wizard-light.png b/site/assets/ui-wizard-light.png new file mode 100644 index 0000000..2c8f715 Binary files /dev/null and b/site/assets/ui-wizard-light.png differ diff --git a/site/index.html b/site/index.html new file mode 100644 index 0000000..bf9365d --- /dev/null +++ b/site/index.html @@ -0,0 +1,313 @@ + + + + + + RefactorPlatform — EMNLP 2026 System Demonstrations + + + + +
+
+

RefactorPlatform

+

+ An Open-Source Harness for Controlled Evaluation of Repository-Scale + Refactoring Agents +

+ + +

1Pi School, 2Translated

+ +

EMNLP 2026 · System Demonstrations

+ +

*Equal contribution.

+ + +
+ +
+
+
+ Coding tool step: the agent plugin, a free-text model field, and the provider's searchable model catalogue +
(a) Configure
+
+
+ The same task's terminal, replayed from the captured PTY bytes with workspace paths redacted +
(b) Watch it run
+
+
+ Run detail: agent steps on the left, live terminal and diff in the centre, evaluation checks and token accounting on the right +
(c) Inspect the diff
+
+
+ Settings, Evaluation and metrics: the verdict rule over three gating stages, each with its failure reason, what it records, which plugin provides it and how many options it accepts, above the metrics recorded on every task +
(d) Verify the result
+
+
+

+ RefactorPlatform walks an operator from configuring a run to + verifying its result, live. +

+
+ +
+

Abstract

+

+ An open-source evaluation harness for repository-scale refactoring + agents. It holds the execution environment fixed and varies one design + axis at a time — model backbone, execution regime (baseline, + retrieval-augmented, multi-agent), retrieval chunking strategy, and + prompt specificity — with per-task isolated workspaces, live terminal + streaming, token and diff telemetry, AST-based verification for + Python, and compilation, test and RefactoringMiner gates for Java. +

+
+ +
+

Results

+ +
+
+ 86% + Best pass rate
(qwen3.6-flash, S2 AST retrieval) +
+
+ +25–30pp + AST-aware vs. naive chunking +
+
+ $0.13 + Cost per successful refactor +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ Cross-model pass rate on RefactorBench (100 tasks), descriptive + prompts. +
ModelS1 BaselineS2 + AST Retrieval
qwen3.6-flash73%86%
minimax-m375%81%
kimi-k2.671%78%
deepseek-v4-pro77%89%
+ +

+ A single lean retrieval-augmented agent (86%) also outperforms the one + orchestrated multi-agent topology evaluated (77%, S3, + Copilot sub-agents), and retrieval's accuracy gains absorb its token + overhead: cost per successful refactoring shifts by no more than $0.01 + across all four models. See the paper for the + full ablation, cost-efficiency, and Java (SWE-Refactor) results. +

+ +
+
+

Structural retrieval

+

+ AST-aware chunking sits alone on the purity/integrity frontier + that naive fixed-line windows only approach at much larger + window sizes. +

+ Integrity against purity for each chunking strategy: fixed line windows trace a steep trade-off from 20-line windows at 30 percent purity and 63 percent integrity to 240-line windows at 10 percent purity and 98 percent integrity, while AST chunks sit alone at 93 percent purity and 92 percent integrity +
+
+

Prompt sensitivity

+

+ AST-chunk retrieval leads at every prompt specificity level; + naive line-window retrieval trails even the no-retrieval + baseline. +

+ RefactorBench pass counts by prompt mode: S1 agent 48/64/73, retrieval over line windows 31/44/57, retrieval over AST chunks 56/74/86 +
+
+ +
+
+

Cross-model gains

+

+ AST-aware retrieval improves pass rate for every model family + evaluated, not just the flagship backbone. +

+ RefactorBench descriptive prompt per model: deepseek-v4-pro 77 to 89, qwen3.6-flash 73 to 86, minimax-m3 75 to 81, kimi-k2.6 71 to 78 +
+
+

SWE-Refactor verification

+

+ Java is held to a triple gate: code extraction, structural + detection, and a green build and test suite, applied in + sequence. +

+ SWE-Refactor funnel: 159 of 177 tasks produced code, 105 were detected as the requested refactoring, 50 kept the build and tests green, 43 satisfied all three +
+
+
+ +
+

BibTeX

+ 
@inproceedings{refactorplatform2026,
+  title     = {RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents},
+  author    = {Ben Amor, Aziz and Mali, Drish and Acharya, Mann and Iyer, Vijayasri and Bratières, Sébastien},
+  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
+  publisher = {Association for Computational Linguistics},
+  year      = {2026}
+}
+
+ + +
+ + diff --git a/site/styles.css b/site/styles.css new file mode 100644 index 0000000..398dfc3 --- /dev/null +++ b/site/styles.css @@ -0,0 +1,302 @@ +:root { + --bg: #ffffff; + --fg: #1a1a1a; + --muted: #5b5b5b; + --accent: #2f5fd6; + --border: #e2e2e2; + --code-bg: #f5f5f7; +} + +* { + box-sizing: border-box; +} + +body { + margin: 0; + background: var(--bg); + color: var(--fg); + font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; + line-height: 1.6; +} + +main { + max-width: 760px; + margin: 0 auto; + padding: 3rem 1.5rem 4rem; +} + +.hero { + text-align: center; + margin-bottom: 2.5rem; +} + +h1 { + font-size: 2.6rem; + font-weight: 500; + margin: 0 0 0.5rem; +} + +.subtitle { + font-size: 1.15rem; + color: var(--muted); + max-width: 620px; + margin: 0 auto 1.5rem; +} + +.authors { + list-style: none; + display: flex; + flex-wrap: wrap; + justify-content: center; + gap: 0 1.25rem; + padding: 0; + margin: 0 0 0.25rem; + font-size: 1.05rem; +} + +.authors sup { + font-size: 0.7em; +} + +.authors a { + text-decoration: none; +} + +.authors a:hover { + text-decoration: underline; +} + +.affiliation { + color: var(--muted); + margin: 0 0 1rem; + font-size: 0.95rem; +} + +.venue { + color: var(--muted); + font-size: 1rem; + margin: 0 0 0.75rem; +} + +.footnote { + color: var(--muted); + margin: 0 0 1.5rem; + font-size: 0.8rem; +} + +.links { + display: flex; + justify-content: center; + gap: 0.75rem; + flex-wrap: wrap; +} + +.button { + display: inline-flex; + align-items: center; + gap: 0.4rem; + padding: 0.4rem 1.2rem; + border-radius: 999px; + border: none; + background: #363636; + color: #ffffff; + text-decoration: none; + font-size: 0.95rem; + font-weight: 500; + transition: background 0.15s ease; +} + +.button:hover { + background: #000000; +} + +.button svg { + width: 1em; + height: 1em; +} + +.teaser { + margin: 2.5rem 0; + text-align: center; +} + +figure { + margin: 0; +} + +figure img { + display: block; + width: 100%; + border-radius: 6px; + border: 1px solid var(--border); +} + +figcaption { + margin-top: 0.4rem; + font-size: 0.8rem; + color: var(--muted); + text-align: center; +} + +.panel-grid { + display: grid; + grid-template-columns: repeat(4, 1fr); + gap: 0.6rem; +} + +.teaser-caption { + margin: 1rem 0 0; + font-size: 0.9rem; + font-style: italic; + color: var(--muted); + text-align: center; +} + +.figure-columns { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 2.5rem; + margin-top: 2.5rem; +} + +.figure-col h3 { + font-size: 1rem; + font-weight: 600; + margin: 0 0 0.5rem; +} + +.figure-col p { + font-size: 0.88rem; + color: var(--fg); + margin: 0 0 1rem; +} + +.figure-col img { + display: block; + width: 100%; + border-radius: 6px; + border: 1px solid var(--border); +} + +@media (max-width: 640px) { + .panel-grid { + grid-template-columns: repeat(2, 1fr); + } + + .figure-columns { + grid-template-columns: 1fr; + gap: 1.5rem; + } +} + +h2 { + font-size: 1.3rem; + border-bottom: 1px solid var(--border); + padding-bottom: 0.4rem; + margin: 2.5rem 0 1rem; +} + +.abstract p { + color: var(--fg); + text-align: justify; +} + +.stats { + display: flex; + flex-wrap: wrap; + margin: 0 0 2rem; + border-top: 1px solid var(--border); + border-bottom: 1px solid var(--border); + padding: 1.25rem 0; +} + +.stat { + flex: 1 1 160px; + padding: 0 1.5rem; +} + +.stat + .stat { + border-left: 1px solid var(--border); +} + +.stat-value { + display: block; + font-size: 1.5rem; + font-weight: 700; + color: var(--fg); +} + +.stat-label { + display: block; + margin-top: 0.3rem; + font-size: 0.8rem; + color: var(--muted); + line-height: 1.35; +} + +table { + width: 100%; + border-collapse: collapse; + font-size: 0.92rem; + margin: 0 0 0.5rem; +} + +caption { + caption-side: top; + text-align: left; + font-size: 0.85rem; + color: var(--muted); + margin-bottom: 0.5rem; +} + +th, td { + text-align: left; + padding: 0.5rem 0.75rem; + border-bottom: 1px solid var(--border); +} + +th { + font-weight: 600; + border-bottom: 2px solid var(--border); +} + +td:not(:first-child), th:not(:first-child) { + text-align: right; +} + +.note { + font-size: 0.88rem; + color: var(--muted); +} + +.bibtex pre { + background: var(--code-bg); + border: 1px solid var(--border); + border-radius: 8px; + padding: 1rem; + overflow-x: auto; + font-size: 0.85rem; +} + +footer { + margin-top: 3rem; + border-top: 1px solid var(--border); + padding-top: 1.5rem; + color: var(--muted); + font-size: 0.9rem; + text-align: center; +} + +.pischool-link { + display: inline-block; + margin-bottom: 1rem; +} + +.pischool-logo { + height: 32px; + width: auto; +} + +a { + color: var(--accent); +}