CleanSlide

A Leakage-Audited and Shortcut-Controlled Benchmark for Whole-Slide Vision–Language Models
_{The largest public and cleanest whole-slide-image (WSI) multiple-choice benchmark on TCGA.}

Paper: coming soon  ·  Data: data/  ·  Features: 🤗 Hugging Face  ·  Method: docs/QUESTION_GENERATION.md

CleanSlide is a whole-slide-image vision–language MCQ benchmark built entirely on TCGA diagnostic (FFPE / DX) slides. It pairs pan-cancer scale with two kinds of cleanliness that competing benchmarks skip: a patient- and tissue-source-site (TSS)-disjoint image split, and a four-dimension question audit whose blind (image-free) baseline is measured and published rather than hidden. Slide features are released with an openly-licensed encoder (H-optimus-0, Apache-2.0) that was not trained on TCGA, so they are redistributable and uncontaminated by the encoder.

At a glance

• 148,654 clean multiple-choice questions over 9,985 diagnostic WSIs, spanning 32 solid cancer types (pan-cancer TCGA FFPE / DX slides).
• Train / val / test are patient- and TSS-disjoint — 0 patient overlap, 0 site overlap.
• Every question is audited on four cleanliness dimensions, and the blind (image-free) baseline is measured and published, not hidden.
• Encoder-agnostic, coordinate-first features from H-optimus-0 (Apache-2.0, not TCGA-trained).
• License: CC-BY-NC 4.0.

CleanSlide is built to answer the three questions a reviewer asks of any pathology VLM benchmark:

(1) How are the image features extracted? (2) How is the image set kept clean? (3) How are the questions kept clean?

Each is answered below. Question-generation details live in docs/QUESTION_GENERATION.md.

---

Scale, measured honestly

• Largest by whole-slide images among public benchmarks: 9,985 WSIs (next: WSI-Bench 9,850, SlideChat/SlideBench 4,181). We report scale in WSIs — the honest unit — because raw question counts inflate freely under templating and duplication.
• Not the largest by raw question count, and that is fine: WSI-Bench (~180k) and SlideBench (~176k) have more questions, and the unreleased TCP-LLaVA claims 186k. Sheer question volume is not our selling point. Our selling point is clean + independently-auditable + whole-slide + pan-cancer.
• Largest public, independently-auditable, four-dimension-clean WSI set. (Sets that claim cleanliness but are not released — TCP-LLaVA, CTIS-QA, Scale-VQA, PathMMU-HR² — cannot be audited by anyone.)

The cross-benchmark comparison above (figures/fig_crossbench.png) plots scale against the four cleanliness dimensions over all comparable benchmarks; see also figures/fig_scale_clean.png.

---

1. How are the features extracted?

A standardized, coordinate-first pipeline (code/extract/extract_features.py) using H-optimus-0. Parameters follow the consensus of CLAM, Prov-GigaPath, UNI, Virchow, and the H-optimus-0 model card.

Step	Setting	Why
Encoder	H-optimus-0 (hf-hub:bioptimus/H-optimus-0), frozen, fp16, 1536-d	Apache-2.0, not TCGA-trained → features are redistributable and encoder-uncontaminated
Magnification	20× = 0.5 µm/px; each slide read at the level matching its own MPP	standardize by physical scale, not pixels (scanners differ: 40×=0.25, 20×=0.5)
Patch	224 × 224, non-overlapping	H-optimus-0 native input
Tissue segmentation	RGB→HSV, Otsu on saturation → morphological close + hole-fill + remove-small-objects	CLAM standard
Keep patch if	tissue ≥ 10 % of patch area	Prov-GigaPath convention
Normalization	mean [0.7072, 0.5787, 0.7036], std [0.2119, 0.2301, 0.1775], bicubic	H-optimus-0 published config
Stain normalization	none	modern SSL FMs run un-normalized; stain-norm does not remove the site signature (Howard 2021) — we keep the honest raw distribution and control site via the split
MPP missing	slide skipped and flagged (skip-no-mpp)	do not guess physical scale

Output — one <barcode>.h5 per slide:

• features [N, 1536] fp16 — H-optimus-0 patch embeddings.
• coords [N, 2] int32 — level-0 patch coordinates = the encoder-agnostic canonical layer: any future encoder is re-run from these coordinates without re-segmenting or re-tiling.

Coverage: 9,985 / 10,302 benchmark slides = 96.9 %. The 317 uncovered slides are openslide-unreadable (unsupported format / missing resolution metadata); every released question is on a slide that has features.

Robustness: resumable (skips existing .h5) with an atomic per-slide lock (safe for concurrent GPUs / re-runs); each job pinned to ≤ 8 CPU cores (taskset, incl. DataLoader workers) so it never hogs a shared node; launched with setsid so it survives SSH disconnect / session teardown (code/run_all_detached.sh).

---

2. How is the image set kept clean? — two disjointness dimensions

A TCGA WSI leaks information two ways that a naive split ignores. CleanSlide controls both:

1. Patient-disjoint. No patient appears in more than one of train / val / test. (A patient can have several slides; putting them on both sides leaks the answer.)
2. Site (TSS)-disjoint. No tissue source site (the institution, encoded as the 2nd field of the barcode, TCGA-A6-2670-…) appears in more than one split. Howard et al. (Nat. Med. 2021) showed TCGA carries a strong site signature (staining / scanner batch) that models exploit as a shortcut; controlling it is essential and almost universally skipped. Because a patient's barcode contains its TSS, TSS-disjoint implies patient-disjoint.

Verified: 0 patient overlap and 0 TSS overlap across train / val / test. Among all surveyed benchmarks, CleanSlide is the only one that controls the site axis — see figures/fig_disjoint.png and docs/split_survey.md.

---

3. How are the questions kept clean? — four dimensions, two tiers

Full method: docs/QUESTION_GENERATION.md. In brief:

Four cleanliness dimensions, measured on the whole released set (lower = cleaner; ideal ①④→25 = chance, ②③→0):

① option-shortcut	② stem-leak	③ duplication	④ blind-solvable
25 %	8.5 %	0.00 %	~41 %

• ① Option-shortcut — the answer must not be guessable from option shape (position or length). We rebalance answer letters to uniform and equalize option length.
• ② Stem-leak — the stem must not state or hint the answer.
• ③ Duplication — no (stem+options+answer) signature crosses a split boundary.
• ④ Blind-solvable — a strong text-only LLM (Qwen3-32B) must not answer above chance without the image. Residual blind-solvability that is genuine medical knowledge (not an artifact) is kept and reported via vision-gain, not deleted.

Two tiers of questions:

• Tier 1 — reuse & clean: import existing public TCGA-WSI MCQ (WSI-Bench, SlideChat/SlideBench, WSI-VQA) and run every one through the four-dimension pipeline → 44,872.
• Tier 2 — self-generated from TCGA ground truth we control, text-only (a report/label is the answer key; an image is never shown to a VLM to author a question): (2a) templated from GDC structured labels → 3,218; (2b) mined from free-text pathology reports (Qwen3-32B) → 100,564.

---

Dataset

The benchmark lives in data/: three MCQ splits plus a slide → fold table.

File	Rows	Contents
train.jsonl	91,894	training questions
val.jsonl	27,837	validation questions
test.jsonl	28,923	test questions
slides_splits.csv	9,985	slide_id, patient, tss, tumor, fold, split

Train / val / test are patient- and TSS-disjoint. Each line of a .jsonl file is one 4-option question:

{
  "source": "SlideBench",
  "slide": "TCGA-08-0244-01Z-00-DX1",
  "patient": "TCGA-08-0244",
  "tss": "08",
  "tumor": "GBM",
  "question": "Examine the cellular morphology in the provided whole slide image of glioblastoma. Which cytoplasmic feature is typically observed?",
  "options": ["Mucin production", "Clear cytoplasm", "Fibrillary background", "Granulomatous changes"],
  "answer": "C",
  "answer_type": "mcq",
  "task": "Microscopy",
  "fold": 3,
  "split": "test"
}

Field	Meaning
source	origin set — WSI-Bench, SlideBench, WSI-VQA, CleanSlide-labels, or CleanSlide-reports
slide / patient / tss	TCGA slide barcode, patient, and tissue-source-site
tumor	TCGA study (cancer-type) code, e.g. GBM
question / options / answer	stem, four options, correct letter (A–D)
task	question category (Diagnosis, Microscopy, Subtype, Grading, …)
fold / split	disjoint fold id and train / val / test

Load the benchmark:

import json

def load_jsonl(path):
    with open(path) as f:
        return [json.loads(line) for line in f]

test = load_jsonl("data/test.jsonl")
q = test[0]
print(q["question"])
print(q["options"], "->", q["answer"])

Get the images. Raw slides are not redistributed here. For any slide barcode, download the TCGA diagnostic (DX) whole-slide image from the GDC Data Portal, or use our pre-computed per-slide features on 🤗 Hugging Face — one <slide_id>.h5 per slide (H-optimus-0 embeddings + level-0 coordinates; see §1). Slide → fold membership is in data/slides_splits.csv.

---

Repository layout

.
├── README.md                          ← you are here (answers the 3 questions)
├── LICENSE                            CC-BY-NC 4.0
├── requirements.txt                   Python env (torch / timm / transformers / openslide / skimage …)
├── data/                              the benchmark itself
│   ├── train.jsonl                    91,894 questions
│   ├── val.jsonl                      27,837 questions
│   ├── test.jsonl                     28,923 questions      (train / val / test: patient + TSS disjoint)
│   └── slides_splits.csv              9,985 slides → slide_id, patient, tss, tumor, fold, split
├── code/
│   ├── extract/extract_features.py    H-optimus-0 feature extraction
│   ├── questions/                     Tier-1 cleaning + Tier-2 generation (labels / reports) + merge + finalize + 4-dim audit
│   ├── splits/build_triple_clean_splits.py   patient + TSS-disjoint fold builder
│   ├── acquire/pull_gdc_clinical.py   pull GDC structured clinical labels
│   └── run_all_detached.sh            SSH-safe launcher (setsid) for the GPU jobs
├── docs/
│   ├── QUESTION_GENERATION.md         authoritative question-build method (2 tiers, 4 dims)
│   ├── split_survey.md                how 14+ benchmarks split (grounds our patient + TSS design)
│   └── cleaning_results.md            running cleaning-results log
└── figures/                           cross-benchmark + disjointness + scale-vs-clean figures + datasets.py

Large files stay off GitHub: the raw TCGA WSIs and the per-slide <barcode>.h5 features (~TB scale) are not in this repo. Download slides from TCGA / GDC by slide_id, or use our released <slide_id>.h5 features; then reproduce the questions with the pipeline below.

---

Reproduce

python code/extract/extract_features.py --slide-list <svs_paths.txt> --out features/   # H-optimus-0 features
python code/questions/clean_merge_reports.py                                            # clean + merge questions
python code/questions/finalize_release.py                                               # drop feature-less → release set
python figures/plot_crossbench.py && python figures/plot_disjoint.py && python figures/plot_scale_clean.py

---

Citation

If you use CleanSlide, please cite:

@misc{zhang2026cleanslide,
  title        = {CleanSlide: A Leakage-Audited and Shortcut-Controlled Benchmark for Whole-Slide Vision–Language Models},
  author       = {Zhang, Wenhao and others},
  year         = {2026},
  note         = {Manuscript in preparation},
  howpublished = {\url{https://github.com/wenhaozhang0066/CleanSlide}}
}

License

Questions and derived features: CC-BY-NC 4.0. Underlying TCGA slides follow the NIH/GDC data policy. Encoder: H-optimus-0 (Apache-2.0).