Reproduction of the WebFAQ paper (Dinzinger et al., SIGIR '25 — "WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval") as the final project for LSDIPro, SoSe 2026, TU Berlin (D2IP).
Team: Arian Kabiri, Louis Mundzia, Kian Nuri, Bertan Caglar
Supervisor: Guido Rocchietti (TU Berlin)
We reimplement the paper's filtering pipeline from scratch on our own Common Crawl (WDC) snapshots, label a subset with a local LLM instead of a paid API, fine-tune a multilingual retriever on the result, and evaluate it against BM25 and zero-shot baselines across 5 languages (EN, DE, FR, TR, PL). See WebFAQ_Abschlusspräsentation_FINAL.pdf for the full walkthrough with numbers and charts — this README mirrors its structure.
Every stage of the pipeline lives in its own numbered top-level folder, in the order data actually flows through them. Scripts inside each folder are numbered too where there's a clear run order.
webfaq1/ ├── 01_extraction/ Parse raw WDC FAQPage data into per-language JSONL ├── 02_filtering/ Filter 1–3: clean → exact-dedup → near-dup → QA consistency ├── 03_embeddings/ Convert/merge embeddings into a multilingual FAISS index ├── 04_labeling/ Topic-label a subset with Gemma 2, then distill into a fast classifier ├── 05_finetuning/ Build (query, pos, neg) triplets, fine-tune XLM-RoBERTa ├── 06_evaluation/ NDCG@10 evaluation: BM25 vs. Base vs. FT vs. mE5 vs. Jina ├── 07_multilingual/ Cross-lingual question matching: find semantically equivalent questions across languages ├── 08_demo/ Streamlit demo apps (live + offline/mock) + demo index builder ├── utils/ One-off inspection helpers ├── legacy/ Archived/superseded code — kept for history, not part of the active pipeline (see legacy/README.md) └── context.md Team/server notes (venv, GPU rules, known issues)
data/, models/, .venv/ and tmp/ are not part of this repo — see "Data & models" below.
cd /home/webfaq1/Datasets/webfaq1
python3.12 -m venv .venv # already exists on the server at .venv/
source .venv/bin/activate
pip install -r requirements.txtServer: lyra.d2ip.tu-berlin.de, user webfaq1. Server-specific rules (see context.md for the full list): book a time slot before using GPUs, monitor htop/nvidia-smi, name your screen/tmux sessions, numpy < 2, set TMPDIR=/home/webfaq1/Datasets/webfaq1/tmp per session.
Parses schema.org/FAQPage annotations out of raw WDC N-Quads snapshots (data/00_raw_wdc_data/*.gz) and writes one JSONL file per detected language to data/01_extracted/.
python 01_extraction/01_extract_qa_pairs.py --input data/00_raw_wdc_data/ --output data/01_extracted/The three filters from paper §3.2–3.3, run in order. run_filters.sh chains all four steps for one language file; resume_filters.sh restarts from step 3 (useful after a crash mid-embedding); process_all_languages.py queues all languages across the available GPUs.
| Step | Script | What it does |
|---|---|---|
| 1 | 01_clean_text.py |
Strips whitespace/boilerplate/emoji/HTML entities, NFC-normalizes, drops malformed samples |
| 2 | 02_exact_dedup.py |
Filter 1: identical questions with the same answer → keep one; same question with conflicting answers → drop all (paper §3.2) |
| 3 | 03_near_duplicates.py |
Filter 2: embeds questions with Jina v3, drops near-duplicates within the same site origin above cosine similarity α = 0.70 |
| 4 | 04_qa_consistency.py |
Filter 3: drops QA pairs below question–answer cosine similarity β = 0.50 |
cd 02_filtering
./run_filters.sh ../data/01_extracted/de.jsonl --batch_size 128Output lands in data/02_filtered/{01_clean,02_exact_dedup,03_near_dedup,04_consistency}/.
convert_all_npy_to_json.py converts the .npy question embeddings (data/03_embeddings/) to JSON for inspection/tooling; merge_multilingual.py merges the per-language filtered corpora + embeddings into one combined multilingual FAISS index (data/04_multilingual/), covering EN, DE, FR, TR, PL (1.4M pairs total).
Paper §goal: attach a topic label to every QA pair, both for a richer dataset and as training signal for fine-tuning. The paper uses GPT-4o-mini; we use local Gemma 2 (9B) via Ollama — free, reproducible, no API key required.
| Step | Script | What it does |
|---|---|---|
| 1 | 01_gemma_label_sample.py |
Small dry-run: labels a ~500-sample batch with Gemma 2, 8-topic taxonomy (paper schema) |
| 2 | 02_gemma_label_all.py |
Full run: ~20k samples per language (EN/DE/FR/TR/PL), resumable via checkpoints → data/05_labeled/ |
| 3 | 03_train_topic_classifier.py |
Distills the 20k Gemma-labeled English pairs into a fast XLM-RoBERTa topic classifier |
| 4 | 04_label_full_corpus.py |
Applies that classifier to the full English corpus (~1.24M pairs, ~10 min, no LLM calls needed) |
The 8 topics follow the paper's taxonomy: Traveling & Hospitality, Products & Commercial Services, Healthcare Wellness & Lifestyle, Entertainment Recreation & Leisure, Employment Education & Training, Banking Financial Services & Insurance, Legal Services Regulations & Government, General Information & Other.
| Step | Script | What it does |
|---|---|---|
| 1 | 01_prepare_training_data.py |
Builds (question, positive answer, negative answer) triplets from the labeled data — negatives sampled from other topics — 80/10/10 split → data/06_finetuning/ |
| 2 | 02_finetune_xlm_roberta.py |
Fine-tunes PaDaS-Lab/xlm-roberta-base-msmarco (already MS MARCO-warmed-up, skipping the paper's 30-epoch English pre-training) with Multiple Negatives Ranking Loss → models/xlmr_msmarco_webfaq_ft/ |
| Script | Scope |
|---|---|
01_evaluate_table3_full.py |
Full 5-model NDCG@10 comparison per language: BM25 → Base (xlm-roberta-base-msmarco) → FT (ours) → mE5 (multilingual-e5-large-instruct) → Jina (jina-embeddings-v3) |
02_evaluate_en_paper_compare.py |
English, full corpus, numbers directly comparable to the paper's Table 3 |
Results on our 20k labeled test split (held-out 10%, same corpus used for fine-tuning — NDCG@10):
⚠️ These numbers are evaluated on our own small labeled test split (~2k pairs per language), not the paper's full held-out corpus. Absolute values are therefore higher than the paper's Table 3. The rankingBM25 < Base < FT < mE5 < Jinais what matters — it is stable across all 5 languages and matches the paper.
| System | DE | EN | FR | PL | TR |
|---|---|---|---|---|---|
| BM25 | 78.4 | 87.2 | 84.5 | 77.8 | 86.5 |
| Base | 91.9 | 93.4 | 91.6 | 91.2 | 90.5 |
| FT (ours) | 96.7 | 97.6 | 97.4 | 97.0 | 97.0 |
| mE5 | 97.6 | 98.4 | 97.8 | 95.7 | 96.9 |
| Jina | 98.4 | 98.9 | 98.7 | 98.3 | 98.5 |
Results on the full English corpus (1.24M pairs, closer to the paper's setup — see presentation for chart):
| System | EN (full corpus) | Paper Table 3 EN |
|---|---|---|
| BM25 | 19.6% | 24.4% |
| FT (ours) | ~55–60% | 59.0% |
| mE5 | ~66% | 66.2% |
| Jina | ~66% | 66.4% |
Gap to paper on BM25 is expected — we evaluate on unfiltered data; the paper uses their fully filtered corpus which we could not reproduce at full scale (no labeled data released by the authors).
Given the filtered corpora, find semantically equivalent questions across languages using embedding-based matching. This extends the paper's multilingual scope by explicitly identifying cross-lingual question pairs, which can serve as additional training signal or coverage analysis.
multilingual_matching.py— core matcher (multi-GPU embedding-based, with an n-gram fallback)prepare_bilingual.py— reads the final filtered data, matches questions across languages, produces per-language record-count summary and bilingual pairs (run/)01_crosslingual_match.py— runs the matcher across all filtered languages,--top-k/--threshold/--languagesconfigurable, output →run/crosslingual_matches.jsonl02_visualize_matches.py— charts + an optional PowerPoint slide from a matches filetests/— unit tests for all three core modules
cd 07_multilingual
python prepare_bilingual.py
python 01_crosslingual_match.py --top-k 5 --threshold 0.5 --languages en,de,frPer-language matched-question counts: EN ~249k, FR ~33k, TR ~5.8k, PL ~5.2k, DE ~3.3k.
01_build_demo_index.py— builds small (10k/lang) FAISS demo indices fromdata/03_embeddings+data/02_filtered/04_consistency→demo_index/02_streamlit_app.py— the full live demo: BM25 + Zero-Shot (Jina v3 + FAISS) search over the real per-language data, entirely configured via env vars (DATA_DIR,INDEX_DIR,EMBEDDING_MODEL,DEVICE,TOP_K)03_app_mock.py— lightweight offline demo (crash-safe, usesdemo_index/JSONL files); currently running as the live demo on the server at port 8501
cd 08_demo
streamlit run 02_streamlit_app.py # full live demo (requires data/ on server)
# or
streamlit run 03_app_mock.py # offline/mock demo (works with demo_index/ only)The demo_index/ JSONL files (~23MB total, 10k pairs per language) are included in this repo so the mock demo runs without any data setup. They can be regenerated with 01_build_demo_index.py.
Everything superseded, exploratory, or dead is kept (not deleted) under legacy/, out of the active pipeline path. See legacy/README.md for what's there and why.
data/ (~257GB) and models/ (~15GB) are not part of this repo — too large to version, and not needed to review the code. Everything under 01_extraction/ … 08_demo/ references them via absolute paths (/home/webfaq1/Datasets/webfaq1/data/...), so cloning this repo elsewhere means re-pointing those paths or re-running the pipeline against your own data. The small demo artifacts (08_demo/demo_index/) are the only data included in the repo and can also be regenerated with 01_build_demo_index.py.
On the team server specifically, this repo's code lives at /home/webfaq1/Datasets/webfaq1/for_github/, with data/, models/, .venv/ and tmp/ kept alongside it in a sibling not_for_github/ folder. Symlinks at the old top-level paths keep every absolute path above resolving exactly as before.
WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval — Dinzinger, Caspari, Ghosh Dastidar, Mitrović, Granitzer. SIGIR '25, July 13–18 2025, Padua, Italy. DOI: 10.1145/3726302.3731934 · GitHub · HuggingFace
Paper dataset: 96M raw QA pairs, 75 languages, filtered to 11.2M pairs (α=0.70 near-dup, β=0.50 QA-consistency, same thresholds as ours). We reproduce the pipeline on 5 languages (EN, DE, FR, TR, PL) rather than the paper's 20, since the paper's filtered/labeled data was never released publicly.
@inproceedings{webfaq2025,
author = {Dinzinger, Michael and Caspari, Laura and Ghosh Dastidar, Kanishka and Mitrović, Jelena and Granitzer, Michael},
title = {WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval},
booktitle = {Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {2025},
month = {July},
location = {Padua, Italy},
pages = {3802--3811},
doi = {10.1145/3726302.3731934},
publisher = {ACM}
}