MultiIdiom

Data-construction pipeline for MultiIdiom: A Multilingual Idiom Detection Dataset with In-Context Idiomaticity Labels and Span Annotations.

The corpus, splits, and sociolinguistic metadata are released on the HuggingFace Hub under CC-BY-SA 4.0: Justarandomperson/MultIdiom_Dataset. This repo is the code that built it — scraping, LLM-based labeling/standardization, and lookup/inference over the resulting idiom inventory.

Setup

git clone https://github.com/JustLetMeBeHello/MultiIdiom.git
cd MultiIdiom
pip install -r requirements.txt

Pipeline stages

MultiIdiom/
├── notebooks/
│   └── Idioms_scraper.ipynb        ← idiom-list scraping (Wiktionary) and
│                                       idiomatic-example scraping (OPUS subtitles)
├── src/
│   ├── Seed_idiom_verifier.py      ← validates scraped idiom lists before labeling
│   ├── Wikipedia_Sentence_extractor.py
│   ├── inference.py                ← IdiomMatcher: idiom lookup over a language's
│   │                                   seed idiom file (used by Main.py)
│   ├── LLM_Prompting/              ← GPT-based labeling: idiom meanings
│   │   ├── Meaning_Prompting/         (Meaning_Prompting/), example/idiomaticity
│   │   ├── Example_prompting/         labeling (Example_prompting/), span
│   │   ├── Tagging/                   tagging + retry passes (Tagging/)
│   │   └── Normalization/
│   ├── Standardizer/                ← post-labeling cleanup: id assignment,
│   │                                   gap-filling, duplicate/discrepancy checks
│   └── Database/                   ← Supabase upload scripts for the labeled corpus
└── Main.py                          ← minimal usage example (IdiomMatcher lookup)

Reproducing the corpus

1. Scrape idiom lists + idiomatic example sentences: notebooks/Idioms_scraper.ipynb.
2. Verify seed idiom lists: src/Seed_idiom_verifier.py.
3. Label with an LLM (meanings, idiomaticity, spans): scripts under src/LLM_Prompting/, run per-language/per-batch (Meaning_Prompting/ → Example_prompting/ → Tagging/).
4. Standardize: src/Standardizer/Standardizer_file.py (id assignment, gap-filling, duplicate/discrepancy detection).
5. Upload: src/Database/upload_meanings.py, upload_span_examples.py.

The released HuggingFace dataset is the output of this pipeline; the raw scrape/label intermediates are not re-released (LLM API keys required to re-run labeling from scratch).

License

Code: MIT. The MultiIdiom dataset has its own license (CC-BY-SA 4.0), released separately on HuggingFace Hub.