Resources for Urdu Large Language Models

Explore us on: • 🤗 Hugging Face • 🌐 Traversaal.ai • 👔 LinkedIn

---

Resources for Urdu Large Language Models

A comprehensive registry of Urdu datasets, models, and tools designed for the urdu NLP and Large Language Models (LLM) development. Resources are taxonomized by use case, from Pre-training (CLM) and Supervised Fine-tuning (SFT/Instruction Tuning) to Evaluation and Inference. This repository aims to standardize the data landscape for Urdu as a low-to-mid resource language.

---

• Collections
• Datasets
• Evaluation and Leaderboard
• Models
• Tools
• Contributing / Pull Requests
• References

---

Collections

Task Description: Curated platforms and search engines that aggregate, host, and catalog datasets across multiple domains and tasks. These collections are useful for dataset discovery, benchmarking, and exploratory analysis rather than direct model pre-training.

Collection	Description	Notes
OpenML	Open platform for sharing datasets, tasks, and benchmarks for machine learning research	Strong metadata, task-oriented dataset organization
Kaggle Datasets	Large repository of community-contributed datasets across diverse domains	Widely used for competitions and applied ML
Google Dataset Search	Search engine for discovering datasets published across the web	Useful for locating datasets via metadata and schema
Hugging Face Datasets (Urdu)	Filtered view of datasets related to the Urdu language on Hugging Face	Central hub for Urdu NLP datasets with model-ready formats

Datasets

1. Pre-training Corpora

Task Description: Raw, unlabeled text used for Causal Language Modeling (CLM). These datasets are essential for the foundational training phase to teach the model grammar, world knowledge, and vocabulary via next-token prediction.

Dataset	Authors	Date	Notes
Urdu Rekhta	Mahwiz Khalil	2025	Literary Domain including Structured poetry metadata and rhymes.
Rekhta Ghazals	Rekhta Community	2020	High-quality literary text for Domain Adaptation
Maḵẖzan	Zeerak Ahmed	2020	Classical Urdu Data for Pre-training
UFAL Urdu Corpus	UFAL	2014	5.4M sentences with linguistic morphology tags.
Urdu Wikipedia Dumps	Wikimedia	Ongoing	High-factuality encyclopedic text for Knowledge Base Injection
iNLTK Wiki Articles	iNLTK	2020	Cleaned Knowledge Data
PK Multilang	Mahwiz Khalil	2025	Multingual Data
OSCAR Corpus (Urdu)	Ortiz Suárez et al.	2020	High-quality Web CLM
CC-100 (Urdu)	Facebook AI	2019	Massive Web Crawl (CommonCrawl)
WMT Raw	WMT	2017	Web Text useful for robustness training.
Leipzig Corpora	Univ. Leipzig	—	Curated Monolingual Text
Urdu Quotes	Mahwiz Khalil	2023	Aphorisms and proverbs for semantic understanding.
Urdu Wikipedia	Huggingface	2024	Pretraining dataset based on various topics.

2. Instruction Tuning

Task Description: Complex prompt-response pairs and semantic clusters. Used for **Instruction Tuning ** to improve reasoning and instruction following capabilities.

General-purpose

Dataset	Authors	Date	Notes
Urdu Chat Alpaca	Traversaal.ai	2025	Synthetic/Distilled data of52k Alpaca samples translated for general instruction following.
Urdu Instruct	Traversaal.ai	2025	Diverse prompts for broad capability alignment.
GSM8K Urdu	Traversaal.ai	2025	Grade-school math problems for logic/reasoning.
Urdu Assistant	Mahwiz Khalil	2023	Dialogue-focused structure for conversational agents and Supervised Dialogue Training.
Dolly Urdu	Aaqib Saeed	2023	Open-ended QA/Brainstorming data useful for Instruction Tuning.
Urdu Alpaca Filtered	Mahwiz Khalil	2025	Filtered version of Alpaca to remove translation hallucinations.
Urdu Alpaca	Ravi Theja	2024	Standard machine-translated Alpaca dataset.
Urdu Legal	Mahwiz Khalil	2024	Legal QA pairs for specialized fine-tuning.
UQuAD	Ahsan Farooqi	-	SQuAD-style (Context → Question → Answer). Ideal for RAG.
QA	Muhammad Irfan	-	Simple fact-retrieval pairs.
UQA	Samee Arif	2024	Open-domain question answering.
Urdu Q-A	Nabeel Danish	2024	Dataset built for retrieval systems.
Wiki- UQA	Samee Arif	2024	Wikipedia-grounded QA pairs for RAG/Context.

Sequence Classification & Alignment

Task Description: Labeled data for Sequence Classification. These datasets are critical for training Reward Models (RM) in RLHF pipelines or building specialized classifiers for sentiment and safety (guardrails).

Dataset	Authors	Date	Notes
Urdu IMDb Reviews	Kaggle Contributors	—	Sentiment Analysis / RLHF
Urdu Sentiment Benchmark	M. Yaseen Khan	2020	Binary Classification
Roman Urdu Dataset	Smat26	—	Roman-script Sentiment
Urdu Financial Phrasebank	Mahwiz Khalil	2023	Sentiment Classification (Financial Domain
UCI Roman-Urdu Sentiment	UCI ML Repo	2018	Noisy Text Classification
Hate Speech Detection	Haroon Shakeel	2020	Toxicity Detection / Safety
Awesome Urdu	Ikram Ali	2019	Differnet Urdu NLP and NLU Datasets

Token Classification (POS, NER)

Task Description: Text with token-level annotations. Used for Token Classification tasks to extract structured entities (NER) or understand syntactic roles (POS), improving the model's information extraction capabilities.

Dataset	Authors	Date	Notes
Urdu NLP Collection	Mirfan	—	General Token Classification
Urdu UD Treebank	Universal Dependencies	—	Syntactic Parsing / Dependency
MK-PUCIT NER	PUCIT	2019	Named Entity Recognition
WikiAnn (Urdu)	Pan et al.	2017	Cross-lingual NER Transfer
Urdu Word Segmentation	Haris Bin Zia	2018	Tokenizer Optimization / CRF

Semantic Tasks (Summarization, Data Augmentation)

Task Description: Paired or unpaired text data used for semantic generation tasks. The model learns to map source text to a semantically faithful target text (summaries or augmented variants), focusing on meaning preservation, abstraction, and contextual coherence rather than token-level labeling.

Dataset	Authors	Date	Notes
UrduSummary Corpus	Humsha et al.	2016	Abstractive Summarization
Text Generation	Mahwiz Khalil	2025	Text Generation
Urdu Paraphrase Corpus	UCREL	2016	Paraphrase Detection
Urdu Short Text Reuse	UCREL	2018	Semantic Similarity (STS)
TaPaCo	Scherrer et al.	2020	Data Augmentation / Paraphrasing
ursum	Mahwiz Khalil	2023	Urdu SUmmarization

Multimodal (Vision–Language)

Task Description: Image-Text pairs used for Visual Language Modeling (VLM). Enables the model to perform image captioning, visual reasoning, and multimodal alignment.

Dataset	Authors	Date	Notes
Flickr8k Urdu Captions	Abdullah Zia	2020	Image-to-Text Generation
CLE Urdu Image Corpora	CLE Pakistan	—	OCR / Visual Grounding

OCR & Text Recognition

Task Description: Images of text paired with transcriptions. Used to train Optical Character Recognition (OCR) encoders or to fine-tune multimodal models on the Nastaliq script.

Dataset	Authors	Date	Notes
Qaida	Atique ur Rehman	—	Synthetic Data Generation
U-HAT	Hazrat	—	Handwritten Text Recognition
45K Urdu Ligatures	UltramindSoft	2019	Ligature Recognition
IIIT-H Urdu OCR	IIIT Hyderabad	2017	Printed Text OCR
Cursive-Text	Scene Text Authors	2020	Natural Scene Text OCR
urdu-ocr-1M	Puristan Labs	2026	Urdu OCR

Machine Translation (Seq2Seq)

Task Description: Parallel bitexts (English-Urdu). Used for Sequence-to-Sequence (Seq2Seq) training to align the Urdu embedding space with high-resource languages like English.

Dataset	Authors	Date	Notes
OPUS Corpora	OPUS Team	—	Massive Parallel Corpus
PM India Corpus	WMT	—	Low-resource Alignment
English–Urdu Religious	UFAL	—	Domain Specific Translation
Anuvaad Parallel Corpus	Govt. of India	—	Legal/Admin Documentation
CLEU Corpus	UCREL	2018	Translation & Reuse Detection

Transliteration & Normalization

Task Description: Character-level mappings between scripts. Used for Text Normalization pipelines to handle Romanized Urdu input or convert scripts before tokenization.

Dataset	Authors	Date	Notes
Google Dakshina	Google Research	2020	Script Transliteration
TRANSLIT	Benites et al.	2020	Named Entity Transliteration
Roman-Urdu Corpus	Irshad Bhat	2020	Roman-to-Nastaliq Mapping
Ur–RomUr–Eng Dict	Moiz Rauf	2019	Multi-script Dictionary

Lexical & Linguistic Resources

Task Description: Dictionaries and N-grams. Used for Tokenizer Training (BPE/WordPiece), vocabulary expansion, and heuristic data filtering/cleaning.

Dataset	Authors	Date	Notes
Urdu Sentiment Lexicon	—	—	Polarity Lookup / Heuristics
UrduHack Word Lists	UrduHack	—	Stopwords / Normalization
CLE Urdu WordNet	CLE Pakistan	—	Semantic Graph / Synsets
IndoWordNet	IIT Bombay	—	Multilingual WordNet
Urdu N-grams	Kaggle	2020	Statistical Language Modeling

Speech & Audio (ASR)

Task Description: Audio paired with text transcriptions. Used for training Automatic Speech Recognition (ASR) models or fine-tuning Whisper-style models for Urdu.

Dataset	Authors	Date	Notes
UAT	Mahwiz Khalil	2025	Urdu Audio and Text
Urdu 250 Isolated Words	Hazrat	2018	Keyword Spotting (KWS)
CLE Phonetic Speech	CLE Pakistan	—	Phonetic Alignment / ASR
CMU Wilderness	CMU	2019	Aligned Speech (Bible)
LibriVox Urdu	Community	—	Audiobooks (Long-form ASR)
Urdu-Sindhi Emotion	Researchers	2020	Speech Emotion Recognition

---

Benchmarks

Task Description: Standardized test sets for Zero-shot and Few-shot Evaluation. These datasets must be excluded from training data to prevent contamination and ensure valid performance metrics.

Dataset	Authors	Date	Notes
Urdu Instruct Test	Traversaal.ai	2025	Diverse prompts for broad capability alignment.
OpenbookQA Urdu	Traversaal.ai	2025	OpenBookQA Benchmark Translation
Math500 Urdu	Traversaal.ai	2025	Math500 Benchmark Translation
CommonSenseQA Urdu	Traversaal.ai	2025	CommonsenseQA Benchmark Translation
MGSM Urdu	Traversaal.ai	2025	MGSM Benchmark Translation
Urdu Error Correction	Mahwiz Khalil	2025	Error Detection & Correction

---

Models

Use Case: Ready-to-use weights (Adapters/LLMs) to use as baselines or functional tools.

Collection / Model	Maintainer	Notes
Alif-1.0-8B-Instruct	Traversaal.ai	8B parameter instruction-tuned model (Llama-3.1 based).
Qalb-1.0-8B-Instruct	Muhammad Taimoor et. al	8B parameter instruction-tuned model (Llama-3.1 based)
Mahwizzzz Models	Mahwizzzz	Collection of fine-tuned models for specific Urdu tasks.

---

Tools

Essential libraries, models, and resources for building Urdu LLM pipelines.

Python Libraries

Use Case: NLP, Data cleaning, normalization, and linguistic analysis.

Tool	Type	Notes
UrduHack	Library	NLP including Normalization, Tokenization, NER etc.
UrduHack-Lite	Library	Lightweight version for low-memory environments.
Stanza	Pipeline	High-accuracy dependency parsing and POS tagging.
spaCy	Pipeline	Industrial NLP pipeline with community Urdu support.
iNLTK	Toolkit	Data Augmentation and embeddings.
LughaatNLP	Library	Lexical processing and stemming.
UNLT	Toolkit	UCREL normalization and stemming utilities.

APIs & Online Services

Use Case: Benchmarking and OCR/TTS without local deployment.

Service	Provider	Notes
Text Processing	CLE Pakistan	APIs for Text Processing.
Speech Processing	CLE Pakistan	APIs for speech Processing.
Image Processing	CLE Pakistan	APIs for Image Processing.
Google Translate	Google	Translator (Other Language ↔ Urdu).
Matnsaz	Tool	Autocorrecter: Writing assistant for text validation.

Input & Typography

Use Case: Solving Nastaliq rendering issues and input testing.

Resource	Category	Notes
Urdu Fonts	Fonts	Testing model output rendering (Nastaliq vs Naskh).
Branah Keyboard	Input	Virtual keyboard for prompt testing.
Lexilogos	Input	Phonetic keyboard layout.

---

Pull Requests

We welcome pull requests (PRs) that add or improve Urdu datasets, models, and tools relevant to Urdu NLP and Large Language Model (LLM) development.

You can contribute by:

• Adding new datasets (pre-training, SFT/instruction tuning, evaluation)
• Adding LLM or embedding models supporting Urdu
• Adding tools, scripts, or benchmarks for training, evaluation, or inference
• Fixing errors, improving documentation, or standardizing metadata

How to Submit a PR

1. Fork the repository
2. Create a new branch (feature/add-urdu-dataset)
3. Add your resource under the appropriate section
4. Include:
   • Short description
   • License
   • Data source / paper / link
   • Intended use (Pretraining / SFT / Eval / Inference)
5. Open a pull request with a clear summary

All contributions help strengthen the Urdu LLM ecosystem.

---

References

Please let us know if a dataset is not properly credited.

• Rekhta Community, "Rekhta Ghazals: High-Quality Literary Text for Domain Adaptation," 2020.
• Mahwiz Khalil, "Urdu Rekhta: Literary Domain including Structured Poetry Metadata and Rhymes," 2025.
• Mahwiz Khalil, "Urdu Assistant: Dialogue-Focused Dataset for Instruction Tuning," 2023.
• Samee Arif, "UQA: Open-Domain Question Answering," 2024.
• Muhammad Ali Shafique et al, "Alif: Advancing Urdu Large Language Models via Multilingual Synthetic Data Distillation," 2025.
• {Muhammad Taimoor Hassan and Jawad Ahmed and Muhammad Awais, "Qalb: Largest State-of-the-Art Urdu Large Language Model for 230M Speakers with Systematic Continued Pre-training".2026