Language Identification Classifier

📋 Project Description

A character-based machine learning classifier that automatically identifies which of six languages a text is written in: Hausa, Indonesian, Manobo, Nahuatl, Swahili, and Tagalog. Since all six languages use the Latin script with minimal diacritics, making visual distinction difficult, the system leverages character frequency patterns and phonotactic structures (how sounds combine) using n-gram features and Multinomial Logistic Regression (Softmax Regression) to achieve accurate language detection.

---

🎯 The Challenge

Problem: Distinguish between six languages that look very similar:

The Six Languages:

1. Hausa - West African language
2. Indonesian - Southeast Asian language
3. Manobo - Philippine indigenous language
4. Nahuatl - Mexican indigenous language
5. Swahili - East African language
6. Tagalog - Philippine national language

Why It's Hard:

• All use the same Latin alphabet
• Minimal use of special characters (diacritics)
• Without linguistic training, humans can't easily tell them apart
• Can't rely on unique characters for each language

Why Traditional Approaches Don't Work:

• Word-based features: Too many possible words = sparse, overwhelming feature space
• Simple pattern matching: No unique character sets to distinguish languages
• Binary classification: This is a 6-way (multinomial) classification problem

---

💡 Solution Approach

Why Character-Based Features?

Key Insight: Languages have unique character frequency patterns and sound combination rules (phonotactics).

Two Feature Types:

1. Character Unigrams (Single Characters)

   • Each language uses letters with different frequencies
   • Example: 'q' common in some languages, rare in others
   • Captures overall character distribution

2. Character N-grams (Character Sequences)

   • Bigrams: 2-character sequences ("th", "ng", "ua")
   • Trigrams: 3-character sequences ("ing", "tion")
   • Captures phonotactic patterns (how sounds combine)
   • More powerful than unigrams alone

Tradeoff: As n increases:

• ✅ Better captures language patterns
• ❌ Feature space explodes (sparsity problem)
• ❌ Risk of overfitting

Sweet Spot: Small values of n (typically 1-3)

---

🤖 Classification Method

Algorithm: Multinomial Logistic Regression (Softmax Regression)

Why This Model?

✅ Handles multinomial classification - Perfect for 6 languages
✅ Fast inference - Quicker predictions than Naive Bayes
✅ Better performance - More accurate than Naive Bayes with same training data
✅ Probabilistic output - Gives confidence scores for each language

Alternative Considered: Naive Bayes

• ❌ Slower inference
• ❌ Lower accuracy
• ✅ Still effective, but not optimal

---

📊 How It Works

Training Process

1. Collect text samples from all 6 languages
   ↓
2. Extract character n-gram features (e.g., bigrams, trigrams)
   ↓
3. Build feature vectors for each text sample
   ↓
4. Train Softmax Regression classifier
   ↓
5. Model learns character patterns for each language

Prediction Process

1. New text input
   ↓
2. Extract same character n-gram features
   ↓
3. Apply trained Softmax model
   ↓
4. Output: Language probabilities for all 6 languages
   ↓
5. Select language with highest probability

---

📈 Key Concepts

Character Frequency

Concept: Different languages prefer different letters.

Example:

• Spanish: High frequency of 'ñ', 'll', 'rr'
• English: Common 'th', 'qu', 'ch'
• Swahili: Frequent 'w', 'm' combinations

Phonotactics

Concept: Rules about how sounds (characters) can combine in a language.

Example:

• English allows "str" at start of words (street, strong)
• Some languages never allow 3 consonants together
• Tagalog has many repeated syllables (pag-pag, kulit-kulit)

Feature Sparsity

Concept: As features increase, data becomes sparse (mostly zeros).

Problem:

• Unigrams: 26 features (manageable)
• Bigrams: 26² = 676 features (still okay)
• Trigrams: 26³ = 17,576 features (getting sparse)
• 4-grams: 26⁴ = 456,976 features (too sparse!)

Solution: Use small n-values (1-3) to balance power and sparsity.

---

🎓 Learning Objectives

1. Character-based NLP - Understand why characters matter for language ID
2. Feature Engineering - Design effective n-gram features
3. Multinomial Classification - Handle 6-way classification problems
4. Softmax Regression - Implement and understand the algorithm
5. Sparsity Management - Balance feature richness vs data sparsity
6. Model Comparison - Evaluate Softmax vs Naive Bayes

---

💡 Real-World Applications

Content Moderation:

• Automatically route multilingual content to appropriate moderators
• Filter content by language

Translation Services:

• Detect source language before translation
• Support code-switching detection

Search Engines:

• Return results in user's language
• Improve multilingual search

Social Media:

• Tag posts by language
• Recommend content in user's language

Customer Service:

• Route inquiries to language-appropriate agents
• Analyze feedback by language

---

🔍 Key Takeaways

1. Not all features are equal - Characters > Words for language ID
2. Context matters - N-grams capture patterns unigrams miss
3. More isn't always better - Large n causes sparsity
4. Right tool for the job - Softmax > Naive Bayes for this task
5. Latin script languages are distinguishable - Through character patterns

---

📚 References

Languages:

• Hausa - West African
• Indonesian - Southeast Asian
• Manobo - Philippine
• Nahuatl - Mexican
• Swahili - East African
• Tagalog - Philippine

Concepts:

• Latin Script
• Diacritics
• Phonotactics

---

🎯 Quick Summary

What: Build a classifier to identify 6 languages
Why: Languages use same alphabet, can't tell them apart visually
How: Character n-grams + Softmax Regression
Challenge: Balance feature richness with data sparsity
Result: Accurate, fast language detection system

---

This assignment demonstrates how character-level features and appropriate classification algorithms can solve complex multinomial classification problems in NLP.