Language Selection and Exclusion Criteria

🚫 Language Selection, Feasibility & Exclusion Criteria

This document details the empirical findings, quantitative performance metrics, and formal evaluation criteria governing which languages are admitted, prioritized, or excluded from the automated neural translation pipeline of Prof. Payer's Sanskrit Course.

---

1. Executive Summary & Context

Between August 28 and September 6, 2026, the translation pipeline operated autonomously for 248.3 cumulative GPU hours using local inference on nyx.local:8000 (Qwen3.6-35B-A3B-4bit on Apple Silicon).

While 2,710 Markdown files were processed across 35+ target languages, the empirical data revealed an extreme operational asymmetry:

• 71.5% of total GPU compute (177.6 hours) was consumed by 10 low-resource languages that suffered severe quality assurance (QA) rejection rates (>50% to 88% rejected files).
• Conversely, once systemic cache-filter desynchronizations (punctuation regex lookaround boundaries and IAST Unicode stripping) were resolved, well-supported target languages finished cleanly with zero residual fallbacks within minutes, advancing the suite of 100% completed languages from 7 to 19 languages (140/140 files each).

To prevent open-ended resource exhaustion and safeguard the scholarly integrity of the published textbook, a formal Three-Tier Exclusion Framework was instituted.

---

2. Empirical Findings from the 248.3-Hour Benchmark

Compute Distribution Across Bottleneck Languages

The following table records the top time-consuming languages during the 10-day continuous run prior to the exclusion policy:

Locale	Language	Family / Type	Compute Time	Clean Files	Fallback / Rejected	Completion Rate	Operational Assessment
et	Estonian	Uralic (Finno-Ugric)	32.7 h	26 / 140	114	18.6%	Severe morphological failure; rejected by QA
am	Amharic	Semitic (Ethiopic)	32.1 h	63 / 140	77	45.0%	Heavy sub-word tokenization penalty; slow throughput
zu	isiZulu	Niger-Congo (Bantu)	17.9 h	70 / 140	70	50.0%	Zero philological nexus to Sanskrit; high repetition rate
cop	Coptic	Afroasiatic (Extinct liturgical)	14.6 h	16 / 140	124	11.4%	Extreme LLM vocabulary sparsity; severe hallucination
sl	Slovenian	Indo-European (Slavic)	14.3 h	56 / 140	84	40.0%	Grammar terminology confusion; excessive retries
si	Sinhala	Indo-Aryan (Insular)	14.1 h	60 / 140	80	42.9%	Script rendering overhead; high token latency
ka	Georgian	Kartvelian	13.4 h	61 / 140	79	43.6%	Agglutinative case misalignment
hy	Armenian	Indo-European (Armenian)	13.4 h	65 / 140	75	46.4%	Grammatical terminology drift
sv	Swedish	Indo-European (Germanic)	13.3 h	53 / 140	87	37.9%	False-positive QA stops on Latinate roots
te	Telugu	Dravidian	11.8 h	68 / 140	72	48.6%	Complex sandhi rendering delays
Sum	Top 10	—	177.6 h	—	—	—	71.5% of total cluster runtime

Root Causes of the Automated Bottleneck

1. Vocabulary Sparsity & Sub-Word Fragmentation: Open-weights LLMs (such as Qwen 3.6) have tokenizers heavily optimized for English, Chinese, and major world languages. For low-resource languages (cop, am, zu), words are split into 4 to 8 byte-fallback tokens. This causes generation speeds to plummet from ~22 t/s down to 3–6 t/s, multiplying compute requirements by a factor of 4x to 6x.
2. Lack of Digitized Reference Grammars: Synthesizing university-level German philological treatises on Sanskrit grammar into languages that lack established Sanskritological research literature forces models to invent artificial neologisms or regress into English/German loan translations.
3. The Cache/QA Desynchronization Loop: When generation filters are less stringent than the gatekeeping verification script (translation_qa.py), tainted chunks enter the Translation Memory (.payer/tm/). The runner repeatedly re-injects these cached chunks, triggering infinite validation loops that cycle for days without producing incremental progress.

---

3. The Three-Tier Exclusion Framework

To establish a principled, reproducible governance model, languages are evaluated across three sequential gates:

                  ┌──────────────────────────────────────────────┐
                  │ 1. Philological & Contextual Relevance Gate   │
                  └──────────────────────┬───────────────────────┘
                                         │ Pass
                                         ▼
                  ┌──────────────────────────────────────────────┐
                  │ 2. Quantitative Efficiency / ROI Gate        │
                  └──────────────────────┬───────────────────────┘
                                         │ Pass
                                         ▼
                  ┌──────────────────────────────────────────────┐
                  │ 3. Production Frontend Visibility Gate       │
                  └──────────────────────────────────────────────┘

Gate 1: Philological & Contextual Relevance

A candidate language must satisfy at least one of the following criteria to qualify for full textbook localization:

• Indo-European Comparative Linguistics: Languages essential for comparative Indo-European grammar, phonology, and historical linguistics (e.g., Latin la, Ancient Greek grc, Lithuanian lt, Old Church Slavonic/Slavic languages, Romance, Germanic).
• Primary Textual Transmission Media: Historical and living vehicular languages of Buddhist, Hindu, or Jain literary transmission (e.g., Classical and Modern Chinese zh/zh-CN, Hindi hi, Tamil ta, Thai th, Sinhala si, Punjabi pa).
• Swiss National Languages & Cultural Heritage: As part of the project's Swiss institutional and cultural anchoring, all Swiss national languages are permanently retained: German (de), French (fr), Italian (it), and Rumantsch Grischun (rm). Rumantsch (rm) Fallback Protocol: Due to the limited digital corpus and the absence of standardized modern Sanskritological literature in Romansh, German (de) fallbacks are explicitly accepted and authorized (DE_FALLBACK_ALLOWED). Untranslatable grammatical passages gracefully default to the German source rather than triggering pipeline failure or exclusion.
• Major Global Scholarly Medium: Languages with substantial academic readership in modern Indology (English, French, German, Russian, Spanish, Japanese).

Exclusion Rule: Languages with no historical relationship to Sanskrit, no textual transmission lineage, no Swiss national language status, and no academic tradition in Sanskrit studies (e.g., zu, nah, qu) are excluded to avoid artificial and unverified translations.

Gate 2: Quantitative Efficiency & ROI (Circuit Breaker)

A language active in the automated pipeline is subject to an automated circuit breaker:

• Max Compute Cap: If a language consumes >10 GPU hours with <30% clean completion (or QA rejection rate >70%), automated execution is immediately suspended.
• Stagnation Trap: If two consecutive full pipeline iterations yield Delta = 0 clean files, the language is automatically marked as Maschinelles Limit and skipped.
• Manual Deferral: Suspended languages may not be retried with the local 35B model. They are scheduled for either:
  1. Manual human post-editing.
  2. Frontier fallback tier (e.g., Anthropic Sonnet 5 batch runs), where authorized.

Gate 3: Production Frontend Visibility Cutoff

To preserve presentation quality for students and scholars:

• Hard Cutoff: A language is only exposed in the public VitePress navigation dropdown and language switcher (docs/.vitepress/languages.mjs) once it achieves 100.0% completion (140/140 canonical files, 0 fallbacks).
• Work-in-Progress Isolation: Incomplete languages (50%–99%) remain tracked in the local repository and testable via the internal QA Viewer (/docs/public/qa_viewer.html), but are hidden from general production navigation to prevent users from encountering half-translated or fallback content.

---

4. Exclusion Dossier: Formal Case Studies

Excluded in Current Cycle (September 2026)

1. Coptic (cop / ⲙⲉⲧⲣⲉⲙⲛ̀ⲭⲏⲙⲓ)

• Status: Fully Excluded.
• Metrics: 14.6 GPU hours; 16 / 140 files clean (11.4%); 124 rejected files.
• Technical Rationale: Extinct liturgical Afroasiatic language. Modern digital corpora are virtually non-existent in base LLM training weights. The model frequently output garbled Greek-derived script, pseudo-phrases, or reverted wholesale to English/German.
• Philological Rationale: No textual or comparative intersection with Sanskrit grammar.

2. isiZulu (zu)

• Status: Fully Excluded.
• Metrics: 17.9 GPU hours; 70 / 140 files clean (50.0%); 70 stalled files.
• Technical Rationale: Severe tokenizer penalty resulting in low token throughput. The agglutinative nominal prefix morphology of Zulu was repeatedly disrupted by Sanskrit grammatical categories (e.g., dual number, thematic vowel stems), leading to unrecoverable LLM hallucination loops.
• Philological Rationale: No historical, liturgical, or academic connection to Sanskrit linguistics.

3. Estonian (et / Eesti)

• Status: Fully Excluded.
• Metrics: 32.7 GPU hours; 26 / 140 files clean (18.6%); 114 rejected files.
• Technical Rationale: Non-Indo-European (Finno-Ugric) low-resource language in this domain. While general conversational Estonian is well-represented, scientific grammar terminology aligned with Sanskrit declensions repeatedly failed Lingua language identification and strict QA phrase checks, resulting in the highest single-language compute waste in the project's history.
• Philological Rationale: Insufficient scholarly audience to warrant 30+ hours of localized fine-tuning.

---

Previously Documented Exclusions

• Akkadian (akk) & Elamite (elx): Ancient extinct languages; zero digital corpora for contemporary linguistic textbooks; would require entirely fabricated neologisms.
• Aramaic / Syriac (arc): Sparse LLM pre-training data; right-to-left layout complications combined with dialect fragmentation.
• Nahuatl (nah) & Quechua (qu): Indigenous American languages lacking established Sanskritological grammatical terminology in open LLMs; generates persistent Spanish fallbacks.

---

5. Architectural Implementation

The exclusion policy is synchronized across four architectural components:

1. VitePress Single Source of Truth (docs/.vitepress/languages.mjs): Excluded locales are removed from ACTIVE_LOCALES and recorded in the module header with their respective technical justifications.
2. Static Site Generator (docs/.vitepress/config.mjs): Locale objects and sidebars for excluded languages are decoupled from VitePress build pipelines, reducing production bundle sizes and memory footprint during static builds.
3. Internal QA Suite (docs/public/qa_viewer.html): Kept strictly in parity with ACTIVE_LOCALES via scripts/sync_qa_viewer.py to ensure side-by-side verification tools accurately reflect active targets.
4. Translation Pipeline Engine (scripts/translation/config.py & scripts/generate_report.py): Excluded language keys are removed from LANGUAGES and LANG_MAP. This immediately prevents run_all_translations.sh and cron report generators from expending cycles on decommissioned locales.