Linguae (latest)

linguae

Topics: language-detection, rust, candle, machine-learning, cli, fastapi

Detect the language of text with a local character-level model — no cloud, no pretrained weights, no network calls.

Linguae is a local-first language identification project. It trains a
character-level CNN from scratch with Candle on the Tatoeba (short sentences),
WiLI-2018 (long paragraphs), and per-language WiLI word datasets and ships a
standalone command-line tool that predicts exactly one language per input
string. Everything runs on your own machine.

Live demo: https://hieudoanm.github.io/linguae/

Table of Contents

• The Problem
• The Solution
• Features
• See It in Action
• Get Started
• HTTP Server and Web App
• Use Cases
• Why This Project?
• Training Visualisations
• Project Status and Trust
• Community and Contributions
• Call to Action
• License

The Problem

Language identification still depends on cloud APIs, pretrained weights, or
third-party services — none of which are acceptable for offline, privacy-sensitive,
or air-gapped workflows. Even open-source models often bundle large pretrained
backports that are hard to inspect or shrink.

The Solution

Linguae trains a small character-level CNN from scratch with
Candle and ships a small Rust CLI
binary that runs inference with the same runtime. The model performs
single-label classification per string; it does not detect mixed-language
spans.

• Local-first & private — Inference runs entirely on your machine. Your text never leaves your environment.
• From text to language in one command — linguae "Bonjour tout le monde" prints fra.
• Honest, task-aware model — One language label per string, with optional confidence scores and top-k alternatives.
• Reproducible by design — Fixed random seed, pinned dependency versions, and a deterministic train/val/test split that groups exact duplicate texts within a single split.
• No pretrained models — The character-level CNN is trained from scratch on the Tatoeba, WiLI-2018, and WiLI word datasets.
• Short and long input — Tatoeba supplies short conversational sentences; WiLI-2018 supplies long sentences and paragraphs; the WiLI word dataset covers one- to few-word strings, all sharing one label space.

Features

• Character-level CNN — Embedding, 1D convolutions (kernel sizes 3, 5, 7), masked max-pooling that ignores padding, and a fully connected classification head.
• Single-label prediction — One ISO 639-3 language code per input string.
• Confidence & top-k — --confidence appends a softmax score; --json emits JSON Lines with text, language, confidence, and alternatives.
• Batch mode — --file PATH classifies each non-empty line of a UTF-8 file.
• Device selection — --device cpu, cuda, or metal (mps is an alias for metal).
• Standalone binary — A small Rust binary distributed via GitHub Releases tagged latest; no Python runtime required.

See It in Action

Or run the commands directly:

linguae "Bonjour tout le monde, comment allez-vous aujourd'hui dans ce beau monde"
# fra

linguae --confidence "Hello world, this is a longer English text to test language detection"
# eng	0.6114

printf 'Привет, мир, это более длинный русский текст' | linguae
# rus

linguae --json "Bonjour tout le monde, comment allez-vous ?"
# {"text": "Bonjour tout le monde, comment allez-vous ?", "language": "fra", "confidence": 0.826987, ...}

Get Started

Prerequisites

• Rust 1.85+ (CLI, data pipeline, training)
• Python 3.14.8 and uv (HTTP server only)

Install the CLI

cd crates
cargo build --release -p linguae

Train the model (optional)

cd crates
cargo run --release -p linguae-data -- all   # build the WiLI, Tatoeba and word CSVs
cargo run --release -p linguae-train -- train

The trainer writes the artifact to crates/linguae-train/artifacts/linguae and
SVG plots to crates/linguae-train/artifacts/images.

Run

crates/target/release/linguae "Bonjour tout le monde"

HTTP Server and Web App

The packages/app/server package exposes the same language detector as a FastAPI HTTP API. The packages/app package contains a Next.js web app to interact with it.

Install and Run

# Start both with docker-compose
docker-compose up --build

# Or run separately
cd packages/app/server
uv sync
linguae-server

cd packages/app
npm install
npm run dev

The server binds to 0.0.0.0:8000 by default. The web app runs on http://localhost:3000.
Open http://127.0.0.1:8000/docs for the interactive API docs.

Endpoints

Method	Path	Body	Description
GET	/health	—	Health check
POST	/detect	JSON	Single text detection
POST	/detect/form	Form-encoded	Single text (form upload)
POST	/detect/batch	JSON	Batch of texts
POST	/detect/file	File upload	Batch from UTF-8 text file

Examples

Detect a single language via JSON:

curl -X POST http://127.0.0.1:8000/detect \
  -H "Content-Type: application/json" \
  -d '{"text":"Bonjour tout le monde, comment allez-vous ?","top_k":3}'
# {"text":"...","language":"fra","confidence":0.99,"top_k":[...]}

Detect via form upload:

curl -X POST http://127.0.0.1:8000/detect/form \
  -d "text=Hello world, this is English" \
  -d "top_k=3"

Batch detection via JSON:

curl -X POST http://127.0.0.1:8000/detect/batch \
  -H "Content-Type: application/json" \
  -d '{"texts":["Bonjour tout le monde","Hello world"],"top_k":2}'

Batch detection from an uploaded file:

curl -X POST http://127.0.0.1:8000/detect/file \
  -F "file=@texts.txt"

Configuration

All settings are overridable via environment variables (see packages/app/server/README.md):

Variable	Default	Description
LINGUAE_SERVER_HOST	0.0.0.0	Bind host
LINGUAE_SERVER_PORT	8000	Bind port
LINGUAE_ARTIFACT_DIR	crates/linguae-train/artifacts/linguae	Artifact directory
LINGUAE_DEVICE	cpu	Torch device
LINGUAE_RATE_LIMIT	true	Enable rate limiting
LINGUAE_RATE_LIMIT_REQUESTS	60	Max requests per window
LINGUAE_RATE_LIMIT_WINDOW	60	Window size in seconds
LINGUAE_CORS	true	Enable CORS
LINGUAE_CORS_ORIGINS	*	Comma-separated origins
LINGUAE_TIMEOUT	30.0	Request timeout in seconds

Middlewares

• Request ID — X-Request-ID header injected and echoed on every response.
• Rate limiting — sliding window keyed by client IP (default 60 req/60s).
• CORS — configurable origins, methods, and headers.
• GZip compression — responses above 500 bytes are compressed.
• Timeout — X-Response-Time-Seconds header on every response.

Use Cases

• Rapid prototyping — Detect the language of text snippets without setting up a full NLP pipeline.
• Offline workflows — Keep sensitive data on-device while classifying text.
• Teaching & learning — Explore character-level CNNs, train/val/test splitting, and evaluation end-to-end.
• Model portability — Ship a single binary that runs on any Linux machine without Python installed.

Why This Project?

• Focused scope — Does one thing well: predict one language per string from a character-level CNN. No AutoML bloat, no pretrained weights.
• Transparent — Clear training pipeline, per-metric explanations, and reproducibility metadata.
• Local-first — Runs entirely on your machine; no vendor lock-in or external dependencies.
• Production-minded demo — Real Rust/Candle training and inference implementation with CI, linting, and GitHub Releases.

Training Visualisations

The model is trained end-to-end on Tatoeba (short sentences) and WiLI-2018
(long paragraphs). The following visualisations are produced in
crates/linguae-train/artifacts/images/:

![Dataset quality](crates/linguae-train/artifacts/...