langchunk

Turn a Universal Dependencies parse into sentences → clauses → phrases → words, with every unit traceable back to the exact characters it came from.

npm install langchunk

Clause extraction from a dependency parse is the piece the NLP ecosystem doesn't ship: parsers give you trees, and nothing turns a tree into the grammar a learner or teacher actually asks about — which clause is independent, what is its subject, where does the relative clause attach. langchunk is that layer: deterministic TypeScript over any UD tree, graded at 100% against hand-annotated gold treebanks — not a threshold, an exact bar — and measured end-to-end per language with real parsers.

import { parseConlluSentences, buildGoldDocument, buildDocument, packOrFallbackFor } from "langchunk";

// Bring any CoNLL-U — from Stanza, spaCy, UDPipe, a treebank, anywhere.
const sentences = parseConlluSentences(conlluText);
const gold = buildGoldDocument(sentences);
const pack = packOrFallbackFor("en");

const doc = buildDocument({
  text: gold.text,
  sentences: gold.sentences,
  language: { code: "en", tier: pack.tier, resolution: "declared" },
  analyzer: { id: "my-parser", version: "1" },
  options: pack.grammar,
});

// doc.sentences / doc.clauses / doc.phrases / doc.words — each with a
// span into the ORIGINAL text: text.slice(span.start, span.end) === unit.text.

What you get

• The taxonomy. Sentences, clauses (independent / coordinated / dependent with role), phrases (NP/VP/PP/AdjP/AdvP with heads), words (including multi-word units) — every unit carrying a span into the original string and a confidence.
• Five language packs built in — English, Russian, Persian, French, German — each with measured accuracy, plus a plugin mechanism where a new language is a JSON file, not a code change. A language with no pack still parses, honestly labelled broad-fallback.
• Segmentation (segmentSentences) with per-language, corpus-measured abbreviation handling.
• Exports (langchunk/export): CSV, JSONL, CoNLL-U, Anki decks.
• Analyzers: the gold CoNLL-U analyzer (browser-safe, shown above), and Node-only bridges — langchunk/analyzers/stanza (spawns Python Stanza, the highest-accuracy path) and langchunk/analyzers/onnx (pure Node, needs the optional onnxruntime-node peer).

Subpaths mirror the internals: langchunk/schema, /grammar, /segment, /lang, /conllu, /pipeline, /eval, /export, /validators, /lang-node, /analyzers/{gold,agreement,stanza,onnx}. The root export is the curated common path. Everything except lang-node and the stanza/onnx analyzers is browser-safe, and a boundary checker enforces that claim in CI.

The design in one table

Tier 1	A dependency parser produces a Universal Dependencies tree. Ambiguous, learned, replaceable — bring your own.
Tier 2	This library. Deterministic mapping from tree to taxonomy. Language-general, zero runtime dependencies beyond zod.

Structural ambiguity belongs in a model; the taxonomy is a designed scheme, so mapping onto it is deterministic — which is why Tier 2 can be graded exactly. Clause boundaries come from advcl/acl/ccomp/conj/mark relations, never from keyword scanning.

Measured accuracy (strict F1, 300-sentence Gate 2 runs)

End-to-end with a real parser, against the same Tier 2 over gold trees — so every gap shown is parser error, not taxonomy error:

language	parser	clause	phrase	word
Persian	Stanza perdt	92.4	93.7	96.4
English	Stanza electra	91.0	94.6	97.3
French	Stanza combined	81.0	80.0	94.0
German	Stanza combined	79.2	83.5	93.6
Russian	Stanza mixed ruBERT	75.6	87.2	91.7

Segmentation is measured separately (Gate 3): of the sentence boundaries a writer actually marked, Persian finds 100%, German 98%, English and French 96.8%. Every number above has a committed report and a test that fails if a change regresses it.

The app

LangChunk the application — a local-first web app over this engine, with on-demand language installs and a local analysis service — lives at khizardevelops/langchunk-app.

Developing this repo

pnpm install
pnpm run ud:fetch en_ewt ru_taiga    # gold treebanks, gitignored
pnpm run verify                      # typecheck + boundaries + the full suite

The suite needs no model, no network, no GPU — a corpora-less clone skips the corpus suites and still passes. Gate 2 needs the Python reference parser:

python3 -m venv --system-site-packages .venv-stanza
.venv-stanza/bin/pip install stanza
pnpm run gate2 --lang en --limit 300
pnpm run gate3 --lang en               # the segmenter; needs no model

pnpm run build:npm builds the publishable package into npm/.

Every non-obvious choice in this repository has a written decision behind it — .agents/decisions.md is the log, and it is meant to be read. docs/ProjectInfo.md is the product contract; docs/UpdatedPlan.md the plan.

License

AGPL-3.0. Use it, modify it, ship it — but keep the source open. The evaluation treebanks and models this repo points at carry their own licenses and are downloaded by you, never distributed here; docs/UpdatedPlan.md Appendix B records them.