text-chunker

npm install @ferrow/text-chunker

Boundary-respecting text chunking for RAG pipelines — sentence, paragraph, or markdown-heading aware, configurable overlap, per-chunk metadata, zero runtime dependencies, strict TypeScript.

Quickstart

import { chunkText } from "text-chunker";

const doc = `# Getting Started

This library helps you chunk documents for RAG pipelines.

## Usage

Longer usage content goes here...`;

const chunks = chunkText(doc, {
  boundary: "markdown",
  targetSize: 300,
  sizeUnit: "chars",
  overlap: 30,
});

for (const c of chunks) {
  console.log(c.index, c.headingPath, c.text.length, c.hardSplit);
}

API

chunkText(text, options?)

• text: string
• options.boundary?: "sentence" | "paragraph" | "markdown" — default "paragraph".
• options.targetSize?: number — default 500.
• options.sizeUnit?: "chars" | "tokens" — default "chars".
• options.overlap?: number — overlap between consecutive chunks, in sizeUnit. Default 0.
• Returns Chunk[].

Each Chunk is:

interface Chunk {
  text: string;
  index: number;
  start: number;   // char offset in the source text
  end: number;      // char offset, exclusive
  headingPath: string[]; // markdown heading stack; [] for non-markdown input
  hardSplit: boolean;    // true if a unit exceeded targetSize and was force-split
}

estimateTokens(text)

Tiny built-in chars / 4 estimator used only when sizeUnit: "tokens". For a fuller heuristic (CJK, code density, per-model calibration), see the sibling token-estimator package.

Limits

• Chunks never split mid-sentence/paragraph/section unless a single unit alone exceeds targetSize — in that case it is hard-split and flagged hardSplit: true. Set a generous targetSize if you never want hard splits.
• markdown boundary mode splits on heading lines only, not sub-paragraphs within a section — a whole section under a heading is one unit until it exceeds targetSize.
• The token estimator is a rough chars/4 heuristic, not a real tokenizer.
• Sentence splitting is regex-based (./!/? boundaries) and does not handle abbreviations, decimal numbers, or non-Latin punctuation specially.
• Overlap is measured in the same unit as targetSize and is built from whole boundary units carried from the tail of the previous chunk — it will not exceed overlap but may be smaller than requested if the last unit alone is larger than overlap.

---

Part of the ferrow-toolkit collection · Sponsored by Ferrow