Repository navigation
FSTGuardrails for RAG A Lucene Developer's Guide
A deterministic, sub-millisecond, dictionary-driven guardrails + normalization layer for any Retrieval-Augmented Generation pipeline, built on the Apache Lucene 9.10 Finite State Transducer.
Most RAG architecture diagrams jump straight from "user query" to "embedding model" to "vector store" to "LLM." That picture is missing a layer that anyone who has actually shipped a RAG system in production ends up building usually twice, usually badly:
A fast, deterministic, dictionary-aware text scanner that runs before embeddings and before the LLM, on every query and every ingested document.
FSTGuardrails is that layer. It uses the same data structure that powers Lucene's suggester, synonym filter, and the venerable Solr Text Tagger the FST (Finite State Transducer) and exposes it as a tiny standalone service (HTTP, MCP, REPL, or library).
This wiki page is written for the audience that benefits most: Lucene / search engineers who are now being asked to bolt RAG onto an existing search stack, and who are looking for the boring-but-correct primitives to do it with.
A typical RAG pipeline:
┌────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────┐
user query │ │ │ │ │ │ │ │
──────────► │ embedder │──►│ vector store │──►│ re-ranker │──►│ LLM │──► answer
└────────────┘ └──────────────┘ └─────────────┘ └──────┘
What FSTGuardrails adds:
┌──────────────────┐
│ FSTGuardrails │ (dictionary FST, ~µs per query)
user query │ • intents │
──────────► │ • entities │──┬──► route to canonical handler (skip LLM entirely)
│ • offensive │ │
│ • products │ └──► annotated query ──► embedder ──► … ──► LLM
└──────────────────┘
And symmetrically on the ingestion side:
┌──────────────────┐
│ FSTGuardrails │
document ──►│ normalize │──► chunker ──► embedder ──► vector store
│ tag entities │
└──────────────────┘
The same FST runs in both directions, so the strings the retriever indexes and the strings the query layer searches against are guaranteed to be canonicalized the same way.
Given a dictionary of phrases (loaded from CSV files at startup) FSTGuardrails will, for any input text, return every matching phrase with:
| Field | Meaning |
|---|---|
start |
Character offset where the match begins |
end |
Character offset where the match ends |
surface |
The exact substring as it appeared in the input |
id |
Stable id for the matched entity (UUID or your own) |
type |
The dictionary the match came from (CSV filename) |
output |
Canonical / normalized form (e.g. STATUS, APACHELUCENE) |
Crucially, it does this in a single forward arc-walk over the input no regex backtracking, no per-phrase hash lookups, no nested loops over the dictionary. The FST is the dictionary.
If you have spent any time in Lucene internals you already know the answer, but for completeness:
- Compactness. An FST shares prefixes and suffixes between phrases. A 168-entry intent dictionary in FSTGuardrails fits in under 33 KB of RAM. A dictionary of a few hundred thousand entities is comfortably in the low MB range.
- Determinism. A match is either in the dictionary or it isn't. There are no false positives from cosine-similarity weirdness. This is exactly what you want for guardrails: blocking offensive language with a 0.83 similarity score is malpractice.
- Speed. One transition per input character/token. Throughput is measured in millions of tokens per second on a single core.
-
Longest-match-wins. Forward maximum match falls out of the FST traversal for free, so
"New York City"correctly beats"New York"without any post-processing. -
Analyzer-aware. The same Lucene analyzer that built the FST is applied to incoming text, so ASCII folding (
"Zürich"→"Zurich") and case handling are consistent end-to-end. - No model, no GPU, no network call. It is a library. It runs in your JVM. It does not talk to OpenAI. It does not need a vector DB. It does not need a sidecar. It is ~12 MB.
For comparison, the alternatives:
| Approach | Throughput | Determinism | Memory | Operational cost |
|---|---|---|---|---|
| Regex per phrase | poor at scale | yes | low | low |
| Aho-Corasick | very high | yes | medium | low |
| Lucene FST (FSTGuardrails) | very high | yes | very low | low |
| Embedding similarity | medium | no | high | GPU / API calls |
| LLM classifier | low | no | very high | $$ per request |
Aho-Corasick is the honest competitor here; the FST wins on memory and on integration with the rest of Lucene's analyzer chain, which matters a lot if your search stack is already Lucene/Solr/Elasticsearch.
The cheapest LLM call is the one you don't make.
data/intent.csv:
intent,action,response
track my order,STATUS,Checking order status
check availability,AVAILABILITY,Checking stock
cancel my order,CANCEL,Cancelling orderAt query time:
POST /tag
{"text": "hey can you track my order #1234"}
→ docs: [{ surface:"track my order", type:"intent", output:"STATUS" }]
Your gateway sees output:"STATUS" and routes directly to your order-status handler. No embeddings. No vector lookup. No LLM. The 95th-percentile latency on that path drops by two orders of magnitude.
data/offensive_en.csv is bundled. You add your own redaction lists (data/pii_email_local_parts.csv, data/internal_codenames.csv, …).
The FST runs over the user message and the LLM's draft response. Any match in type=offensive_en or type=pii_* is blocked or redacted before the response leaves your service. Because it's deterministic, you can prove which rule fired and where in the text auditors love this; embedding-based classifiers cannot give you that.
A common RAG bug: the document corpus says "Apache Lucene", the user types "lucene", the embedding model thinks they're 0.91 similar, the retriever ranks them below something with 0.93, the answer is wrong.
Run FSTGuardrails over both ingestion and query text. Replace every recognized surface form with its canonical output token (APACHELUCENE). Now the retriever is matching identical strings, the embedder sees identical strings, and your top-k stops drifting.
If you run BM25 + vector hybrid search (and if you don't, you probably should), FSTGuardrails gives you a clean way to:
- Boost the BM25 side for any term whose
typeis a product / SKU / brand. - Add filter clauses for entities (
type:CITY,type:MACHINE_FAMILY) without doing a second LLM call to extract them. - Strip recognized stop-intents (
"please","can you") before embedding, so the vector reflects the substantive query.
During ingestion, tag every document once with the full dictionary and store the offsets alongside the chunk. At retrieval time you can:
- Promote chunks that contain entities matching the query's tags.
- Build a citation pane that highlights exactly which dictionary phrase matched, with character offsets.
- Generate structured metadata (entities, intents, topics) for free, without a second model pass.
FSTGuardrails ships four runtimes from the same fat JAR:
| Mode | Command | Use when |
|---|---|---|
| HTTP | java -jar tagger.jar serve 8080 |
Any RAG service that can make an HTTP call |
| MCP | java -jar tagger.jar mcp |
Exposing the tagger as a tool to an MCP-aware LLM (Claude) |
| REPL | java -jar tagger.jar repl |
Interactive dictionary debugging |
| Library | embed App.java / TextTagger directly |
In-process inside your Java/Kotlin/Scala RAG service |
The HTTP endpoint supports two response shapes (format=simple and format=solr), which matters if you're already running a Solr Text Tagger and want to swap the implementation underneath without changing the clients.
curl "http://localhost:8080/tag?text=Ada+Lovelace+uses+Apache+Lucene+in+New+York+City"{
"totaltime": 1,
"text": "Ada Lovelace uses Apache Lucene in New York City",
"docs": [
{"start":0, "end":12, "surface":"Ada Lovelace", "id":"per:ada", "type":"PERSON", "output":"ADALOVELACE"},
{"start":18,"end":31, "surface":"Apache Lucene", "id":"sw:lucene", "type":"PRODUCT", "output":"APACHELUCENE"},
{"start":35,"end":48, "surface":"New York City", "id":"geo:nyc", "type":"CITY", "output":"NEWYORKCITY"}
]
}Add to claude_desktop_config.json:
{
"mcpServers": {
"lucene-tagger": {
"command": "java",
"args": ["-jar", "/absolute/path/to/tagger.jar", "mcp"],
"env": { "DATA": "/absolute/path/to/data" }
}
}
}The LLM now has a tag tool. You can let it decide when normalization is needed (cheap models do this reliably) or you can call the tool yourself before prompting and inject the results into the system prompt.
For the lowest possible latency, embed TextTagger directly. No JSON serialization, no HTTP, no process boundary just an arc-walk over a string.
This is the part most people get wrong, so it deserves its own section.
Filename = type. So intent.csv produces matches with type:"intent". offensive_en.csv produces type:"offensive_en". Keep types granular it's much easier to filter, weight, and audit downstream when you have pii_email, pii_phone, pii_ssn than one giant pii bucket.
intent,action,response
track my order,STATUS,Checking order statusIf you omit action, the canonical output is derived from the phrase (uppercased, non-alphanumerics stripped). Use that for entities where the phrase itself is already canonical (machine_families.csv).
The temptation is to throw every term from a Wikipedia category dump into a CSV. Don't. The value of an FST guardrails layer is precision. A 500-entry hand-curated product dictionary will outperform a 500,000-entry scraped one in every RAG metric that matters, because the scraped one will tag every English word that happens to also be a brand name.
The library supports incremental dictionary updates (see demo section 6 in App.java). For RAG pipelines, the right cadence is usually:
- Intents, offensive, PII manual review, weekly or per-incident.
- Products / SKUs / entities nightly job from your authoritative catalog DB.
- Topics / taxonomies whenever the taxonomy itself changes.
Don't rebuild on every request. Don't rebuild on every document ingest. Batch.
- Memory: dictionaries with up to a few hundred thousand entries fit in tens of MB. Pre-size your JVM heap accordingly; this is not a workload that wants a 32 GB heap.
-
Concurrency: the tagger is thread-safe for reads; demo section 7 in
App.javashows the read/write lock pattern for rebuilds. - Analyzer mismatch is the #1 footgun. If you change the analyzer (stemming, stop words, different tokenizer) without rebuilding the FST, your matches will silently drop. Lock the analyzer in code and treat changes to it as a rebuild trigger.
-
reset()your TokenStream. Same Lucene gotcha as always; demo section 8 covers it. - Don't add stop-word phrases to dictionaries. If your analyzer strips them, they will never match and you will waste hours wondering why.
-
Containerize it. The repo ships a
Dockerfileanddocker-compose.yml. Mount your CSV directory at/dataand setDATA=/data.
Be honest about the tool's limits:
-
Fuzzy / typo-tolerant matching not its job. Use an embedding model, or layer a Levenshtein/Lucene
FuzzyQueryafter the FST falls through. - Free-form intent classification with no fixed phrase list use a small classifier model. FSTs need a dictionary.
-
Semantic similarity by definition, an exact-match dictionary cannot tell you that
"refund my purchase"≈"give me my money back"unless both are in the dictionary.
The right mental model is: FSTGuardrails handles the cases you already know about, cheaply and deterministically; the LLM handles the long tail. That is the entire pitch.
A production RAG stack with FSTGuardrails in place tends to look like this:
┌────────────────────────────────┐
│ FSTGuardrails (HTTP / library) │
│ intents, entities, offensive │
└──────────────┬─────────────────┘
│
┌────────────────────────┼───────────────────────────┐
▼ ▼ ▼
block / redact canonical-route annotated query
(skip LLM entirely) │
▼
┌───────────────┐
│ hybrid search │
│ BM25 + vector │
└──────┬────────┘
▼
┌───────────────┐
│ re-ranker │
└──────┬────────┘
▼
┌───────────────┐
│ LLM │
└──────┬────────┘
▼
FSTGuardrails (output side)
• offensive / PII redaction
• citation entity highlight
Two passes of the same FST. One dictionary. One place to audit. One place to update when legal hands you a new banned-word list at 4pm on a Friday.
- Repo: https://github.com/jsclosures/fstguardrails
- Apache Lucene FST package:
org.apache.lucene.util.fst - Solr Text Tagger (the conceptual ancestor): https://github.com/OpenSextant/SolrTextTagger
- Model Context Protocol: https://modelcontextprotocol.io