Home

Java FST Guard Rails Wiki

A high-performance, dictionary-driven text scanning layer built on the Apache Lucene 9.10 Finite State Transducer (FST).
Drop CSV files in, start the server, and tag millions of tokens per second no ML model, no GPU, no Solr.

1. What it does

Java FST Guard Rails scans free-form text against one or more phrase dictionaries and returns every match with its character offsets, a type label, a canonical id, and a normalized output token.

Use it as a guard rails layer in front of any LLM or search pipeline to catch, classify, and canonicalize:

Category	Example phrase matched	Output token
User intent	`"track my order"`	`STATUS`
Offensive language	`"bullshit"`	`BULLSHIT`
Equipment family	`"backhoe loader"`	`BACKHOELOADER`
Product name	`"Apache Lucene"`	`APACHELUCENE`
Part number	`"6I-6454"`	`6I6454`
Your custom entity	anything in a CSV	anything you define

Every match is returned in a consistent JSON envelope so downstream code never needs to inspect raw text:

{
  "totaltime": 1,
  "text": "I want to track my order and check availability",
  "docs": [
    {"start":10, "end":24, "surface":"track my order",     "type":"intent", "output":"STATUS"},
    {"start":29, "end":47, "surface":"check availability", "type":"intent", "output":"AVAILABILITY"}
  ]
}

2. How the FST works

The Lucene FST is a Finite State Transducer a compact, read-only automaton that maps input byte sequences to output values. The tagger builds it in three explicit steps at startup:

Step 1  createIndex()           build a ByteBuffersDirectory (in-memory Lucene index)
Step 2  addData() / loadCsv()   write phrase documents via IndexWriter
Step 3  TextTagger.fromIndex()  read all phrases via DirectoryReader → compile FST

At query time, tagger.tag(text) runs the analyzer pipeline over the input (tokenizing, lowercasing, ASCII-folding, hyphen-stripping) and then walks the FST arc-by-arc, one token at a time, using a forward maximum-match algorithm. There is no regex backtracking and no per-candidate hash lookup the entire dictionary is traversed in a single linear pass regardless of how many phrases it contains.

Why this matters for guard rails:

A 168-entry intent dictionary and a 1.2 M-row part-number dictionary both produce sub-millisecond response times on the same warm JVM.
The dictionary lives entirely in heap RAM (~33 KB for 168 phrases); there is no disk I/O at query time.
Longest-match-wins is baked in: "New York City" is returned instead of the shorter "New York".

3. Quick start

# 1. Build
cd java-tagger-demo
mvn package -q
# → target/tagger.jar  (~12 MB fat jar)

# 2. Start the HTTP server with bundled sample data
DATA=data java -jar target/tagger.jar serve 8080

# 3. Tag something
curl "http://localhost:8080/tag?text=I+want+to+track+my+order"

{
  "totaltime": 1,
  "text": "I want to track my order",
  "docs": [
    {"start":10,"end":24,"surface":"track my order","id":"...","type":"intent","output":"STATUS"}
  ]
}

4. Run modes

4.1 HTTP server

java -jar target/tagger.jar serve [port]
# PORT env var is also respected (Replit / Docker convention)
DATA=data java -jar target/tagger.jar serve 8080

Method	Endpoint	Input	Response
`GET`	`/tag?text=...`	URL-encoded text	JSON envelope
`POST`	`/tag`	`{"text":"..."}`	JSON envelope
`GET`	`/health`		`{"status":"ok"}`

4.2 MCP server (Claude Desktop)

java -jar target/tagger.jar mcp

Exposes the tagger as a single tag tool over the stdio transport (newline-delimited JSON-RPC 2.0). All startup diagnostics go to stderr so the protocol channel (stdout) carries only JSON-RPC frames.

Claude Desktop config (claude_desktop_config.json):

{
  "mcpServers": {
    "lucene-tagger": {
      "command": "java",
      "args": ["-jar", "/absolute/path/to/tagger.jar", "mcp"],
      "env": { "DATA": "/absolute/path/to/data" }
    }
  }
}

Once connected, Claude can call the tool directly:

"Tag this sentence and tell me if there is any offensive language:
that was total bullshit and I want to track my order"

MCP protocol handled:

JSON-RPC method	Server response
`initialize`	Capabilities handshake (`protocolVersion: 2024-11-05`)
`notifications/initialized`	(none notification, no reply)
`tools/list`	`tag` tool schema with JSON Schema for `text` argument
`tools/call`	Tag envelope embedded in `content[].text`
anything else	`-32601` method-not-found error

Manual test:

{
  echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test"}}}'
  echo '{"jsonrpc":"2.0","method":"notifications/initialized"}'
  echo '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'
  echo '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"tag","arguments":{"text":"Ada Lovelace uses Apache Lucene"}}}'
} | java -jar target/tagger.jar mcp 2>/dev/null

4.3 Interactive REPL

java -jar target/tagger.jar repl
DATA=data java -jar target/tagger.jar repl

Type sentences at the > prompt. Commands: dict (list all phrases), help, quit.

4.4 Demo mode

java -jar target/tagger.jar

Runs all nine built-in demo sections in one pass and exits useful as a sanity check after building.

#	Section
1	Building the dictionary index
2	Basic tagging (forward maximum match)
3	Longest-match-wins `"New York City"` beats `"New York"`
4	ASCII folding `"Zürich"` matches `"Zurich"`
5	Case-sensitive analyzer `"US"` ≠ `"us"`
6	Incremental updates add entries without full rebuild
7	Concurrent tagging 4 threads, read-write lock
8	Common pitfalls analyzer mismatch, stemming, `reset()`, stop words
9	No-match case

5. Tag response format

Every /tag response (HTTP and MCP) uses the same envelope:

{
  "totaltime": 1,
  "text": "the original input string",
  "docs": [
    {
      "start":   0,
      "end":     12,
      "surface": "Ada Lovelace",
      "id":      "per:ada",
      "type":    "PERSON",
      "output":  "ADALOVELACE"
    }
  ]
}

Field	Type	Description
`totaltime`	integer	Query processing time in milliseconds
`text`	string	The input text echoed back verbatim
`docs`	array	One object per matched span, empty array if no matches
`docs[].start`	integer	Start character offset (inclusive)
`docs[].end`	integer	End character offset (exclusive)
`docs[].surface`	string	Matched substring from the original text
`docs[].id`	string	Canonical identifier from the dictionary
`docs[].type`	string	Category label (CSV filename without `.csv`, or hard-coded type)
`docs[].output`	string	Normalized output token for downstream routing

6. Loading CSV dictionaries

Set the DATA environment variable to a directory containing .csv files before starting any run mode. No code changes needed.

DATA=/path/to/my/dictionaries java -jar target/tagger.jar serve 8080

Processing rules:

Files are loaded in alphabetical order.
The first row is the header; every subsequent row becomes one Lucene document.
The first column is the phrase indexed as F_TEXT (blank rows are skipped).
The action column (if present) is stored as output; otherwise output is derived by uppercasing the phrase and stripping non-alphanumeric characters.
type is set to the filename without the .csv extension.
A random UUID is assigned as id for each row.

CSV schema:

Lucene field	Source
`phrase` (`F_TEXT`)	First column value
`id` (`F_ID`)	Random UUID
`type` (`F_TYPE`)	Filename minus `.csv`
`output` (`F_OUTPUT`)	`action` column value, or `deriveOutput(phrase)`

Example intent with explicit action mapping:

intent,action,response
track my order,STATUS,Checking order status
check availability,AVAILABILITY,Checking stock
find a dealer,DEALER_LOCATOR,Finding dealers

Example entity list with no action column:

machine_family
ARTICULATED TRUCK
BACKHOE LOADER
MOTOR GRADER

Error behaviour:

Situation	Behaviour
`DATA` not set	Skip silently; built-in phrases only
Path does not exist	`IllegalArgumentException` at startup
Empty directory / no `.csv` files	Skip with one log line
Blank first column	Row skipped

7. Bundled dictionaries

The data/ directory ships with six CSV files:

File	Rows	Description
`intent.csv`	168	CAT parts app user intent phrases → action tokens
`machine_families.csv`	273	Equipment family names derived from `model_numbers.csv`
`model_numbers.csv`	4,009	Model code → machine family mapping (source file)
`offensive_en.csv`	40	English offensive language terms
`part_numbers.csv`	1,204,314	CAT part numbers
`serial_numbers.csv`	10,426	Equipment serial number prefixes

Note: model_numbers.csv and part_numbers.csv are reference/source files. Point DATA at a subdirectory containing only the files you need to keep startup time fast.

Loading all six at once:

DATA=data java -jar target/tagger.jar serve 8080

Loading only intent and offensive language:

mkdir -p data/guardrails
cp data/intent.csv data/offensive_en.csv data/guardrails/
DATA=data/guardrails java -jar target/tagger.jar serve 8080

8. Docker

# Build image (includes data/ directory)
cd java-tagger-demo
docker build -t lucene-tagger .

# Run with bundled data
docker run --rm -p 8080:8080 lucene-tagger

# Run with your own CSV files
docker run --rm -p 8080:8080 \
  -v /path/to/my/csvs:/data \
  -e DATA=/data \
  lucene-tagger

# One-command start with Docker Compose
docker compose up

Environment variables:

Variable	Default	Description
`PORT`	`8080`	HTTP server port
`DATA`	`/app/data`	Directory of `.csv` files to load

9. Source layout

Everything lives in a single source file: src/main/java/com/harcor/App.java.

Symbol	Kind	Role
`Tag`	record	Immutable value object: `start`, `end`, `surface`, `id`, `type`, `output`
`buildAnalyzer()`	static method	Constructs the analyzer pipeline (see §10)
`createIndex()`	static method	Creates an in-memory `ByteBuffersDirectory`
`addData()`	static method	Writes hard-coded phrase documents via `IndexWriter`
`loadCsvData()`	static method	Appends CSV documents; reads `DATA` env var
`buildPhrases()`	static method	Returns the built-in `List<String[]>` dictionary
`buildTagger()`	static method	Orchestrates Steps 1–3 for server/MCP mode
`jsonEscape()`	static method	Escapes a string for embedding in JSON
`toJson()`	static method	Builds the response envelope (`totaltime`, `text`, `docs`)
`TextTagger`	static class	FST-backed phrase dictionary; forward-maximum-match arc walk
`TaggerServer`	static class	`com.sun.net.httpserver` HTTP wrapper; GET+POST `/tag`, `/health`
`McpServer`	static class	MCP stdio server (JSON-RPC 2.0); exposes `tag` tool to Claude Desktop
`App.main()`	entry point	Dispatches `mcp`, `serve`, `repl`, or demo mode

10. Analyzer pipeline

The same analyzer is used at both index time and query time this is critical. A mismatch causes phrases to never match.

Input text
    │
    ▼
PatternReplaceCharFilter   strips hyphens and Unicode dashes
    │                      so "xx-yy" and "xxyy" are identical
    ▼
StandardTokenizer          splits on whitespace / punctuation
    │
    ▼
LowerCaseFilter            "Apache" → "apache"
    │
    ▼
ASCIIFoldingFilter         "Zürich" → "zurich"
    │
    ▼
Analyzed token stream      used for both indexing phrases and tagging input

Hyphen variants stripped: ASCII hyphen (-), non-breaking hyphen, en-dash, em-dash, minus sign (and two additional Unicode dash characters).

11. Requirements & build

Requirement	Version
Java	17+
Maven	3.6+
Apache Lucene	9.10.0 (bundled in fat jar)

# Clone and build
git clone https://github.com/YOUR_ORG/fstguardrails.git
cd fstguardrails/java-tagger-demo
mvn package -q

# Verify
java -jar target/tagger.jar

The fat jar is ~12 MB and has zero runtime dependencies outside the JDK.

Part of the Java FST Guard Rails project.

Navigation Menu

Repository navigation

Home

Java FST Guard Rails Wiki

Table of Contents

1. What it does

2. How the FST works

3. Quick start

4. Run modes

4.1 HTTP server

4.2 MCP server (Claude Desktop)

4.3 Interactive REPL

4.4 Demo mode

5. Tag response format

6. Loading CSV dictionaries

7. Bundled dictionaries

8. Docker

9. Source layout

10. Analyzer pipeline

11. Requirements & build

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally