Skip to content

Text Tagging

Jump to bottom
Steve edited this page May 7, 2026 · 1 revision

Named Entity Extraction with the Lucene Text Tagger

This page describes how to use the Lucene Text Tagger (the FST-based tagger underlying solr-text-tagger) for Named Entity Extraction (NER), and how to expand a dataset with alternative surface forms so the tagger can match real-world text robustly.

Table of Contents

Overview

The Lucene Text Tagger matches input text against a pre-built dictionary of entities. To make matching robust, each entity in your dataset should be indexed under multiple surface forms so the tagger can recognize variations as they appear in real text.

1. Model Each Entity as canonical_id → [surface forms]

Your dataset becomes a mapping from a stable identifier to every way that entity might appear in text:

Canonical ID Canonical Name Surface Forms
E001 International Business Machines IBM, I.B.M., Intl. Business Machines, international business machines
E002 New York City NYC, New York, N.Y.C., new-york city
The tagger indexes every surface form, but each one points back to the same canonical ID.

2. Generating Alternative Forms

For each canonical value, run it through a pipeline of generators:

  • Case variants — lowercase, uppercase, title case.
  • Acronym expansion / contraction — International Business Machines ↔ IBM; build acronyms from capitalized tokens.
  • Punctuation variants — strip or insert dots, hyphens, apostrophes (I.B.M., IBM, I B M).
  • Abbreviation substitution — a small map: Street ↔ St., Corporation ↔ Corp., Doctor ↔ Dr., and ↔ &.
  • Stopword removal — drop the, of, inc, ltd so The Coca-Cola Company matches Coca-Cola.
  • Token reordering — Smith, John ↔ John Smith for person names.
  • Diacritic folding — Zürich → Zurich (Lucene's ASCIIFoldingFilter).
  • Common misspellings / typos — generated via edit-distance or a curated list.
  • Plural / singular — light stemming for product or category names.
  • Translations / aliases — pulled from a knowledge base such as Wikidata's also known as.

3. Lean on Lucene Analyzers Where Possible

Many of the transformations above — lowercasing, ASCII folding, stopword removal, light stemming — can be handled by configuring the analyzer on the tagger's field rather than by exploding the dictionary. Reserve explicit alternative-form generation for transformations the analyzer can't do:

  • Acronyms
  • Abbreviations
  • Token reorderings
  • Aliases / translations

4. Index and Tag

  1. Push every (surface_form, canonical_id) pair into the Lucene index used by the tagger.
  2. At query time, the tagger emits spans annotated with the canonical ID.
  3. Downstream code always sees the normalized entity, regardless of which variant appeared in the source text.

5. Canonical ID as a Semantic Token

The canonical_id (e.g. E001, E002) is not just a database key — it is a semantic token: a stable, language-independent symbol that represents the concept of the entity, not any particular way of writing it. Think of it as the difference between a word and a meaning:

  • IBM, I.B.M., International Business Machines, Intl. Business Machines are all lexical forms — different strings humans use.
  • E001 is the semantic token — it denotes "the company IBM, the concept" regardless of how it was spelled in the source text.

Why It Matters

  1. Many-to-one mapping — every surface form indexed in Lucene points back to the same semantic token. The tagger collapses lexical variation into meaning.
  2. Form-agnostic downstream code — once the tagger emits E001, the rest of your pipeline (search, analytics, knowledge graph, dedup, entity linking) operates on concepts, not strings. You never have to ask "is IBM the same as I.B.M.?" again — that question was answered at index time.
  3. Join key to a knowledge base — the canonical ID is what you use to attach metadata (industry, location, Wikidata QID, embeddings, etc.) to the entity. The string IBM can't carry that — E001 can.
  4. Cross-lingual and cross-modal recognition — IBM, 国际商业机器, and Big Blue can all map to E001. The semantic token is the anchor; surface forms are interchangeable skins over it.
  5. Stability across dataset edits — adding IBM Corp. as a new alt form, or fixing a typo in the canonical name, does not change the ID. Anything previously tagged stays valid.

Mental Model

   surface forms (lexical)              semantic token        knowledge
   ────────────────────────              ──────────────       ──────────
   "IBM"                ─┐
   "I.B.M."             ─┤
   "International ..."  ─┼──────►          E001       ──────►  { industry: tech,
   "Intl. Business ..." ─┤                                        founded: 1911,
   "Big Blue"           ─┘                                        wikidata: Q37156 }

Summary

The Lucene Text Tagger's job is to bridge the left column (messy real-world text) to the middle column (clean semantic tokens). Generating alternative forms is how you widen that bridge — and treating the canonical ID as a semantic token is what makes everything downstream simpler, more stable, and ready for knowledge-base integration.

Clone this wiki locally