pg_tokenizer 1.0.0

pg_tokenizer 1.0.0 is the first release maintained by PGSTY. It builds with pgrx 0.19.3 for PostgreSQL 14–18 on Linux x86_64 and aarch64.

Highlights

• Native full-text search: analyze_to_tsvector and tokenize_to_tsvector return native tsvector values with consecutive positions. tokenize() still returns INT[].
• Custom vocabularies: optional corpus filters, schema-qualified source tables, and vocabulary writers that take a per-model lock only when they add tokens.
• Privileges: the database owner can manage definitions; the cluster-wide preload list stays administrator-only.
• Definition caches: create/drop operations invalidate other backends at commit; rollbacks only clean up locally; objects loaded under an older snapshot are not reused in later statements.
• Preloading: preloaded models survive cache invalidation, and broken preload entries are skipped with a WARNING instead of preventing startup.
• Logical backups: pg_dump includes analyzer, tokenizer, model, synonym, and user stopword definitions with their vocabulary tables.
• Text analysis: ngram counts Unicode characters, so CJK text and emoji work and tokens shorter than min_gram produce no n-grams; pg_dict calls dictionaries the way ts_lexize does.
• Dependencies: Lindera 6.2, Jieba 0.11, Hugging Face tokenizers 0.23, and Snowball stemmers from pgsty/rust-stemmers v1.2.1, whose stemming output is unchanged.

Upgrading

Install the 1.0.0 library and SQL files, keep pg_tokenizer in shared_preload_libraries, restart PostgreSQL, then run in each database:

ALTER EXTENSION pg_tokenizer UPDATE TO '1.0.0';

Upgrade paths are provided from 0.1.0 and 0.1.1. They keep catalog rows, vocabulary IDs, and the original six-argument helper. Read the compatibility changes first:

• apply_text_analyzer is now STABLE; the upgrade refuses to run while an index, generated column, partition key, or statistics object depends on it.
• Lindera moves to 6.2 and the embedded IPADIC dictionary changes, so stored Lindera token IDs must be regenerated.
• The built-in lucene_english, nltk_english, and iso_english stopword lists can no longer be dropped or replaced.
• A matching custom-model row with a NULL source now gets a NULL embedding.

Packages

.deb and .zip packages for PostgreSQL 14–18 on amd64/x86_64 and arm64/aarch64, built with lindera-ipadic. See the installation guide.