What's new in v0.1.0
This is the first versioned release of pg_cjk_parser. It fixes two silent data-corruption bugs in cjk_zht2zhs, adds PostgreSQL 18 support, and backports several upstream parser improvements from PG14–PG16.
Bug fixes
cjk_zht2zhs silently skipped conversions (Bug 002 — high severity)
The else branch in prsd2_zht2zhs read *cur (byte 0 of the buffer) instead of *(cur + pos) to determine the UTF-8 sequence length of the current character. When the first character and the current character had different byte lengths, pos advanced by the wrong amount, landing the scan mid-sequence and leaving Traditional Chinese characters unconverted without any error.
Example: cjk_zht2zhs('éa漢') returned 'éa漢' instead of 'éa汉'.
utf8_cjkCodePoint returned 0 for 4-byte sequences (Bug 001 — medium severity)
The 4-byte UTF-8 branch computed the correct codepoint but fell through to return 0. In practice this caused every 4-byte CJK character (CJK Extension B–F, U+20000+) to amplify Bug 002, silently skipping subsequent Traditional Chinese conversions in the same string.
Example: cjk_zht2zhs('𠀀漢𠁐漢') returned '𠀀汉𠁐漢' instead of '𠀀汉𠁐汉'.
New: PostgreSQL 18 support
lc_ctype_is_c(Oid) was removed in PostgreSQL 18. The extension now uses database_ctype_is_c on PG15+ and falls back to lc_ctype_is_c on PG11–14. CI matrix extended to PG11–18.
Upstream compatibility improvements
pg_mblen_range (PG14+): replaced pg_mblen() with pg_mblen_range() in TParserGet and the 2-gram extension in prsd2_nexttoken. Prevents a buffer overread when a multibyte sequence runs to the exact end of a string. A compile-time shim provides the same protection on PG11–13.
hlCover / TS_execute_locations (PG16+): backported the PG16 headline algorithm rewrite. On PG16 and later the extension uses TS_execute_locations() to pre-compute match positions, replacing the O(n²) recursive scan with a linear algorithm. The PG11–15 implementation is preserved under a #if PG_VERSION_NUM < 160000 guard.
Other improvements
utf8_setCjkCodePointnow raises an error instead of silently doing nothing if an unexpected codepoint is passed.- Dockerfile
AScasing fixed (was lowercaseas, causing lint warnings on every build). - CI PG13 job was running the wrong test script; corrected.
- Pre-commit hook now passes
--no-cachewhenDockerfileorMakefileis staged, preventing Docker layer cache from hiding packaging errors. - Upgrade path:
ALTER EXTENSION pg_cjk_parser UPDATE TO '0.1.0'supported for installations upgrading from 0.0.1.
Supported PostgreSQL versions
PostgreSQL 11, 12, 13, 14, 15, 16, 17, 18