pg_fts 1.8.3

1.8.3 - 2026-09-18

Correctness release: two deadlocks that shipped in every prior version, found while
fixing the bulk-ingest bloat -- which is also fixed. No on-disk format change; no
REINDEX required. Upgrade recommended for any index taking concurrent inserts.

Fixed

• Deadlock: concurrent insert + VACUUM on the same index. The insert path's
  "directory is full, merge to make room" (bm25_add_segment_with_room) was the only
  merger that did not take the maintenance mutex, so it could run concurrently with
  autovacuum's merge. Under the old extend-only allocation the two never touched the same
  block and the race was merely wasteful; once merges reuse pages (below) both mergers
  handed out the same block and deadlocked on its buffer lock. Confirmed on v1.8.2 as
  shipped: the same row-per-transaction ingest with autovacuum on deadlocks at row 1,674
  with no other change. Every merger now takes the mutex, and both merge entry points
  elog(ERROR) if entered without it -- which immediately caught a second unprotected
  site, CREATE INDEX CONCURRENTLY's finalize (holds only ShareUpdateExclusiveLock), now
  also covered.

• Deadlock: a live page handed out as merge output. bm25_page_recyclable() returned
  true for a page without the BM25_FREED flag ("older free, or in-use race"). The FSM
  records free space, not liveness, so a partially filled live posting page qualified
  -- instrumented live: blk=65 flags=0x4 nextblk=66, mid-chain -- and was written as
  output while the same merge held it pinned as input. Self-deadlock. An un-flagged page
  is now treated as live and never handed out, checked before the AccessExclusiveLock
  bypass as well. Pages freed by builds predating the flag are reclaimed by tail
  truncation instead of reuse.

• Bulk-ingest growth eliminated on row-per-transaction ingest. Root cause: the merge
  allocated EXTEND_ONLY for its whole loop and so never consulted the free list -- not
  for pages freed by this call (the hazard it guards) and not for pages freed by
  previous calls (safe, and nearly all of them). At high terms-per-document every insert
  triggers a merge, so the merge was the dominant writer and its freed pages were reachable
  only by fts_vacuum. New BM25_ALLOC_SNAPSHOT mode gathers the free list once at
  entry, before this call frees anything, hands out only from that snapshot, and never
  re-consults the live FSM -- so the recycle-race guard holds by construction and earlier
  frees are reused.

  Measured at field shape (40k docs at 1,660 terms/doc, then row-per-transaction churn,
  autovacuum on, nothing manual): v1.8.2 grew 1,823 -> 4,975 -> 8,383 MB then deadlocked;
  1.8.3 held 1,823 -> 1,823 -> 1,823 -> 1,832 -> 1,831 -> 1,823 -> 1,823 MB over 30,000
  documents. A 100,000-document run ended at 1,875 MB with fts_vacuum finding nothing
  to reclaim -- the resident size is the live size. Parity vs regex exact throughout.

Known issue (narrowed)

• A single very large INSERT ... SELECT of oversized rows still bloats until
  fts_vacuum. Inside one statement the freeing transaction is itself the oldest
  snapshot, so no page it frees can pass the recyclability horizon until it ends; no
  allocation policy changes that. Modelled the alternative (fewer, larger merges inside a
  long statement): the leveled merge already amortises to log_F(N) rewrites, so it is worth
  ~2x, not the 8x it looks like. fts_vacuum after a bulk load remains the guidance for
  that one shape; it takes seconds.

Tests

• t/007_segment_cap.pl runs repeated VACUUMs concurrently with its four
  directory-filling inserters, under a 300 s deadline. Verified red on v1.8.2's
  pg_fts_am.c and green on 1.8.3 with the harness restored between runs. Two harness
  defects fixed on the way: pumping one IPC::Run handle at a time starved the others
  (mimicking the deadlock on correct code -- the wait events said ClientRead), and psql
  fed from a scalar needs an explicit \q or completion is unobservable.

Retracted

• 1.7.2's diagnosis that the bloat mechanism was "freed pages fail the recyclability XID
  gate in the inserting transaction" was half right: that binds only inside a single
  multi-row statement. On row-per-transaction ingest -- the field's live shape -- the
  cause was the extend-only merge, and a fresh transaction per row changed the result by
  only 1.4x until that was fixed. Details and the falsified prediction that exposed it:
  bench/RESULTS_I1_2026-09-18.md.

Full Changelog: v1.8.2...v1.8.3

Full Changelog: v1.8.2...v1.8.3

pg_fts 1.8.2

1.8.2 - 2026-09-17

Code-quality release from the fresh-eyes review (REVIEW_2026-09-17.md), plus one real
fix found while doing it. No on-disk format change; no REINDEX required.

Fixed

• A ninth unvalidated pd_lower read, in the trigram blob reader
  (pg_fts_trgm_index.c). It computed a memcpy length as pd_lower - contents_offset;
  on a corrupt or recycled page whose pd_lower is below the contents offset that wraps
  to a huge Size before the Min() clamps it, and the copy runs past the page. Same
  defect class as the 1.7.0 P0 that made an index permanently unvacuumable. Now routed
  through bm25_page_data_end(), as are the 14 remaining reads in pg_fts_am_scan.c.
  1.7.1 claimed to have fixed "all eight" sites; this makes it nine, and the count is now
  every read of pd_lower in the tree.

Changed (behaviour-preserving; full gate green after each)

• Allocator state is a scoped struct, and misuse is a hard error. The four file-scope
  globals (bm25_lowfree, _n, _i, bm25_alloc_extend_only) became one
  BM25AllocCtx reachable only through bm25_alloc_scope_enter() /
  bm25_alloc_scope_exit(), which nest by returning the previous context. In the 1.7.1
  work, code read those globals without owning them and handed out garbage block numbers
  ("could not open file ... target block 829694001: previous segment is only 527 blocks");
  only t/007_segment_cap.pl noticed. bm25_new_buffer() now elog(ERROR)s on that
  condition. Deliberately an elog, not an Assert -- the release gate is not a cassert
  build, and a check that fires only in a build nobody ships is documentation, not
  enforcement.
• bm25_collect_matches split, 412 -> 226 lines. The per-segment loop body is now
  bm25_collect_segment() (returns SEG_RESTART for the positional-phrase fallback the
  loop used to express as s = -1; continue) and the pending-list walk is
  bm25_collect_pending(). Shared state travels in a BM25CollectCtx.
• The single translation unit is now a documented decision, not an accident.
  pg_fts_am.c #includes pg_fts_am_scan.c, pg_fts_trgm_index.c and pg_fts_lev.c.
  Splitting was costed: 15 statics would go extern, ~10 shared types would move into the
  on-disk-format header, and the hot-path static inline helpers inside the 45%/37%
  common-term profile would stop inlining (PGXS does not use LTO). In return, three .o
  files and no behaviour change. Kept; the reasoning is at the #include site and in both
  included files' headers, with the two prerequisites if separate compilation is ever
  needed.

Full Changelog: v1.8.1...v1.8.2

Full Changelog: v1.8.1...v1.8.2

Full Changelog: v1.8.1...v1.8.2

pg_fts 1.8.1

1.8.1 - 2026-09-17

Counting-path work from the TIN feasibility review. No on-disk format change; no REINDEX
required.

Added

• Ten gate-refusal tests for the single-term count(*) fast path. That fast path
  (answering from the dictionary's df with no posting decode and no heap access) has
  existed since the COUNT pushdown work, but shipped with a single positive test and nothing
  proving its gates actually refuse — the shape where a missed gate silently returns a
  plausible wrong count. Each case is now compared against a ground truth computed
  without the index: prefix, conjunction, disjunction, negation, unmerged pending documents,
  tombstones, and a not-all-visible heap all fall back and agree exactly, and the fast path
  is confirmed to resume after VACUUM.

Changed

• fts_count() and the COUNT(*) pushdown now consult the visibility map once per run of
  matches on the same heap page instead of once per matching tuple, and create one tuple
  slot per call instead of one per probed tuple. Safe because matches arrive sorted and
  de-duplicated and a docid is block × MaxHeapTuplesPerPage + offset, so matches on a page
  are contiguous; the visibility map is page-granular.

  Measured effect: none significant (~1–2%, inside run-to-run noise). An apparent 8.5%
  improvement came from a single high outlier in the baseline; repeated same-arm runs
  overlap (base 397.3–408.2 ms, fix 397.4–402.7 ms). The loop is dominated by
  table_index_fetch_tuple heap probes, not by visibility-map lookups. Counts are
  identical in both arms. Shipped as a code-quality change, not a performance feature.

Documentation

• README and the SGML manual now describe when the df fast count applies and, more
  importantly, when it refuses.
• bench/RESULTS_ABC_2026-09-17.md records the measurements, including the two corrections
  above; bench/NOTE_TIN_FEASIBILITY_2026-09-14.md is annotated with the outcome. A third
  proposed item (copying posting bytes verbatim during a merge) was withdrawn after
  reading the merge path: it decodes through the build hash table and re-encodes at flush,
  so there is no byte-stream splice point.

Full Changelog: v1.8.0...v1.8.1

Full Changelog: v1.8.0...v1.8.1

pg_fts 1.8.0

1.8.0 - 2026-09-14

Query parsing fix with a behaviour change. No on-disk format change; no REINDEX
required — stored data was never affected.

Fixed

• -, . and / inside a word are terms, not operators. Reported from the field:

  query	before	after
  pkg-config	('pkg' & !'config')	'pkg-config'
  install-info	('install' & !'info')	'install-info'
  foo/bar	'foo' (rest swallowed)	'foo/bar'
  python3.14	('python3' & '14')	'python3.14'

  The hyphen case was the damaging one: the ! clause actively excluded the documents
  being searched for, so searching pkg-config returned everything except
  pkg-config, and install-info matched 1 row instead of 10. / was worse in a
  different way — it opened a regex and swallowed the remainder of the query. As the
  reporter put it, this is a worse failure mode than operator injection: injection raises
  a visible error, this silently returns a different, wrong answer.

  The separator set is not a guess — it is what the document analyzer already joins,
  verified against to_ftsdoc('simple', 'a-b c/d e.f g_h i+j'), which yields
  'a-b' 'a' 'b' 'c/d' 'e.f' 'g' 'h' 'i' 'j': -, . and / stay inside a token while
  _ and + split. PostgreSQL's own parser agrees, classifying them asciihword,
  file and file. So the query lexer was the only side that disagreed, and the tokens
  needed to match these queries were already stored.

Behaviour change

• A - between two word characters no longer negates. a -b still excludes b
  (prefix position), and !b is unchanged, but an application relying on a-b meaning
  "a AND NOT b" must now write a !b or a - b.
• c++ and gtk+ still lex to 'c' and 'gtk'. A trailing separator is dropped, which
  is what to_tsvector and our own document analyzer do, so this is parity rather than a
  bug — noted because the original report listed it alongside the others.

Regression cases covering all four inputs, plus prefix negation, leading -, and
/regex/, are pinned in sql/pg_fts.sql.

Full Changelog: v1.7.2...v1.8.0

Full Changelog: v1.7.2...v1.8.0

pg_fts 1.7.2

1.7.2 - 2026-09-14

Measured the cause of the bloat known issue and removed ~31% of it. No on-disk format
change; no REINDEX required.

Fixed

• Bulk ingest grows the index ~31% less. At high terms-per-document every document
  exceeds one pending page, so each one mints a one-document segment — and the
  insert-time merge then folded it in immediately, rewriting a whole level-0 run for every
  single document. Measured: 23–30 index pages extended per document (linear) against
  roughly 2 pages of actual postings, a ~12–15× write amplification with page reuse at
  ~0.3%.

  The merge is now gated on there being BM25_MERGE_FANOUT small runs waiting. Below that
  threshold the leveled compactor would find no level over capacity and do nothing anyway,
  so this skips work without changing behaviour. Over six 5,000-document batches the index
  peaked at 21,874 MB instead of 31,537 MB, and the size after one fts_vacuum is
  byte-identical (124 MB).

  The segment-directory bound this protects was re-verified under the worst case for
  segment minting — one row per transaction, 4,000 transactions — reaching a maximum of
  15 segments against the hard cap of 128. t/007_segment_cap.pl now asserts <= 64
  rather than <= 128, since a bound at the cap only fails once the index is already in
  the state that motivated the eager merge (a field deployment went 8 → 128 segments in
  ~1 h and could then neither merge nor VACUUM).

Known issues

• The bloat is reduced, not eliminated. Growth is still ~3.7 GB per 5,000 documents at
  field shape, and fts_vacuum after bulk ingest is still recommended (it is fast — tens of
  seconds for millions of pages — and recovers the space completely).

  The mechanism is now measured rather than guessed: freed pages are found and then
  rejected by bm25_page_recyclable(), because they were freed by the inserting
  transaction itself and GlobalVisCheckRemovableXid() cannot yet clear them
  (norecyc=3,169 of 5,924 allocations). That gate is correct and must stand — bypassing it
  previously corrupted a concurrent reader. So in-transaction reuse is impossible by
  construction, and fully fixing this means moving the merge out of the inserting
  transaction: a design change, not a point-release edit.

Measurements: bench/RESULTS_KNOWN_ISSUES_2026-09-14.md.

Full Changelog: v1.7.1...v1.7.2

Full Changelog: v1.7.1...v1.7.2

pg_fts 1.7.1

1.7.1 - 2026-09-14

Follow-up to 1.7.0's page-corruption fix, plus a retraction of one of 1.7.0's known
issues. No on-disk format change; no REINDEX required.

Fixed

• The pd_lower bounds guard is now applied at every page-read site. 1.7.0 fixed the
  dict walk in merge_source_load_page, where an unvalidated pd_lower made the merge
  request an impossible allocation and left the index permanently unvacuumable. Auditing the
  siblings found eight sites forming page + pd_lower from unvalidated on-page data —
  including both walks in bm25_free_segment and the doclen and posting readers. All now
  route through one helper, bm25_page_data_end(), which validates in the integer domain
  (forming the pointer at all is undefined behaviour for an absurd value) and returns an
  empty range for anything out of bounds, so a caller degrades to "this page has nothing to
  read" rather than walking off the page. 1.7.0 fixed one instance of this defect; this
  fixes the class.

Retracted

• 1.7.0's "bm25_free_page emits one WAL record per page" known issue was wrong.
  Measured directly: fts_vacuum freed 8,686,917 pages in 46 seconds — 0.005 ms/page,
  roughly 2,800× cheaper than the ~14 ms/page I published, and a second run confirmed it
  (7,912,288 pages in 33 s). There is no per-page WAL problem and no WAL batching is needed.
  My figure came from a 113-minute run on an index that had already hit the 1.7.0
  allocation bug repeatedly; gdb showed the backend inside bm25_free_page and I turned
  "where it is" into "why it is slow". A stack sample gives a location, not a bottleneck.

Known issues

• The transient bloat spike is confirmed and larger than reported: ~210×, not 45×.
  Measured at field shape (1,660 terms/doc), inserting 5,000 documents at a time with no
  maintenance: the index grows ~5.6 GB per batch with nsegments pinned at 8, reaching
  61,814 MB after ten batches, and a single fts_vacuum returns it to 236 MB. The space
  is freed-but-never-reused, not live.

  The cause is not yet known. Four hypotheses were eliminated: one-doc segment
  accumulation (impossible — BM25_MAX_SEGMENTS is 128 and the insert path forces a merge),
  128-segment cycling (nsegments sits at 8), freed pages failing the recyclability XID gate
  (instrumentation showed the free-list scan is never reached: probe=0 reject=0), and
  loop-wide bm25_alloc_extend_only (scoping it per merge produced byte-identical growth
  — reverted rather than shipped as a fix). The next step is a counter on each of
  bm25_new_buffer's three outcomes rather than another hypothesis.

  Practical guidance unchanged: run fts_vacuum after bulk ingest. It is fast (tens of
  seconds for millions of pages) and recovers the space completely.

Details and measurements: bench/RESULTS_KNOWN_ISSUES_2026-09-14.md.

Full Changelog: v1.7.0...v1.7.1

pg_fts 1.7.0

1.7.0 - 2026-09-13

Two field-blocking fixes found by reproducing the reported ~2.87M-doc email-body index
shape. No on-disk format change; no REINDEX required.

Fixed

• An index could become permanently unvacuumable. The dict-page walk in
  merge_source_load_page took its end bound from the page's pd_lower with no
  validation, and stepped by an untrusted termlen. On a recycled or malformed page the
  walk ran past the page and counted garbage entries, so the caller's doubling asked for an
  impossible allocation:

  ERROR:  invalid memory alloc request size 3406063183
  

  Because this runs under bm25_merge_segments_streaming, it failed every merge, every
  autovacuum cleanup, and fts_vacuum — the index could never be vacuumed or reclaimed
  again. This is the likely cause of the reported "VACUUM/merge do not reclaim bloat".
  Isolated with gdb; both the counting and filling walks are now bounds-checked, and
  pd_lower is validated as an integer before any pointer is formed from it (forming
  page + pd_lower for a corrupt value is itself undefined behaviour — the new fuzz target
  for this loop caught that with UBSan).

• Huge-allocation gaps in bm25_doclens_load's resident docid array and bulkdelete's
  carry/newdead tombstone arrays, which used plain palloc/repalloc and so failed the
  same way on a large or delete-heavy index. The FTS_ALLOC_MAYBE_HUGE macros already
  existed for the per-term posting arrays; these sites were missed.

• Index cleanup no longer grows the index. bm25_vacuum_compact now skips a compaction
  pass when its free space is not yet reusable: bm25_page_recyclable() gates on
  GlobalVisCheckRemovableXid(), so pages freed by the same cleanup's merge are all
  rejected, and the pass would relocate live data upward while reclaiming nothing. Measured
  before: 35 → 52 → 69 MB across three cleanups with no rows added. After: flat.

Added

• Fuzz target for the dict-page walk (test/fuzz/fuzz_block.c), asserting the walk stays
  inside the page and can never report more entries than a page can physically hold, for
  arbitrary page bytes and arbitrary pd_lower.
• t/010_vacuum_delete_heavy.pl now asserts no growth across repeated cleanups, and
  additionally that cleanup still reclaims after deletes.

Known issues

• A transient bloat spike at nsegments=8, reproduced at field shape: an index went
  299 MB → 66,796 MB → 1,016 MB across three churn rounds, settling at 1,480 MB. The
  index is not permanently bloated — it inflates ~45× while segments accumulate and
  collapses once merges catch up, so an index sampled during that window looks like
  unbounded bloat. Not yet fixed.
• bm25_free_page emits one WAL record per page. fts_vacuum on a 3.8 GB index ran
  113+ minutes without finishing (progressing, not hung): ~489k pages × a full
  GenericXLog delta each. Needs WAL batching. Not yet fixed.

Both are documented with reproductions in bench/RESULTS_FIELDSHAPE_2026-09-13.md.

Documentation

• README and the SGML manual no longer recommend scheduling a periodic fts_vacuum.
  Measured at 1M docs with autovacuum on and no manual maintenance: flat at 511 MB over
  five cleanups, flat at 875 MB over six churn rounds, and 875 → 106 MB after deleting half
  the table, with results exact throughout.

Full Changelog: v1.6.1...v1.7.0

Full Changelog: v1.6.1...v1.7.0

v1.6.1

Full Changelog: v1.6.0...v1.6.1

v1.6.0

Full Changelog: v1.5.10...v1.6.0

v1.5.10

Full Changelog: v1.5.9...v1.5.10