Skip to content

pg_sift 4.3.3

Choose a tag to compare

@github-actions github-actions released this 13 Nov 02:21
· 60 commits to main since this release

pg_sift 4.3.3: a patch release of 4.3.2 that fixes a failed merge of a sift_col index: a merge that took the maintenance lock while a rewrite of the same index was still to commit failed with sift index "<name>" is corrupted. Extension SQL 4.3.3 (no new object, no format change): install the files, restart the server (a standby too), then ALTER EXTENSION pg_sift UPDATE in each database.

Highlights

  • Fixed: a merge that ran while a rewrite that chose key_cut again was still to commit failed with sift index "<name>" is corrupted (4.0.0 to 4.3.2). A rewrite of a sift_col index (a merge that rewrites the whole index, for example sift.col_merge(index, true) or the rewrite that dead_rewrite_pct asks for) may choose key_cut again; it writes that option in its own transaction, which commits after the maintenance lock was let go. A second merge (sift.col_merge, the maintenance worker, or the merge inside VACUUM) that took the lock in that window kept the options of before the commit, read the segments the rewrite had cut by the key as if they were in the order of the TIDs, and its check stopped it: ERROR: sift index "<name>" is corrupted (Segment N is not in the order of the TIDs). A transaction that wrote an option of the index now keeps the maintenance lock to its end, and the next holder reads the options again. Held by the isolation test col_key_cut_merge_lock, which fails on 4.3.2 (the second merge reports the error) and passes on 4.3.3 (the second merge waits for the rewrite and completes).

  • Tests: the plan checks of the test suite accept the plans PostgreSQL 18.7 makes; no change to the extension.

Who is affected and what to do

  • Affected: 4.0.0 to 4.3.2 (every 4.x release before 4.3.3); 3.x does not decide key_cut again and is not affected. The window is a rewrite that changes key_cut (the option is written in a transaction that has not committed) while a second merge of the same index takes the maintenance lock: a table with writes and a merge of its index that overlaps such a rewrite. An ordinary open transaction cannot reach it.
  • What you saw: a failed call and nothing else. The merge fails before it replaces a segment, so the index is as it was; sift.col_verify passes afterwards and the same call succeeds when it is run again. A VACUUM or a maintenance worker step that met it stopped with the error and was retried by the next run. No wrong answer of a query was seen from it (see the open case below). Nothing stored is known to need repair; no REINDEX.
  • Update (the steps, in this order): install the files and restart the server (a standby too), because a backend that already loaded the old library keeps it until its connection ends; then ALTER EXTENSION pg_sift UPDATE in each database (4.3.2 to 4.3.3 runs an empty script: no object changes; no format change). A database at 4.3, 4.3.1 or 4.3.2 goes through the earlier scripts by itself.
  • If you cannot update yet: run the failed sift.col_merge or VACUUM again.
  • Going back: install the 4.3.2 files and restart after ALTER EXTENSION pg_sift UPDATE TO '4.3', which runs the empty script back to 4.3 (4.3.2 itself is reached by UPDATE TO '4.3.2' from there). The cause of the failed merge is back with those libraries.

Open case

  • Not shown to happen, not ruled out: a stale merge (one that kept the options of before the commit) whose segments cut by the key all lie in one superblock would pass the check of the order of the TIDs, and could write a misordered unit. The segments of a rewrite in that window span many superblocks, and the case could not be made to happen; the fix removes its cause, since the merge no longer takes the lock with options of before the commit.

Compatibility

  • PostgreSQL 17, 18 and 19; the packages below cover all three (other builds from source with make USE_PGXS=1). A replacement of 4.3.2 with the install and a restart; extension SQL 4.3.3 (pg_sift--4.3.2--4.3.3.sql, and an empty pg_sift--4.3.3--4.3.sql back); no column format minor and no BM25 metapage minor changes.

Packages

x86-64 and arm64 packages for PostgreSQL 17, 18 and 19 are attached below (SHA256SUMS covers every asset); other versions build from source with make USE_PGXS=1.

  • Debian 12 and 13, Ubuntu 22.04 and 24.04 (PGDG postgresql-N, N = 17, 18 or 19): sudo apt install ./postgresql-N-pg-sift_4.3.3-1_amd64.deb (_arm64.deb on arm64)
  • RHEL, Rocky, Alma 9 and 10 (PGDG postgresqlN-server): sudo dnf install ./pg_sift_N-4.3.3-1.el9.x86_64.rpm (.aarch64.rpm on arm64)
  • Amazon Linux 2023 (the distribution's postgresqlN-server, N = 17 or 18): sudo dnf install ./pg_sift_N-4.3.3-1.amzn2023.x86_64.rpm (.aarch64.rpm on arm64)
  • Arch Linux: AUR pg_sift.

Every change: CHANGELOG.md.

SQLite extension

pg_sift for SQLite 4.3.3 (sift, a loadable extension; a separate component from the PostgreSQL extension above). Rebuilt with the PostgreSQL 4.3.3 release: no change to the SQL interface or the shared format.

  • No change: the changes of this release are in the PostgreSQL column index, which is not in this extension. No speed-up or slow-down is claimed.
  • Needs SQLite 3.38 or newer; Linux x86-64, glibc 2.34 or newer. Built on Debian 12, loaded into the SQLite of Debian 12, Ubuntu 24.04 and AlmaLinux 10 before release.
  • Storage format 2: tables of 3.4.0 to 3.7.0 open as they are, nothing to rebuild.

Download sift-sqlite-4.3.3-linux-x86_64.tar.gz (below, covered by SHA256SUMS with the packages), unpack it and .load ./sift from the sqlite3
shell (sha256sum --ignore-missing -c SHA256SUMS). Documentation: sqlite/README.md.
Every change: sqlite/CHANGELOG.md.

MariaDB extension

pg_sift for MariaDB 4.3.3 (sift, a UDF library and two SQL files; a separate component from the extensions above). Rebuilt with the PostgreSQL 4.3.3 release: no change to the SQL files.

  • No change: the changes of this release are in the PostgreSQL column index, which is not in this extension. No speed-up or slow-down is claimed.
  • One library per MariaDB series, because it calls the server's own symbols and refuses to load into another series: 10.11, 11.4, 11.8, 12.1 and 13.0; Linux x86-64. Each was built in the stock mariadb:<series> image against that server's headers and loaded into a server of the same image before release. BUILD-INFO in the tarball lists the server version and the glibc each library needs.
  • Index format 2: an index of 3.4.0 to 3.7.0 opens as it is, nothing to rebuild.

Download sift-mariadb-4.3.3-linux-x86_64.tar.gz (below, covered by SHA256SUMS with the packages), unpack it, copy lib/<series>/libsift_udf.so of your server's series
(SELECT VERSION()) into its plugin_dir, then SOURCE sql/functions.sql once per server and sql/sift.sql once per database (sha256sum --ignore-missing -c SHA256SUMS).
Documentation: mariadb/README.md.
Every change: mariadb/CHANGELOG.md.