Skip to content

pg_sift 4.3.4

Choose a tag to compare

@github-actions github-actions released this 13 Nov 16:46
· 51 commits to main since this release

pg_sift 4.3.4: a patch release of 4.3.3 that fixes a statement that failed with requested DSM segment size does not match size of existing segment after a package upgrade without a PostgreSQL restart, or after a changed setting. Extension SQL 4.3.4 (no new object, no format change): install the files, then ALTER EXTENSION pg_sift UPDATE in each database; a restart of the server is recommended, no longer needed for this error.

Highlights

  • Fixed: a statement failed with ERROR: requested DSM segment size does not match size of existing segment (4.x before 4.3.4, after the library was replaced under a running server, or a setting changed). The shared tables of pg_sift (the rows of joined tables and the dimension maps of the aggregate scan, the choices of the planner, the shared dictionaries, the statistics of the leaves, the write-buffer arena and the rollup state) are named segments of the server's dynamic shared memory registry. PostgreSQL keeps the name and size of the first request until the server restarts, and raises that error for a request of the same name at another size. A library replaced while the server runs keeps older and newer backends side by side: the registry still held the size an older library asked, and a newer one asked the same name at the size of its own structure (the table of joined rows is 11,544 bytes in 4.2.1 and 13,592 bytes in 4.3.0 to 4.3.3, under one name). The tables that are optional caches downgrade the error to a line in the log; the read of the rows of a joined table, the dimension maps, the write-buffer arena and the rollup state did not, so the statement failed, on and off, for the backends that connected after the install. The registry name now carries the size asked, in the one function that asks the registry (a name is never asked at two sizes: a layout of another size is a table of its own), and no other code asks the registry (a lint rule holds it). The rows of joined tables and the dimension maps are attached the way the other shared tables are: a table that cannot be had (no room in /dev/shm) leaves the statement on the path without shared tables, with a message in the log. Held by the TAP test col_dsm_registry_stale: it leaves entries of an earlier release in a running server (the names of 4.3.3 at other sizes), then runs a joined aggregate, a dimension map and a write that keeps its rows in the arena, in new backends; on 4.3.3 the three raise the error, on 4.3.4 they return the rows of the stock plan.
  • A setting that sets a size is a table of its own: sift.col_leaf_cache_kb is the one setting that decides the size of a named table (the others are fixed structures or record their size in the table itself). A size that no table has yet, set after the server started, now makes a table of that size for the backends that use it; before, such a backend logged that the statistics were not shared and went without.

Who is affected and what to do

  • Affected: every 4.x release before 4.3.4 where the library was replaced while the server ran (a package upgrade without a restart, which keeps backends of the older library in the registry), and the same releases where a size of a named table changed with a setting. A server that was restarted after each install, and never changed sift.col_leaf_cache_kb while it ran, is not affected.
  • What you saw: a statement that failed with the error above, on and off, in the new connections; the same statement ran in a connection of an older library, and again after a restart. Nothing stored is involved: no wrong answer, no repair, no REINDEX.
  • Update (the steps, in this order): install the files, then ALTER EXTENSION pg_sift UPDATE in each database (4.3.3 to 4.3.4 runs an empty script: no object changes; no format change). A restart of the server is recommended (a standby too): a backend that already loaded an older library keeps it until its connection ends, and keeps asking the registry under the name and size of that library; it cannot share a table with the new library (the names differ), so nothing is corrupted, but only after the restart is there a single library. With 4.3.4 installed the error stops in the backends that start after the install without a restart, because they ask the registry under names no older entry has; a backend of an older library that meets an entry another library made at a size of its own can still raise it until it ends.
  • What stays in memory: the registry cannot give a segment back. The entries an earlier library or size made stay until the server restarts and are never joined again: one entry for each name and size ever asked. The names are the nine tables of this release and up to 128 segments of 1 MB of the write-buffer arena; the sizes of the tables are 11 kB to 1 MB for the directories, 16 MB by default for the leaf statistics (sift.col_leaf_cache_kb), 64 MB by default for the dictionary ring (sift.col_dict_shared_cache_kb) and up to 128 MB for the arena, and only the tables that backends of the earlier library used were made. Releases do not change a size often: a server that replaces the library many times without a restart keeps one such set for each size that was asked, not one for each install. A restart gives all of it back.
  • If you cannot update yet: restart the server, which empties the registry; the error comes back only if the library is replaced under the running server again.
  • Going back: install the 4.3.3 files and restart after ALTER EXTENSION pg_sift UPDATE TO '4.3', which runs the empty script back to 4.3 (4.3.3 itself is reached by UPDATE TO '4.3.3' from there). The cause of the error is back with those libraries.

Compatibility

  • PostgreSQL 17, 18 and 19; the packages below cover all three (other builds from source with make USE_PGXS=1). A replacement of 4.3.3 with the install; extension SQL 4.3.4 (pg_sift--4.3.3--4.3.4.sql, and an empty pg_sift--4.3.4--4.3.sql back); no column format minor and no BM25 metapage minor changes. The registry names change with this release: a library of 4.3.3 or earlier and one of 4.3.4 never share a table.

Packages

x86-64 and arm64 packages for PostgreSQL 17, 18 and 19 are attached below (SHA256SUMS covers every asset); other versions build from source with make USE_PGXS=1.

  • Debian 12 and 13, Ubuntu 22.04 and 24.04 (PGDG postgresql-N, N = 17, 18 or 19): sudo apt install ./postgresql-N-pg-sift_4.3.4-1_amd64.deb (_arm64.deb on arm64)
  • RHEL, Rocky, Alma 9 and 10 (PGDG postgresqlN-server): sudo dnf install ./pg_sift_N-4.3.4-1.el9.x86_64.rpm (.aarch64.rpm on arm64)
  • Amazon Linux 2023 (the distribution's postgresqlN-server, N = 17 or 18): sudo dnf install ./pg_sift_N-4.3.4-1.amzn2023.x86_64.rpm (.aarch64.rpm on arm64)
  • Arch Linux: AUR pg_sift.

Every change: CHANGELOG.md.

SQLite extension

pg_sift for SQLite 4.3.4 (sift, a loadable extension; a separate component from the PostgreSQL extension above). Rebuilt with the PostgreSQL 4.3.4 release: no change to the SQL interface or the shared format.

  • No change: the changes of this release are in the PostgreSQL column index, which is not in this extension. No speed-up or slow-down is claimed.
  • Needs SQLite 3.38 or newer; Linux x86-64, glibc 2.34 or newer. Built on Debian 12, loaded into the SQLite of Debian 12, Ubuntu 24.04 and AlmaLinux 10 before release.
  • Storage format 2: tables of 3.4.0 to 3.7.0 open as they are, nothing to rebuild.

Download sift-sqlite-4.3.4-linux-x86_64.tar.gz (below, covered by SHA256SUMS with the packages), unpack it and .load ./sift from the sqlite3
shell (sha256sum --ignore-missing -c SHA256SUMS). Documentation: sqlite/README.md.
Every change: sqlite/CHANGELOG.md.

MariaDB extension

pg_sift for MariaDB 4.3.4 (sift, a UDF library and two SQL files; a separate component from the extensions above). Rebuilt with the PostgreSQL 4.3.4 release: no change to the SQL files.

  • No change: the changes of this release are in the PostgreSQL column index, which is not in this extension. No speed-up or slow-down is claimed.
  • One library per MariaDB series, because it calls the server's own symbols and refuses to load into another series: 10.11, 11.4, 11.8, 12.1 and 13.0; Linux x86-64. Each was built in the stock mariadb:<series> image against that server's headers and loaded into a server of the same image before release. BUILD-INFO in the tarball lists the server version and the glibc each library needs.
  • Index format 2: an index of 3.4.0 to 3.7.0 opens as it is, nothing to rebuild.

Download sift-mariadb-4.3.4-linux-x86_64.tar.gz (below, covered by SHA256SUMS with the packages), unpack it, copy lib/<series>/libsift_udf.so of your server's series
(SELECT VERSION()) into its plugin_dir, then SOURCE sql/functions.sql once per server and sql/sift.sql once per database (sha256sum --ignore-missing -c SHA256SUMS).
Documentation: mariadb/README.md.
Every change: mariadb/CHANGELOG.md.