Repository navigation
Releases: TheKnarf/pg-automerge
Release list
v0.1.0
First release: PostgreSQL 18 only, Automerge 0.12.0 (pinned). As a source
build (cargo pgrx install, or a package with mise run package) and as
two Docker images. Later versions update from it with ALTER EXTENSION pg_automerge UPDATE (see Updating).
The SQL surface:
- The
automergetype: full Automerge saves, validated and normalized on
input; text form\x+ hex (the dump format), bytea casts both ways,
and an implicit cast tojsonb, so every jsonb operator, function and
index (GIN, B-tree expressions, STORED generated columns) works on it. - Merging:
merge(doc, changes)and||for documents and bare changes,
themerge_aggaggregate; concurrency-safeUPDATE .. SET doc = merge(doc, $1); expanded in-memory values for merges in PL/pgSQL. - Reading: the jsonb mapping,
automerge_to_jsonb(doc, heads)as of
earlier heads,automerge_heads,automerge_contains. - Rich text:
automerge_spans(doc automerge, path text[]) RETURNS jsonb
andautomerge_spans(doc, path, heads text[]): a text object's
structure (text runs with their marks, and blocks), as Automerge's
JavaScriptspans()returns it; NULL if nothing is at the path, 22023
if it is not a text object, 54000 for blocks nested more than 32 levels
deep. The jsonb view never makes Automerge render blocks (a text shows
each block as U+FFFC), so a document with blocks nested hundreds of
levels deep reads as jsonb like any other, instead of overflowing the
stack inside Automerge (see
Deep blocks). - History (read-only):
automerge_changes,automerge_changes_meta,
automerge_changes_bytes,automerge_get_change,
automerge_change_count. If the owner ofautomerge_changeor
automerge_change_metaalters, drops or adds an attribute, these fail
withXX000(type automerge_change has been altered: ...). - Change notifications: the
automerge_notify()trigger (NOTIFY with the
row key and the heads of changed columns). A virtual generated column
(PG18) is refused as a key column (22023, with a HINT): a trigger
sees NULL in it. STORED generated columns work. - Memory observability:
automerge_memory_usage(), the memory
pg_automerge holds in the current backend outside Postgres' memory
contexts (whichpg_backend_memory_contextsandwork_memdo not
see): bytes held now and at peak (exact, from a counting allocator),
loaded documents alive, and the number and total time of Automerge
loads;automerge_memory_reset()restarts the peak and the load
counters, to measure one statement. Per backend; see
Monitoring. The allocator
costs a few ns per allocation (mise run bench-alloc: +6.5 ns per
allocation and free on an Atom C3758R), not measurable in
mise run bench-sql. - Settings, all superuser-only (see
Configuration):
pg_automerge.verify_writes;pg_automerge.max_load_memory, a load
memory limit: every load is priced before Automerge allocates and
refused with53400above it (measured inputs peak at 0.80 of the
estimate at most);pg_automerge.trim_threshold(default64MB): at
the end of a transaction in which pg_automerge's allocation fell by at
least this much, the memorymallockept is returned to the operating
system (malloc_trim), so a long-lived connection does not keep the
most it ever needed (-1turns it off). The library reserves the
pg_automergeprefix: a misspelled setting is an error (42602) once
it is loaded, and one frompostgresql.conf,ALTER SYSTEMor
ALTER ROLE/DATABASE .. SETis removed with a WARNING. - The extension is relocatable (
CREATE EXTENSION .. SCHEMA,ALTER EXTENSION .. SET SCHEMA) and not trusted: only a superuser installs it.
Images (built locally with mise run docker-build; see
Docker):
pg-automerge:0.1.0: the officialpostgres:18image plus the
extension, created inPOSTGRES_DBon first start
(PG_AUTOMERGE_CREATE_EXTENSION=0skips it), and acompose.yamlfor
local development.pg-automerge-cnpg:0.1.0-18-trixie: the extension's files in the
layout of a CloudNativePG image volume extension (FROM scratch), from
the same compile. List it in aCluster'sspec.postgresql.extensions
with a PostgreSQL 18 trixie operand (CNPG 1.27 or later, Kubernetes
image volumes); see
CloudNativePG.- Releases: publishing a GitHub release (tag
v<version>) runs
.github/workflows/release.yml, which tests the tag's commit like CI
(plus arm64 and the CloudNativePG end-to-end test) and only then pushes
both images, each as one amd64 + arm64 image, to
ghcr.io/theknarf/pg-automerge:<version>(andlatest) and
ghcr.io/theknarf/pg-automerge-cnpg:<version>-18-trixie, attaches the
package tarball, the image archives andSHA256SUMSto the release, and
adds the pull commands and a CNPG snippet to its notes; see
Releasing.
Notable behaviour:
- A flat (stored)
automergeargument is detoasted once and read in
place; a loaded document takes far more memory than its stored size (a
3 MB text about 400 MB), outside Postgres' accounting: size containers
and pods as Operations describes. - GIN indexes on documents: with Postgres' default
fastupdate = onthe
planner stops using them while writes keep the pending list full, so
create them withfastupdate = off(more WAL); found by the soak test
(mise run soak). - Reading documents with blocks as jsonb skips rendering the blocks
altogether, which also makes it about three times faster than
rendering them.
Known issue:
- Changes with many overlapping rich-text marks take time quadratic in
their number to apply (Automerge's mark bookkeeping), so a small
crafted change can keep a backend busy and uncancellable for minutes
while staying far belowpg_automerge.max_load_memory(32,000 marks
in 122 bytes: 3 s; a million: about 50 minutes, extrapolated). Saves
are not affected. Not priced yet: the cheap bound would refuse
ordinary rich text (see What the limit cannot
do).
Tested by mise run ci (unit, #[pg_test], regress and multi-session
suites, the upgrade test of the install script against
sql/snapshots/pg_automerge--0.1.0.sql), mise run docker-test (both
images, a 60-second soak run) and mise run cnpg-e2e (the CNPG operator
on kind). The documentation is a static site built from docs/.
Docker images
Built and tested by the release workflow from 081f7f4, for
linux/amd64 and linux/arm64:
docker pull ghcr.io/theknarf/pg-automerge:0.1.0
docker pull ghcr.io/theknarf/pg-automerge-cnpg:0.1.0-18-trixie # CloudNativePG extension image (files only)The first is the official postgres:18 image plus the extension
(Docker). The second is mounted by
CloudNativePG 1.27 or later into its own
PostgreSQL 18 trixie image through Kubernetes image volumes
(CloudNativePG):
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: automerge
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:18-minimal-trixie
storage:
size: 10Gi
postgresql:
extensions:
- name: pg-automerge
image:
reference: ghcr.io/theknarf/pg-automerge-cnpg:0.1.0-18-trixie
---
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: automerge-app
spec:
name: app
owner: app
cluster:
name: automerge
extensions:
- name: pg_automerge
version: "0.1.0"Assets
pg_automerge-0.1.0-pg18-linux-amd64.tar.gz: the extension built against the PGDG PostgreSQL 18 on
Ubuntu 24.04 (amd64);sudo tar -C / -xzf pg_automerge-0.1.0-pg18-linux-amd64.tar.gzputs it into
/usr/lib/postgresql/18/liband/usr/share/postgresql/18/extension.pg-automerge-0.1.0-linux-<arch>.tar.gz,
pg-automerge-cnpg-0.1.0-18-trixie-linux-<arch>.tar.gz: the
pushed images asdocker savearchives (docker load -i <file>).SHA256SUMS:sha256sum -c SHA256SUMS.