Repository navigation
Mason v0.17.0
Two fixes a real deployment met come first. Several first syncs on a small machine could make the synced data
unusable, so that every table's sync failed and every restart failed again: a first sync no longer can, and a run
already stuck that way recovers when it starts on this release, with nothing lost. A run whose synced data met a fatal
error (a full disk, an I/O error) stayed "running" while it answered nothing, until it was restarted by hand: it now
ends as a crash does, its cause said, and is started again. The web page's Tables now choose how each table is kept
current: Off, a poll, or Live (logical replication) where your database is ready for it. mason wait waits for a
table's catch-up poll, mason share opens with the URL to paste, and the card of a user just added has a Copy on every
field. Every project has something to do first: its pin moves. The first fix has a cost on disk until a future release
compacts tables: watch the disk under .mason/ ("Before you upgrade").
Before you upgrade
- Move the pin:
version: 0.17.0inmason/mason.yml. A project pinned to 0.16.0 is refused by the 0.17.0 command
and image, with the way out named: before 1.0 they run a project pinned to their own minor version only, at their own
patch or an older one. Set the pin and start 0.17.0 in the same step. - On a hosted replica, upgrade the replica before you deploy a project pinned to 0.17.0.
mason deployasks the
replica's Mason version first and refuses a project whose pin the replica would not run, naming both versions and the
ways out;mason replica upgrade <org>/<name>is one of them, where the service offers it. - Watch the disk under
.mason/. The synced data now grows on disk where it used to be compacted, and tables kept
current by the poll or the stream get slower to read on a busy server ("What was wrong and is now right" gives the
numbers). To compact a table, sync it again whole. With Mason running, runmason remove <table>. Wait untilmason logsshowsdropped <table>'s synced data, the table named with its schema (dropped public.orders's synced data), then runmason add <table>. The table is not served in between. If you
add it back while Mason is stopped, or before that line appears, the add cancels the remove and nothing is compacted.
The file does not shrink, but later writes reuse the space.mason syncdoes not compact a table. - If anything reads
tables[].sync.statefrom--json(mason status,mason wait), handle the new state
catching-up: a table served and being caught up by one poll, which wassyncingbefore ("What changes"). mason add <table> --sync replicationwhile Mason is up now reads your database's checks for logical replication
first (as the web page's Live does, below), in a run that already replicates other tables too, and refuses the
change, writing nothing, when one fails, naming the failing check (replication role, for instance). The web page's
Sync choice shows each failing check with the SQL or setting that fixes it. A script
that switches tables to replication may now be refused at the add, where the edit used to be written and the table
then failed in the run.
New
- The web page's Tables choose how each table is kept current: every added table says its sync on its line (Off,
Every 5 minutes, Live (replication)), and where the page may edit the sources file it opens a choice — Off (the
default), a poll every 5 minutes, 15 minutes, hour or day, each saying it scans the whole table on your database at
that interval (a table over a million rows says what each poll costs and asks before saving), or Live, logical
replication. Live can be chosen only when your database is ready for it: otherwise it is shown greyed out with each
check that fails — no primary key,wal_level, a user without REPLICATION, no free replication slot or WAL sender, a
read replica before Postgres 16, the table missing from your publication — and the SQL or setting that fixes it. The
checks read your database's settings and catalog only, never a row, when the choice opens and again when it is saved,
so a database that changed in between is refused. Saving writes the table'ssync:line in the sources file, as
mason add <table> --sync …does. Where the project's files are read-only, the page shows the sync without a choice.
What changes
mason waitnow waits for a table's catch-up poll (after its first sync, a sync of it again whole, or a change
of itssync:toreplication). A first sync reads the table as of its start, then catches it up by one poll for
what changed during the read, async: offtable too; on a large table that poll is a scan of its own.mason wait
said Ready while it ran, so a tool started on Ready read data as of the read's start, and the table's line said "NOT
kept current" with a yellow bar. Now the table's line says it is being caught up by one poll, its bar at work, and
mason waitis ready once that poll is over (a script waiting on it may wait that much longer);mason status, the
web page and EXPLAIN say the same. In--jsona table in its catch-up hassync.statecatching-up(it was
syncing), andreadinesslists it underwaiting.mason shareopens with the URL to paste: its first line is nowTry mason with your favorite postgres client: postgres://<user>:<password>@<host>:<port>/<database>?sslmode=require, the share's own login and address, so a
client that takes a connection string connects from that one line. The host, port, database, user and password rows
below it stay, for a tool that asks for each field, andmason share --jsoncarries the same URL asshare.url.
The password is still made for this run alone and kept nowhere.- The web page's card for a user just added has a Copy on every field: URL, User, Password, Database, Host, Port
and SSL mode, so each can be pasted into a BI tool's connection form one by one. The card no longer shows a psql
line: paste the URL afterpsql.mason user add --jsonkeeps itspsqlfield. mason user add, beside amason startrun on this machine's own address, now ends its tips with the way to a
tool outside this machine:mason shareopens an address on the internet, with a login and password of its own
(not the new user's), for as long as it runs. It is not said besidemason serve, nor with--replica, nor while
Mason is not running.--jsonis unchanged.
What was wrong and is now right
-
A first sync of a table of wide values on a small machine could stop all of Mason. After the sync read the
table, the synced data's own checkpoint could fail when it reorganised that table, withfailed to offload data block …orfailed to pin block …in the log. That made the synced data unusable: every other table's sync failed
withdatabase has been invalidated, and every restart failed again in the same place. Mason now never reorganises
a table's storage that way, so the sync finishes and the other tables sync. A run already stuck like this recovers
when it starts on this release; nothing is lost.What this costs until a future release compacts tables: the synced data grows on disk, and tables kept current
by the poll or the stream get slower to read on a busy server.- A table keeps the space of every row the poll or the stream updates or deletes. A table that had every row
updated three times over took 2.1 times the disk and scanned in 1.7 times the time. - Every table written to between two of the synced data's checkpoints keeps a separate block of at least 256 KiB.
A checkpoint comes about every 16 MiB written and at every stop. In measurements, small appends with 2,000
checkpoints took a 1 M-row table from 31 MiB to 536 MiB, and its count from 1.2 ms to 2.8 ms. 25 restarts grew a
project of 8 polled tables by 50 MiB, where it grew by 1 MiB before.
Meanwhile, watch the disk under
.mason/. To compact a table, sync it again whole. With Mason running, runmason remove <table>. Wait untilmason logsshowsdropped <table>'s synced data, the table named with its schema (dropped public.orders's synced data), then runmason add <table>. The
table is not served in between. If you add it back while Mason is stopped, or before that line appears, the add
cancels the remove and nothing is compacted. The file does not shrink, but later writes reuse the space.mason sync
does not compact a table. EXPLAIN's "about N rows" also counts rows updated or deleted since the table's last sync
whole. - A table keeps the space of every row the poll or the stream updates or deletes. A table that had every row
-
A run whose synced data met a fatal error stayed "running" and answered nothing. When a write to the synced data
fails in a way it cannot recover from inside the run (a checkpoint on a full disk, an I/O error), every statement
was refused withXX000 … database has been invalidated,mason statusstill saidrunning, the request log
stopped, and nothing restarted it. Now the run ends as a crash does, with the cause and the way out in its log:
sessions are told57P02and to reconnect,mason start's supervisor starts it again (mason statusshows the
restart and its reason), andmason serveexits 1,/healthzsayingfaileduntil it does, for the platform to
start it again. Starting again opens the synced data again; nothing synced is lost. When the restart meets the same
cause (the disk still full), the supervisor says it is the same as the last restart, and gives up after five
restarts in ten minutes as for any crash; a full disk that stops the synced data from opening is said as the disk's,
never withmason destroyas the way out. -
A table of large values could fail its first sync under a small
MASON_DUCKDB_MEMORY, now and then, with
Failed to commit: could not allocate block of size …inmason statusand the log, and wait for its retry (at
least 5 minutes later). When a chunk of the first sync runs out of memory as it commits, it is now rolled back and
the sync reads on one chunk at a time, smaller, as it already did when the out-of-memory came in other words. -
mason add <table> --sync …refused a table added withkey:(one with no primary key) as "cannot be added: no
primary key". It now changes the table'ssync:and keeps its key; the web page's sync choice does the same.
Known
- The synced data is not compacted on its own yet: a table kept current by the poll or the stream keeps the space
of the rows it updated or deleted until it is synced again whole ("Before you upgrade" says how). Compaction is the
next release's work. - A generated table can't yet hold a numeric that divides (an average is one); is a variance or a standard deviation, which Postgres computes exactly at decimals it picks per value, rounded, cast to numeric(p,s) or not; is a sqrt, exp, ln, log or power, whose decimals Postgres picks per value, rounded, cast to numeric(p,s) or not, whatever the decimals of what it is computed from; is a JSON number's text cast to numeric with no scale (cast to numeric(p,s), it is stored); has more than 18 decimals, at the end or on the way; sums values whose decimals the rows decide; is a CASE, GREATEST, LEAST or COALESCE of values with differing decimals; or is cast from a double; nor a domain or a composite, and
kind: liveis refused. Each is refused by file and key before anything is built, said as Postgres types the value (a sqrt or an average of a double is a double, so cast to numeric it is a cast from a double), wherever it sits in the column (under an operator, a coalesce, a round, a cast or an aggregate, or in a scalar subquery, aWITHquery, a subquery inFROMor aUNION's arm), with the way out your own SQL with the cast moved and the aggregate kept (sum(sqrt(x)::double precision)), or theWITHquery, subquery orUNIONthe cast belongs in, given whole. Each of these keeps its reason read back from its text (::text,::varchar,|| '', a jsonb's->>, aWITHcolumn), cast to numeric(p,s) or not, so a typmod over a variance's text never stores an approximation; only a stored column's or a JSON number's text read as a numeric is said as a text: a JSON number's, a stored double's or any other. A run that fails (a value past 38 digits,bigint out of range,division by zero) records Postgres's error, SQLSTATE and DETAIL (numeric field overflowsays the precision and scale it must fit, on every path), and is tried again after its back-off. mason statusshows generated tables; the web page doesn't yet.- Pages of every LIMIT with no ORDER BY over a join may overlap:
LIMIT n OFFSET mwith no ORDER BY over a join has
no stable order, so a tool paging one can see a row twice and miss another. Add an ORDER BY where the pages matter. - A replicated table whose re-read never completes can rejoin the stream with no catch-up, and miss what the stream
passed over, as in 0.8.0 to 0.16.0. It was found by reading the code and has not been reproduced.