Repository navigation
Mason v0.21.0
A column added to, renamed in or dropped from a synced table on your database now reaches Mason without a restart.
A numeric column with no precision, such as Rails makes from t.decimal, prints each value as your database does:
12.5, not 12.50. A table under sync: replication no longer blocks ALTER TABLE … DROP COLUMN on your database.
mason stop ends within seconds when a read waits on a lock there, and removing a table stops its sync at once.
A sync says the rows it added, updated and deleted. Listing a JSON column's keys, and the smallest and largest value
of a text column, are many times faster. node-pg's array parameters work. This is a new release line. Every project
has something to do first: its pin moves. A generated table with a jsonb column makes the first start longer, and
nothing is served until it is done. A script that reads mason sync --json or its exit code may need to change
("Before you upgrade").
Before you upgrade
- Move the pin:
version: 0.21.0inmason/mason.yml. The 0.21.0 command and image refuse a project pinned to
0.20.x, and say how to fix it. A 0.20.x command or image refuses a project pinned to 0.21.0. Before 1.0 they only
run a project pinned to their own minor version, at their own patch or an older one. So set the pin and start 0.21.0
in the same step, once everyone and everything that runs the project has 0.21.0. - The 0.20 line still gets security fixes. They are released as 0.20.x patches, from the
release/0.20branch.
A project pinned to 0.20.0 can stay on that line until you are ready to move. - On a hosted replica, upgrade the replica first, then deploy the project pinned to 0.21.0.
mason replica upgrade <org>/<name>installs 0.21.0 beside the version the replica runs. The replica keeps running your project as
it is until you deploy. Then setversion: 0.21.0, make the changes these notes ask for, and runmason deploy.
0.21.0 starts with that deploy.mason deployrefuses a pin the replica would not run, before it sends anything,
and names both versions. - Allow for a longer first start if a generated table has a
jsonbcolumn. At Mason's first start on 0.21.0,
Mason rewrites each such table'sjsonbvalues once, in the form Postgres stores them ("What was wrong and is now
right").- Nothing is served until it is done.
/healthzanswers 503 meanwhile. - It takes about 30 to 45 seconds a million rows on a 4-core machine. The log names each table rewritten.
- If your platform restarts a container that stays unhealthy, give the first start room in its health check for
your largest generated tables. The image's own check gives up after about 2 minutes 45 seconds. The rest of the
start counts against it too, so it covers about 3 million rows.
- Nothing is served until it is done.
- Some synced tables are synced again once, at the first start. Each such table keeps serving its old data until
its new sync is done. A table takes about as long as its first sync did, and is read from your database again.
mason statussays so on the table's line while it waits. You don't need to do anything. These are the tables:- A table with a
numericcolumn of no precision. Mason now keeps each value's own decimals, which the data
synced before does not hold. Until the new sync is done, its values print the old way. - A table that an older Mason left without one of its columns. This is a table with a column whose name begins
with__mason_json_, which that Mason dropped ("What was wrong and is now right"). - A table with a column whose name begins with a byte-order mark (U+FEFF), when that column is not of a text
type. An older Mason stored such a column as text. A column of typetext,varchar,char,name,inet,
citext,jsonorjsonbwas not affected, and its table is not synced again. Nor is a column of a domain.
- A table with a
- Over replication, an older Mason removed some invisible spaces from array elements. This hit the elements of a
text[], avarchar[]or an enum array. An element that began or ended with a byte-order mark (U+FEFF), a
non-breaking space (U+00A0), an ideographic space (U+3000) or a similar space lost it in the synced data. 0.21.0
keeps these spaces in every new change. A row changed before the upgrade stays as it is until it changes again, or
until its table is synced again. If this matters to you, sync again each replicated table with such a column:
mason remove <table>, thenmason add <table>. The table is not served until its new first sync ends. - Check any script that reads
mason sync, its--jsonorPOST /api/sync("What changes" has each):mason sync --jsongivesadded,updatedanddeletedfor each table, in place ofchanged.- A sync of a table that is already syncing is refused with the new code
busy, and exits 1. Its--jsonhas a
syncingkey beside the failure. Where a table's first sync, whole sync again or catch-up is under way, the code
wasnothing_yetand is nowbusy.nothing_yetstays for its other cases, such as a first sync that failed. POST /api/syncanswers202as soon as the sync starts. Add"wait": trueto the body to get the old answer.- A script that matched the words of a
synced,polled,streamedorfull_syncevent must match the new words. - A script that matched the text of
sourcesReadOnlyinmason status --jsonmust match its new text.
What changes
- A sync now says how many rows it added, updated and deleted, such as "1,200 added, ~282,770 updated, 0 deleted".
Before, it said "283,970 changed", which lumped new rows and edited rows together. The new words are inmason sync,
the web page's Sync now and Sync events, andmason logs --events. A first sync, or a table synced again whole, adds
every row.- A poll's updated count is an estimate, shown as
~N updated. A poll counts each changed row once, by its key. So
it can count rows that it read again while a transaction was open on your database, which makes it too high. It
also counts a row updated several times as one, and a row whose key changed as one added and one deleted. A count
of 0 is shown without~. mason sync --jsonkeeps the plain number inupdated.- Over replication, the counts follow your database's own inserts, updates and deletes, and are exact.
mason status, the web page and the run's log say them the same way, such as "1,000 added, 200 updated, 4 deleted".
Before, they said "1,204 changes (1,000 inserts, 200 updates, 4 deletes)". - A poll's own lines in the run's log still say "changed".
- A poll's updated count is an estimate, shown as
- A sync of a table that is already syncing, or waiting to start, is refused at once. This holds for the web
page's Sync now, formason syncand forPOST /api/sync. It is never queued for another read of the table.- It is said as progress, not as a failure.
mason syncprints, on stdout, "public.orders is already syncing. It
started 2 min ago. Mason never starts a second sync beside it." Then it says how to follow the sync. The page says
it beside Sync now, and that the table's row shows when it ends. - The refusal has a new code,
busy, and exits 1, since the sync you asked for did not run. - With
--json, it is the failure shape with codebusy, andsyncingbeside it.syncingis{ tables, startedAt }of the sync running, or null when it is Mason's own sync of the table. mason syncnamesmason statuswhen a sync by hand or a scheduled poll is running. When the table's first sync,
whole sync again, catch-up or column change is under way, it namesmason wait <table>first.
- It is said as progress, not as a failure.
POST /api/syncanswers as soon as the sync has started. It answers202with{ sync: { id, tables, state, startedAt, endedAt } }. Each table in/api/statethen hassyncByHand. It sayssyncinguntil the sync ends.
Then it sayssyncedwith what the sync did, orfailedwith why. A script that calls the endpoint and needs the
old answer adds"wait": trueto its body.mason syncdoes this, so it still waits and prints the same report. A
refusal is answered at once, as before.- A sync you start by hand that fails is now listed under Sync events, as "Manual sync failed" (
sync_failedin
mason logs --events). - The web page's Sync events list shows each sync while it runs. A sync you start by hand, a first sync, a table
synced again whole and the catch-up after it each get a row at the top of the list. The row says "Syncing…", and
Took counts how long it has run so far. When the sync ends, its event takes the row's place. Scheduled syncs are not
shown while they run, as each takes a moment.mason status --jsongives each table's own sync running now as a new
field,rebuilding, withkind(copyorcatch-up) andstartedAt. - The Sync events list shows how long each sync took in a column of its own. The Took column holds it, such as
570 msor7 min 57 s, and you can sort the list by it.mason logs --events --jsongives it as a new field,
tookMs. A first sync's event now ends with its time as a poll's does, such assynced: 1,000 row(s) · 2.0 s.
Before, it saidin 2.00 s. Events recorded before the upgrade keep their time in the message.- In a narrow window, the list scrolls sideways inside its box, as the Tables and Users lists do. A very long table
name is cut short. Hover over it to read it whole.
- In a narrow window, the list scrolls sideways inside its box, as the Tables and Users lists do. A very long table
- Listing a JSON column's keys is now about as fast as on Postgres.
jsonb_object_keysandjson_object_keysof a
column took 16 to 70 times as long as on Postgres. This was so grouped as Metabase sends it, withDISTINCT, and in
aLEFT JOIN LATERAL. Over about 5 million rows they timed out, and a small query in another session waited up to 8
seconds meanwhile. Over 3 million rows they now take 3 to 5 seconds, a little less than Postgres. Other queries no
longer wait. This holds for a synced table's JSON columns and a generated table's alike. - The smallest and largest value of a text column, and a page sorted by one, are many times faster when your
database sorts text by a language's rules (a collation such asen_US.UTF-8). Metabase reads them when it scans
a table, and for a table's preview.- Over 4.5 million rows, most took 2 to 22 seconds, and now take 0.2 to 2 seconds. The slowest was a column where
most rows hold the same value, such as a status. - A column where every value is long and different gains least. Over 4.5 million rows it now takes 3 to 4 seconds.
- Over 30 million names, they took 2 to 4.5 minutes, close to the 5-minute limit. They now take 13 to 23 seconds.
- Names that begin with an accented letter, such as É, Ø or Ł, are as fast as the rest.
- The answers are the same as before. A list of distinct values, a smallest value per group and
FETCH FIRST … WITH TIESare still slow.
- Over 4.5 million rows, most took 2 to 22 seconds, and now take 0.2 to 2 seconds. The slowest was a column where
- A generated table that computes a
jsonbcolumn takes longer to build. Writing each value in Postgres's form
costs about 30 to 40 seconds a million rows on a 4-core machine. Ajsonbcolumn taken as it is from another table
costs nothing more. mason status --jsonwordssourcesReadOnlyas sentences, and a new field says which reason it is.sourcesReadOnlyis still null when Mason can write the project's sources file. Otherwise it is now one or two
whole sentences. Before, it was one lower-case clause, with the fix after a semicolon.- The new field
sourcesReadOnlyKindisdeployedwhen the project's files are deployed read-only
(MASON_PROJECT_READONLY=1). It isunwritablewhen a server cannot write its sources file. It isunknown
when Mason cannot tell whether the files are read-only. It is null whensourcesReadOnlyis null. - Match on
sourcesReadOnlyKindrather than on the text./api/statehas both fields too. - On such a server, the web page's Tables page links to how to deploy a change, where a deploy is the way out.
- The host's card (
MASON_UI_OVERVIEW_CARD) is now the first thing on the web page's Overview, above Mason's own
cards, at the full width. A host with nothing to say answers{ "refresh": 60, "sections": [] }, and may leave the
title out. The Overview then draws no card at all. It reads the path again everyrefreshseconds, so a card added
later shows up on the next read. A read that fails is still shown in the card. Before, an empty answer drew a box
with only its title, and the card came last. - Each release of the image now also has a tag for its line. The tag
0.21moves to each new patch of 0.21, and
never back. SetMASON_TAG=0.21to take each patch of the line with a restart. A patch released for an older line
never moveslatest, on the image or on npm. On npm it is published underlatest-0.<minor>, such aslatest-0.20.
What was wrong and is now right
- A column added to, renamed in or dropped from a synced table on your database now reaches Mason without a restart.
Before, a new column never reached the synced data until Mason was started again, whilemason statussaid every
table was synced. A column renamed or dropped made the table's sync fail again and again until a restart.- Now Mason checks the table's columns before each poll, at
mason sync, and when the change log says the table
changed. The check reads only your database's catalog. - When the columns changed, Mason syncs that table again whole, and serves its old data until that ends.
- A new column that looks like a secret, such as
api_token, is left out, asmason addleaves it out. Mason
writes it inexclude:on the table's entry in the sources file, and says so in its log, its events andmason status. To sync it, runmason add <table> --include <column>. When the project's files are read-only, Mason
leaves the column out and says the line to write. - A column that
exclude:names and that is renamed while Mason runs is never synced. Mason writes its new name in
exclude:in its place. One that is dropped is taken out ofexclude:. - A column that
columns:names and that your database dropped no longer stops Mason from starting. Mason says so
inmason status, and syncs the columns that are left. - A whole sync that fails because a column changed under it now reads the columns again and starts over with them.
Before, it failed again and again. - A publication you name in the sources file (
publication:) is never changed by Mason. If it lists a table's
columns, Postgres does not send a new column. Then Mason stops keeping that table current, and says theALTER PUBLICATIONline that adds the column.
- Now Mason checks the table's columns before each poll, at
- A table under
sync: replicationno longer blocksALTER TABLE … DROP COLUMNon your database. On Postgres 15
and later, the publication Mason makes for itself listed each table's columns. So your database refused to drop a
listed column, with "publication … depends on column". It also never sent a column added later.- Mason's publication now names each table whole. Mason updates a publication it made before, once, soon after its
first start on 0.21.0. It keeps syncing while it does. - Columns that a table's entry leaves out, with
exclude:orcolumns:, now reach Mason over replication. Mason
drops them as they arrive. They are never stored, logged or served. - A publication you name with
publication:is still never changed. If it lists a table's columns, Mason says once
that the list blocksDROP COLUMN, and how to name the table whole. - This was listed as known in 0.20.0. The steps given there to drop a column are no longer needed.
- Mason's publication now names each table whole. Mason updates a publication it made before, once, soon after its
mason stopnow ends within seconds when one of Mason's reads waits on your database. A table's first sync, its
catch-up or a scheduled sync can wait on a lock that another session holds on the table. Before, the stop waited
until that lock was released, which could be never. Now Mason cancels its own waiting reads on your database, and
nothing of Mason's is left running there. A first sync keeps what it had synced, and goes on at the next start. If
your database refuses the cancel, the log says so once, and the stop waits for the read as before.- Removing a table while it syncs now stops its sync at once. This covers a table's first sync, a table synced
again whole, the catch-up after it, and a sync you started by hand. Before, the sync ran to its end, which took hours
for a large table. Every table added beside it waited until then, and your database kept reading the removed table.
Now Mason cancels its reads on your database, drops what it had synced of the table, and starts the next table's
sync within seconds. The log and Sync events say "Sync stopped" with the reason, and it is never shown as a failure.
A table you add again right after removing it starts a new first sync. - Sync now on the web page no longer shows a timeout for a big table behind a proxy. The page waited for the
whole sync in one request. A reverse proxy or load balancer in front ofmason serveclosed the request when its
own time limit ran out. The sync went on, but the page said it failed. Now the button says Syncing… while the sync
runs, even after the page is reloaded. Then it says what the sync did, or why it failed. - A
numericcolumn with no precision now prints each value as your database does. Rails makes such a column from
t.decimal. Mason printed every value with the most decimals any value in the column had. So12.5read12.50,
and0read0.00.- Now these give your database's text: a plain
SELECT,::text, arithmetic,sum, most window functions, a
subquery inFROM, and the binary wire. - So does every way such a value is turned into text:
to_jsonand the other JSON functions, a row's text,
concat,format,||,string_agg, an array's text, and the text of amin,maxorsum. SELECT DISTINCT x::textkeeps12.5and12.50apart, as your database does.- A few readers still print the old way. They include
array_aggitself,minandmaxthemselves, a scalar
subquery,percentile_disc,mode(), aUNIONwithoutALL, and a generated table's column.
docs/SQL-SUPPORT.mdlists them all, undermixed-scale-numeric. To get the same text there, cast the column to
numeric(p, s)or useround(x, s).
- Now these give your database's text: a plain
- A whole number beside a decimal column no longer loses the column's decimals.
- This covers
div()of two integers, an integer cast tonumeric, a whole constant such as2.5e2, and a whole
part ofextract. - When one met a decimal column in an
IN (SELECT …), aUNION, anINTERSECT, anEXCEPT, aCASE, a
COALESCE,GREATESTorLEAST, the column's decimals were rounded away with no error. So
a IN (SELECT div(i, 7) …)matched 0.01 to 0, and rows came back wrong or in the wrong order. Now the rows are
your database's. - A
UNION,INTERSECTorEXCEPTwithoutALLprints every row at one scale, the widest. So it prints3.00
where your database prints3. - Beside a column with very many decimals, such as
numeric(38,30), Mason keeps every digit before the point and
the column's first 19 decimals. It rounds the rest. Your database keeps all 30. A number of 1 or more is the
same to at least 19 significant digits. A smaller one is the same to its first 19 decimals, so a value smaller
than 0.00000000000000000005 reads 0. Beside a sum of whole numbers, Mason keeps 10 decimals. - Values that differ only after the decimals kept compare equal there.
UNIONandDISTINCTmerge them, andIN
and a join match them. - Comparing such a column with
=, or with anINlist of constants, to a whole number that does not fit its type
still fails withnumeric field overflow, as before. - To get your database's rows exactly, cast the column to 18 decimals or fewer first, such as
::numeric(38,18).
- This covers
- A list of values passed as one query parameter now works, as node-pg passes it. node-pg, and the ORMs built on
it, send a JavaScript array as text such as{"1","3"}, and let the database work out its type. Mason answered
WHERE id = ANY($1)andWHERE id = ANY($1::int[])with an error for such a value. Now it answers them as Postgres
does, for arrays of integers, text, uuids, numerics, booleans, dates, timestamps and floats. That includes an empty
array, NULL elements, and quoted elements that hold commas or braces. An array sent in binary, as Npgsql sends it,
works too. - A
numeric[]parameter is compared with every digit kept. This holds for a numeric or an integer column, in
= ANY,<> ALLand the other comparisons withANYorALL. So doesx = ANY(ARRAY[$1::numeric, $2::numeric]),
which matched values that differ past the 15th digit. A NULLnumeric[]is answered everywhere.- Mason refuses any other
numeric[]value by name, and says what to write instead. This is anumeric[]read
anywhere else, such as in the SELECT list or inunnest, or one compared with a float column.
- Mason refuses any other
- A timestamp with time zone compared with a
timestamp[]ordate[], or with a subquery of timestamps or dates,
now uses your session's time zone. This coversANY,ALL,INandNOT IN. Before, a session outside UTC got
other rows. The same holds for a timestamp or a date compared with a wider time type. - Text that starts with a byte-order mark (U+FEFF) now keeps it. A CSV import can leave one at the start of a
value or of a column's name.- Mason dropped the mark from every value that began with one. So a value that was only a mark read as empty, and
two values that differed only by it looked like duplicate rows underGROUP BYorDISTINCT. - This held for text and varchar columns, elements of a
text[],->>of a JSON string, and atext[]parameter
sent in binary, as Npgsql sends it. Now each reads as your database has it. - A column whose name began with a mark, and whose type is not a text type, was synced as text. So a
numeric
column served text, not numbers. Now it keeps its type. - An element of a
text[]that begins with a mark or a non-breaking space is no longer put in quotes where your
database leaves it bare. - Over replication, such an element no longer loses its mark or its space ("Before you upgrade").
- Mason dropped the mark from every value that began with one. So a value that was only a mark read as empty, and
- A column whose name begins with
__mason_json_is no longer dropped from the synced data. Mason uses names like
this for columns of its own. At each start, Mason dropped a column of your table named like that, and its values were
lost. Then every poll of that table failed. Undersync: replication, the stream stopped for every table. Now Mason
knows its own columns from what it records, whatever their names. So every column of your table is synced and
served. A column named like Mason's other hidden columns, such as__mason_scale_, is kept the same way. - A generated table's
jsonbcolumn is now stored as Postgres stores it, so->,@>and its keys answer right.
When a generated table's SQL wrote a key twice, as injsonb_build_object('a', 1, 'a', 2), Postgres keeps only the
lasta. Mason kept both, in the order written.->read the first one, and@>checked against the first one.
Now eachjsonbvalue is written in Postgres's own form when the table is built: keys in Postgres's order, each key
once, at every depth. These reads are right, and as fast as on a synced table. - Removing a synced table whose name a generated table also uses no longer stops the sync. While a synced table
holds a name, a generated table of that name is refused, as before. Once you remove the synced table, the generated
table is built again and served. Before, if no review of the generated tables ran between the add and the remove,
every sync pass failed with "Table with name … does not exist". Queries on the name failed until one of the
generated table's bases changed. A project already in that state recovers at its first start after the upgrade. - A generated table whose name differs from a synced table's only in case is refused. The synced data does not
tell such names apart, soOrders.ymlbeside a syncedorderscould replace the synced table's rows. Now it is
refused by name, as a generated file of the synced table's own name is. Two generated files whose names differ only
in case are refused the same way. Rename the file to fix either. - Mason no longer crashes when it stops while a client is still connecting. A connection that was still opening
as Mason stopped could crash the process with a segmentation fault or an abort. The supervisor then saw a signal and
no reason. Now the stop waits for each connection still opening, and then closes the synced data. - Every button on the web page that changes something now shows that it is working, and can't be sent twice. Its
label says what it is doing, such as Removing…, Disabling… or Opening…, and the button stays the same size. Before,
most of them gave no sign until the answer came. A double click on a Remove or a Cancel query sent it twice, and the
second, refused, hid that the first had worked. Now one request is sent, and the page says what it did. - A change the web page sends that fails is said beside its button, with Dismiss. Escape dismisses it too. This
includes Add a user's error, which could not be dismissed and pushed the rest of the page down, and the cloud
login's. Nothing on the page moves when a failure is said, or while a button is working. A request that gets no
answer is given up after 60 seconds, and says so. Before, its button could stay busy for ever. - A refused Stop of a temporary cloud login is now said, beside Stop. It says that the login is still open, and
that its address and password still work. Before, the page said nothing, and the login stayed public. - Changing a filter or a page of Queries or Sync events keeps the last list on the page, greyed, until the new one
is read. Before, the list went blank and the page jumped.