Skip to content

Releases

Trellis 0.3.1

Choose a tag to compare

@mmmries mmmries released this 19 Nov 18:08
dedfe5e

Bugfix. Capture keys a nullable KEY column the way the build does (0.3.1) (#2126) by @mmmries in #2150

Full Changelog: v0.3.0...v0.3.1

Trellis 0.3.0

Choose a tag to compare

@mmmries mmmries released this 16 Nov 16:03
63eaabd

Highlights

  • A rolling deploy needs no migrate step. Once every old process has
    stopped, a background process of the new release runs the pending
    migrations itself and takes over in the same pass (#1852). The takeover
    work runs once per binary version, recorded with the work (#1855).
  • Faster aggregate and JOIN builds. A plain aggregate's fresh build
    writes each group once, per leaf on a partitioned source; rebuilds update
    hidden rows in key order; JOIN … GROUP BY builds write their hidden
    tables in sorted single passes. At 10M groups a fresh aggregate build takes
    82 s from define to live, against 1,535 s for 0.2.0's form (#1938,
    Benchmarks).
  • Correct on the PostgreSQL 2026-11 minors (15.20, 16.16, 17.12, 18.7),
    whose attribute versions (CVE-2026-6460) hid a same-type column rewrite
    from capture (#1947). Also in 0.2.1.
  • A revoked schema USAGE halts only the definitions that read or write
    that schema,
    instead of failing the staging pass for every table
    (#1921, #1925).
  • Alertable metrics for definitions by status, pauses by cause, held
    keys, tripped column fuses, drain failures, capture holdups and worker
    liveness (#1957).
  • The Ruby gem follows Rails conventions: logs to Rails.logger, a
    metrics endpoint and exporter middleware, read-only Trellis::Target
    models, a test-environment off switch with refresh_trellis_target, and a
    CI check for risky deploys. Trellis is now off in a Rails app's test
    environment by default
    (Behaviour changes).

0.3.0 includes every fix in 0.2.1. 80 pull requests merged since v0.2.0
(changelog).

Bugfixes

  • A deadlock between a page and a rebuild start (#2019, #2021; also in
    0.2.x).
    A page that parked a quarantined key's change, or wrote a 1-1
    JOIN's hidden state, took definition rows in two places out of id order.
    A rebuild start (request_backfill, the last capture install) locks the same
    rows in one sorted statement, so the two could deadlock. The start usually
    lost and returned deadlock detected (40P01) to its caller; no data went
    wrong. A page now takes all its definition rows in one id order, after its
    other targets. Truncate clears run in target order too.

  • A 1-1 JOIN build could go live with a stale target row (#2027; also in
    0.2.x).
    A page that re-derived an entry while the build's M pass ran,
    whose a row was deleted before its chunk read the range, left a target row
    joined to a replaced M entry until the delete's page drained. The sweep's
    stale bound now sits past the M pass, and every 1-1 JOIN build, fresh
    ones too, ends with the sweep. A fresh build takes about 18% longer (5M
    rows: 36.7 s to 43.4 s); a build in 0.3.0 is still much faster than in 0.2.x.

  • Same-type column rewrites on the 2026-11 minors (#1947, #1948; also in
    0.2.1).
    Capture missed an ALTER COLUMN … TYPE … USING that kept the
    type, and its readers' targets stayed silently stale. The rewrite check now
    leaves the attribute version out of its hash. Tests simulate attribute
    versions on every server, so pull-request CI covers these minors.

  • A revoked USAGE on a source's schema (#1921, #1925). The definitions
    reading or writing a table in that schema halt (capture_failure kind
    halt, naming the role and schema), and status() answers for them.
    Every other table keeps staging. After GRANT USAGE, RESUME rebuilds
    them. In 0.2.x the drain failed the page forever, status() of an affected
    aggregate returned a permission error, and the staging pass failed for
    every table.

  • A star JOIN paused by an ON-column retype (#1950) names only the
    retyped column, once. A two-table JOIN whose only ON column re-renders
    (integer to text) now also says the pair no longer matches.

  • Key types are matched by OID, not type text (#1875). A user type or
    domain named like a key type (create domain "integer" as numeric) is not
    taken for the built-in one in a retype pause reason. A refused extension
    type is named in full (public.citext).

    Behaviour changes

  • Ruby: Trellis is off in the test environment (#1959).
    config.trellis.off_in defaults to %w[test], read from every
    environment, so db:prepare from development leaves Trellis out of the
    test database too. There nothing connects at boot; Trellis::Railtie.connect
    and trellis:migrate warn and do nothing; a schema load creates the
    targets as plain tables and leaves the definitions in the restore queue;
    the migration helpers (define, apply, status) raise
    ActiveRecord::MigrationError, so build the test database with
    db:schema:load or db:test:prepare, not db:migrate. Tests write
    targets directly or fill one with refresh_trellis_target.
    Action: a suite that runs Trellis in its tests sets
    config.trellis.off_in = [] (the gem README's "Running Trellis in
    tests"). Without it nothing connects and Trellis::Railtie.connect warns.

  • Ruby: Rails sends the engine's log lines to Rails.logger, at its
    level (#1954). Action, if a debug log is too much:
    config.trellis.log_level = :info, or config.trellis.log_bridge = false.

  • Rust: while migrations are pending, a failing status() or
    definitions() returns TrellisError::MigrationsPending
    (code
    version_handover, the SQL error as its source), not the bare SQL error.
    Action: code matching that SQL error matches the new variant.

  • A worker role that can't alter Trellis's tables stands by until
    migrate runs
    (above). Action: keep the deploy step.

  • A queued restore runs before the takeover (#1855). While it refuses (a
    conflicting definition, a role without DDL), every background process
    stands by, retries every minute and reports why in its log and status().
    0.2.x took over and left the queue unrestored. Action: fix what the
    reason names.

  • More rebuilds on the 2026-11 minors (also in 0.2.1). There an
    ALTER COLUMN … TYPE re-stamps every later column. When the table's file
    also moved, each later column a definition reads counts as rewritten and
    its readers rebuild, some for nothing: one rule that errs toward
    rebuilding (decided on #1948). Action: expect such an ALTER to
    rebuild the readers of the table's later columns.

  • Rolling back to 0.2.x and forward again skips the takeover work
    (below). Action: none
    for 0.3.0.

v0.2.1

Choose a tag to compare

@mmmries mmmries released this 13 Nov 22:52
e2693e0

What's Changed

This patch version introduces an important bugfix where the latest postgres minor versions now stamp attvers in a way that broke our detection mechanism for cases where an ALTER TABLE could re-write all the values in a particular column. We now detect this case reliably both before and after the latest postgres minor versions. See #1947 and #1971 for details

We also had a few minor CI updates.

Full Changelog: v0.2.0...v0.2.1

Trellis 0.2.0

Choose a tag to compare

@mmmries mmmries released this 11 Nov 00:20
f2afc74

Trellis 0.2.0

Highlights

0.2.0 is the first minor release after 0.1.0. The main changes:

  • A much broader subset of Postgres' expression language is now supported.
    more functions and operators, all follow postgres semantics.
  • JOINs are now supported in 1-1 transforms with up to 5 joins in a combination
    of LEFT and INNER joins in a typical start pattern.
  • MIN/MAX work in plain GROUP BY aggregates, backed by a required
    source index.
  • SUM/AVG over any numeric, including unconstrained columns,
    expressions and another transform's target columns, show exactly what
    Postgres shows.
  • Source keys are explicit. Trellis keys a table by its primary key or by
    a KEY (…) the definition names. It never picks a unique index on its own.
  • Upgrades and rollbacks are managed. Two versions never run side by side
    (version handover). The catalog and every definition record a
    compatibility floor, and a release below a floor stands by or pauses
    instead of misreading state.
  • Schema dumps carry definitions, not Trellis internals. New trellis dump
    and trellis restore commands, Rails and Ecto load hooks, and
    ADOPT EXISTING TABLE to define into an empty table you already created.
  • Connection budget. The background pool needs one more connection than in
    0.1.0: 2N + 6 with staging and N drain threads (was 2N + 5), 2N + 5
    with drain threads only, and 2 with staging only. The call-pool options
    (max_call_pool_size, call_pool_wait_timeout_ms) are unchanged.
  • TRANSFORM <target> ADOPT EXISTING TABLE FROM … builds into an existing
    empty table that is shaped exactly as define would create it. It may carry
    anything a user may add to a target (indexes, constraints, triggers,
    grants, policies, …). Anything else is refused with TargetNotAdoptable,
    naming the property.

Bugfixes

  • Name quoting is now supported and unquoted names are interpreted in lowercase
    to match how Postgres handles them in its own query parsing.

Validations

Validation beyond CI since v0.1.0 (2026-10-31), from the maintainers'
validation logbook: 214 recorded runs, 112.9 hours, and about 5.12 billion
source rows or cases. That breaks down as:

  • 114 generative sweeps (70.6 h, about 107 million source rows);
  • 7 expression-conformance sweeps that checked about 1.83 billion generated
    expressions against Postgres (6.9 h);
  • 13 soak runs, 12 of them segments of the 0.2.0 generative soak (12.0 h,
    about 2.03 billion rows or generated expressions);
  • 37 benchmark runs (9.0 h, about 1.10 billion rows);
  • 28 repro loops (9.5 h, about 46 million rows);
  • 15 acceptance runs, the upgrade battery among them (4.9 h, about 14.8 million
    rows).

Most row counts are the runs' own estimates. The Validation
section has the details. We also ran an Upgrade battery)
to test upgarde and downgrade cases.

Upgrading

Take a backup of the whole database before upgrading. Rolling back to 0.1.x
means restoring that backup (Rollback below).

Future versions will have strong support for rollbacks, where transforms will
continue to run as much as the older version can support.

Procedure

The steps are in
upgrading,
and the same steps apply to the CLI and to embedded applications:

  1. Start 0.2.0, beside 0.1.0 or after it. While a 0.1.0 process is live,
    every 0.2.0 process stands by. It stages, drains and builds nothing, and
    checks again every 5 seconds. Its state-changing calls refuse with the new
    version_handover error code. trellis status prints a Standing by:
    line, and trellis run prints trellis standing by: …. 0.1.0 keeps
    running, and the capture triggers keep staging changes.
  2. Stop every 0.1.0 process. 0.2.0 takes over at its next poll, or up to
    30 seconds later after a process was killed without a clean shutdown.
  3. Run migrate. 0.2.0 adds migrations V9__worker_versions.sql through
    V20__definition_join_tables.sql. trellis run migrates by itself once
    0.1.0 has stopped. An embedded fleet runs migrate
    (bin/rails trellis:migrate, or a release task calling Trellis.migrate!)
    in a deploy step after the old processes are gone. While a 0.1.0 process
    is live, migrate refuses with version_handover.

Definitions 0.2.0 refuses

0.2.0 refuses some definitions that 0.1.0 accepted. At the takeover, each
such definition pauses, with a capture_failure naming the refusal and the
remedy. The other definitions keep running. RESUME and ALTER TRANSFORM
refuse it, and DROP TRANSFORM works. Remedy: DROP TRANSFORM, then
define it again in the current grammar.

  • A bare reserved word as a column name, such as SELECT order AS o.
    Write "order" or orders.order. This covers every word
    pg_get_keywords() marks R or T (order, user, end, group,
    limit, …).
  • An expression nested more than 100 levels deep, as an operator chain or
    as nested parentheses. Split it into calculated fields.
  • A number directly followed by a letter, such as a+1AS x or 1AS x.
    Put a space between them. (Confirmed in the pre-#1816 battery run
    20261109T071955-e-acceptance. v0.1.0 accepted 1AS and refused 0x1,
    1e, 1.5e3x and 1_000. 0.2.0 pauses it with "trailing junk after
    numeric literal … drop the transform and define it again", and that remedy
    works.)
  • Definitions that parse but fail a define-time check keep the state they
    had (#1826, decided 2026-11-09).
    The takeover pauses only what the new
    grammar can't parse (the cases above). A stored definition that parses but
    that 0.2.0's define would refuse keeps the behaviour it had under the
    release that accepted it. For 0.2.0 this is a chain of calculated fields
    too long to inline (FieldExpansionTooLarge, 51 or more fields): it stays
    live and correct. A stored constant that Postgres can't evaluate
    (2147483647 + 1) is no longer a refusal at all (below): the takeover
    leaves its state and held keys as 0.1.x left them, and rows that fail
    after the upgrade are handled as described there, except that keys 0.1.x
    held count toward the whole-definition fuse.
  • A bare true, false or null read as a column or field name (#1825,
    fixed).
    A stored 0.1.x definition keeps reading the column or field, as
    0.1.x did. Confirmed from the final RC battery on 949240d from v0.1.0 and v0.1.1: null AS x,
    NULL AS x, COALESCE(null, a), false AS x, true AS x, true + 1
    and a field named null stay live and equal Postgres's reading of the
    column through updates, inserts and deletes, with self_check clean. The
    same text defined now reads the literal (grammar 2).

Names over 63 bytes. A definition 0.1.x stored with an AS label over 63
bytes keeps running. Postgres had truncated that target column at define.
Its text can't be written in grammar 2, so ALTER TRANSFORM refuses it
(NameTooLong). If the takeover has to write a KEY (…) into it (below),
the definition pauses instead. Remedy: DROP TRANSFORM and define it again
with a name of 63 bytes or fewer.

PostgreSQL 18 virtual generated columns. In 0.1.0, a definition that read
a GENERATED ALWAYS AS (…) VIRTUAL column made every insert into its source
fail. 0.2.0 refuses such a definition at define, ALTER TRANSFORM and
RESUME, and pauses its readers. Remedy: make the column STORED and
RESUME, or drop the transform and define it again with the expression
written in the definition. A definition 0.1.0 already built over such a
column pauses at the first staging pass after the takeover, with that reason
and remedy, and inserts into its source work again. (Confirmed on
PostgreSQL 18.6 in the pre-#1816 battery run 20261109T071955-e-acceptance:
a 1-1 transform and a SUM over the column paused, RESUME refused, and
after the column was dropped and added back STORED, RESUME brought both
back matching Postgres.)

What new text means differently (grammar 2)

Grammar 2 is the first grammar-version bump
(ADR-0018).
A stored definition keeps its meaning. Text written after the upgrade reads
the new way:

  • An unquoted name folds to lower case, and a quoted name is exact, as in
    Postgres. SUM(Total) reads total. A column, table or target created
    with upper-case letters must be written quoted: SUM("Total").
  • Scripts and migrations that address a target 0.1.x created with a
    mixed-case name must quote it:
    PAUSE TRANSFORM "OrderTotals",
    DROP TRANSFORM "OrderTotals". The unquoted spelling is refused with
    did you mean "OrderTotals"? and acts on nothing.
  • Stored 0.1.x definitions keep reading their names as written.
    ALTER TRANSFORM stores such a definition with its names quoted
    (SUM("Amount")). Text with no quoted name and no upper-case letter is
    stamped grammar 1 whichever release wrote it.
  • Refused in new text:
    • names over 63 bytes after folding (NameTooLong);
    • unquoted reserved table names (FROM user, FROM order,
      TRANSFORM order; public.order is fine), a reserved word as a
      GROUP BY or KEY (…) column, and a reserved JOIN qualifier;
    • the quoted forms "", U&"…", a quoted name holding . or a non-ASCII
      character, and a quoted function or type name ("lower"(x),
      CAST(x AS "int4")).
  • TRUE, FALSE and NULL are literals in new text.
  • Error messages spell function and construct names in canonical upper case.

Definitions that pause because 0.2.0 keys a table differently

Each case below has its remedy. The details are in
upgrading.

  • A source key with INCLUDE columns (`primary...
Read more

Trellis 0.1.1

Choose a tag to compare

@mmmries mmmries released this 09 Nov 16:38
0e13fef

What's fixed

  • A SUM or AVG that fell into NaN was stuck, we now track and repair this (see #1756, #1763)
    • If you have any targets that are already stuck, you'll need to request_backfill on them. You can use self_check to check for this stuck state
  • Unique expression indexes no longer qualify as a keyspace definition (see #1725)
  • self_check had a false-positive case for aggregates with active data changes which was fixed (see #1688)

Upgrading from 0.1.0

  • Stop every 0.1.0 process before you start 0.1.1. Releases before 0.2.0 have no version handover, so two of them must never run on one database at the same time. Stop every worker and every handle, deploy 0.1.1, then start it.
    • This will be improved in the 0.2.0 release to automatically standby until the old processes stop
  • Rolling back to 0.1.0: with no migration, 0.1.0 opens a database 0.1.1 has run. One exception: 0.1.1 stores a numeric(p,s) sum that counts a NaN as the finite values plus 10^2100 per NaN, and 0.1.0 reads that as a 2,100-digit sum once the group is next written (until then the visible value stays correct). Running request_backfill under 0.1.1 doesn't help, because 0.1.1's rebuild writes the same encoding again. After rolling back, run request_backfill under 0.1.0 on every source with a numeric(p,s) SUM/AVG that holds a NaN, ideally before writes reach it; this also repairs a group that already shows the 2,100-digit sum.

Known issues in the 0.1 line, fixed in 0.2.0

  • On PostgreSQL 18, a definition reading a virtual generated column makes every insert into its source fail (#1570). Make the column STORED, or don't read it.
  • ALTER COLUMN … SET EXPRESSION on a stored generated column that a definition reads rewrites its values with no trigger, so the readers stay live and stale (#1540). Run request_backfill on the table afterwards.
  • A source key column re-collated after define, between deterministic collations, can make a rebuild or self_check fail with "could not determine which collation to use". This covers a part of a composite 1-1 key (#1537), an aggregate JOIN's GROUP BY column (#1558) and an aggregate JOIN's ON column on one side (#1564). The failure is loud, never silent. The remedy is DROP TRANSFORM and define again.
  • A source key with INCLUDE columns is taken as keyed by those columns too (#1722). A deferrable primary key is accepted as a table's key (#1742).

v0.1.0 the first real release

Choose a tag to compare

@mmmries mmmries released this 31 Oct 22:07
3ddf4f7

This is the first release of the trellis library and it's embedded clients (trellis-pg) as well as the CLI. In this initial version, we are outlining the structure of the project, it's operating process and a small initial set of features.

What works today?

  • Building 1-to-1 transforms that compute values based on a single record
  • Building aggregated transforms (with a GROUP BY) so you can compute basic values from a group of records in the same table
    • including aggregates across large partitioned tables up to 1,000 leaf nodes
  • A small subset of SQL expression language like sum, count, avg, +, -
  • Strict correctness guarantees that once your transform shows a status of live, it will contain the same data as an equivalent query of its source data for anything other than staged data that is actively being processed
  • Prometheus metrics are available to track the number of source changes processed, a histogram of how long each individual transform takes and an end-to-end latency across all transforms in a given chain

** Validations **

Basic project metrics are published nightly.
This project has a rich set of test and validation harnesses that cover unit tests, integration tests, benchmarks and generative tests.
We capture notes on a subset of these kinds of activities and so far we've done 450+ validation runs, totaling 210+ hours of machine time and 13.6B+ database rows that have transited through Trellis and been verified correct against a live query.
We also have mechanisms built into the tool so you can run sampled self checks to detect any issues.
We keep detailed notes of all known correctness gaps which mostly come down to changes or actions we can't prevent the database owner from taking (like modifying our internal tables).

Full Changelog: The entire git history of up v0.1.0, too much to summarize here

Trellis 0.1.0-gamma

Trellis 0.1.0-gamma Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 29 Oct 17:01
c271ef1

Added the Trellis::Metrics.render_prometheus method to read the prometheus data

What's Changed

  • Ruby binding: Trellis::Metrics.render_prometheus, the process-wide registry as Prometheus text by @mmmries in #1479

Full Changelog: v0.1.0-beta...v0.1.0-gamma

Trellis 0.1.0-beta

Trellis 0.1.0-beta Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 29 Oct 15:31
1820fd7

The trellis CLI, and the precompiled builds of the Elixir (trellis_pg) and Ruby (trellis-pg) bindings. See docs/releasing.md.

Several small changes noted below, but the big change is documented, and smaller connection pools for the Trellis clients (including the embedded clients). Each client now has two connection pools, they each lazily establish connections and they each cleanup idle connections after 60sec:

  • Call pool is used for API calls like define and `status. By default this pool can establish up to 20 connections, but it will remain empty until it receives an API call from the client application
  • Background pool is used by staging and drain threads. It will have a maximum size of 2N + 5 where N is the number of drain threads.

What's Changed

  • Packaging hygiene for the alpha: slim crate package, publish flags, stale test shards, release-binary gate (part of #1425) by @mmmries in #1432
  • Trim the trellis crate's public API; public enums and report structs are #[non_exhaustive] (part of #1425) by @mmmries in #1433
  • Purge references to closed issues and PRs outside ADRs; present-tense docs, metrics and environment tables (part of #1425) by @mmmries in #1435
  • catching_up is a reported status, not a stored one: split StoredStatus from TransformStatus; V8 drops the word from the CHECK and triggers (part of #1425) by @mmmries in #1436
  • self_check compares only a definition whose reported status is live; drop plan-history references (closes #1434) by @mmmries in #1437
  • Document what a handle uses: connections, threads and memory by @mmmries in #1469
  • Fewer idle connections: one LISTEN and heartbeat per client, role-sized pools that close when idle, standby staging workers by @mmmries in #1475

Full Changelog: v0.1.0-alpha...v0.1.0-beta

Trellis 0.1.0-alpha

Trellis 0.1.0-alpha Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 26 Oct 20:07
31985aa

This is the first release of the trellis library and it's embedded clients (trellis-pg) as well as the CLI. In this initial version, we have are outlining the structure of the project, it's operating process and a small initial set of features.

What works today?

  • Building 1-to-1 transforms that compute values based on a single record
  • Building aggregated transforms (with a GROUP BY) so you can compute basic values from a group of records in the same table
    • including aggregates across large partitioned tables up to 1,000 leaf nodes
  • A small subset of SQL expression language like sum, count, avg, +, -
  • Strict correctness guarantees that once your transform shows a status of live, it will contain the same data as an equivalent query of its source data for anything other than staged data that is actively being processed
  • Prometheus metrics are available to track metrics on the number of source changes processed, a histogram of how long each individual transform takes and an end-to-end latency across all transforms in a given chain

** Validations **

Basic project metrics are published nightly.
This project has a rich set of test and validation harnesses that cover unit tests, integration tests, benchmarks and generative tests.
We capture notes on a subset of these kinds of activities and so far we've done 400+ validation runs, totaling 190+ hours of machine time and 13B+ database rows that have transited through Trellis and been verified correct against a live query.
We also have mechanisms built into the tool so you can run sampled self checks to detect any issues.
We keep detailed notes of all known correctness gaps which mostly come down to changes or actions we can't prevent the database owner from taking (like modifying our internal tables).