v2.2.0-rc0: Hey folks - XTDB **2.2.0-rc0** is here. 🚀

Hey folks - XTDB 2.2.0-rc0 is here. 🚀

Building on the multi-database foundation from 2.1, this is a big one: XTDB can now source data from systems you already run, speaks Arrow ADBC & Flight SQL end-to-end, elects leader nodes to share cluster load, and picks up a good deal more SQL surface along the way. Here's the tour:

External sources - bring your data to XT without migrating to it

XTDB can now source transactions from systems you already run, rather than requiring everything to be submitted through an XT node.

Two built-in sources ship in 2.2:

• PostgreSQL source - XTDB connects to your existing Postgres cluster as a logical-replication client and mirrors changes in as bitemporal history. (setup guide)
• Kafka source, interoperable with existing Kafka Connect transformers and predicates, so XT sits naturally in a Kafka Connect topology. (setup guide)

Setting one up is an ATTACH DATABASE away, pointing a secondary database at the upstream:

ATTACH DATABASE pg_test_db WITH $$
  log: !Local
    path: 'pg_test_db/log'
  storage: !Local
    path: 'pg_test_db/storage'
  
  externalSource: !Postgres
    remote: pg_remote
    publicationName: xtdb
    slotName: xtdb
    indexer: !DirectMirror {}
$$

See the external sources overview for the full picture.

This means that you don't have to commit to XTDB as your primary store to get its value. Keep your existing database, run XT alongside as a secondary, and get bitemporal audit and time-travel reporting over data you're already producing. Or hang XT off your event-sourcing infrastructure as a time-oriented projection.

Under the two built-in sources sits the same pluggable source SPI we use to build them (in xtdb.api.tx) - so you can write your own external source against whatever upstream you like. It's a supported part of the public API from 2.2. If you build one, we'd love to see it - please do get in touch.

There's also a first-class ingest-only node for running sources independently of your query nodes: xtdb.main ingest -f config.yaml (or the Docker ingest command) starts one from YAML config - external sources, indexers, logs and caches all configurable - no need to embed it in your own JVM app or stand up a full query node. (docs)

Arrow ADBC & Flight SQL

XTDB now supports Arrow ADBC in-process and a Flight SQL server for remote access - a database client API in the spirit of JDBC/ODBC, but built around Apache Arrow's columnar format throughout.

Why this is a particularly natural fit for XT: we store everything internally as Arrow. Queries through the Postgres wire-protocol ("pgwire") pay a hidden tax of translating out of Arrow into pgwire's row-oriented types, only for the client to translate back into something columnar (a pandas DataFrame, Polars frame, DuckDB table, Arrow Dataset). With ADBC and Flight SQL, that round-trip goes away - results stay in Arrow from XTDB's storage all the way into your application's memory. The practical upshot: faster bulk reads for analytical workloads, and a much closer fit for the modern data-science stack (pandas, Polars, DuckDB, dbt).

From Python, over the wire via the FlightSQL driver - bulk-ingest an Arrow table in one round trip, and get results back as Arrow batches:

import adbc_driver_flightsql.dbapi as flight_sql
import pyarrow as pa

with flight_sql.connect("grpc://localhost:9832") as conn:
    with conn.cursor() as cur:
        # Bulk-ingest an Arrow table: one round trip, no row shredding.
        users = pa.table({
            "_id":  ["alice", "bob", "carol"],
            "name": ["Alice", "Bob",   "Carol"],
            "age":  [30,      25,      28],
        })
        cur.adbc_ingest("users", users, mode="create_append")

        # Query it back: results stream straight back as Arrow.
        cur.execute("SELECT _id, name FROM users WHERE age > ?", parameters=[26])
        print(cur.fetch_arrow_table().to_pydict())

From Kotlin, in-process - node.connect() returns an org.apache.arrow.adbc.core.AdbcConnection, so results stay in Arrow with no network hop and no copy:

Xtdb.openNode().use { node ->
    node.connect().use { conn ->
        val stmt = conn.createStatement()
        stmt.setSqlQuery("SELECT _id, name FROM users WHERE age > 26")
        stmt.executeQuery().use { result ->
            // result.reader is an org.apache.arrow.vector.ipc.ArrowReader
        }
    }
}

If you've been waiting to point an ADBC-native client at XTDB - PyArrow's Flight SQL client, Python's adbc-driver-flightsql, or the JDBC Flight SQL bridge - this is the release where it works end-to-end. The full supported surface is in the ADBC reference.

A shout-out too to the folks at Columnar - founded by core Apache Arrow / ADBC developers - whose CLI dbc takes the friction out of installing ADBC drivers. We're aiming to land an XTDB driver in dbc in due course.

Elected leader nodes & read-only secondaries

Since XTDB 1, all nodes consumed the same transaction log and relied on deterministic processing to converge on the same state. In 2.2, following multi-db in 2.1, XT now elects a leader node per database to share the processing load across the cluster. Largely a behind-the-scenes change - you should notice an overall drop in load on high-write-throughput clusters - but the main thing is what it unlocks:

• Read-only secondary nodes - nodes that don't contend for leadership, they simply read the leader's results. This opens the door to XTDB as a federated database: connect to and query across other XTDB instances in your company, just by opening read-only access to their log and object stores.

Under the hood this relies on Kafka's consumer groups (for Kafka-based logs), extended for the consistency guarantees we require. See the log configuration docs for how to upgrade.

PostgreSQL-ecosystem compatibility

One of the main reasons we went with SQL + Postgres wire-compatibility in XT 2.x was to make the vast Postgres tool/library ecosystem available to XTDB users. 2.2 widens that compatibility considerably - Metabase and Grafana both now connect over their standard Postgres connectors.

New SQL functions and operators BI tools and Postgres-native libraries expect:

• Window / aggregate: LEAD, LAG, PERCENTILE_CONT, PERCENTILE_DISC (handy for medians); MIN/MAX now work on variable-width values like strings.
• String / identifier: parse_ident, quote_ident, string_to_array, current_setting.
• Array: array_lower, array_length.
• Predicates: IS [NOT] DISTINCT FROM.
• Generators: RANGE as an exclusive-end alias for GENERATE_SERIES; GENERATE_SERIES is now end-inclusive (breaking), matching Postgres.

We've also extended where expressions can go - GROUP BY accepts expressions and select-list aliases (e.g. GROUP BY DATE_TRUNC('day', created_at)), subqueries can appear in ORDER BY, and function calls / arbitrary expressions can appear in FROM.

And more transaction/DDL surface:

• COMMIT [SYNC | ASYNC] - choose whether a commit blocks until durable or returns as soon as it's accepted.
• CREATE TABLE with columns and empty CREATE TABLE; reads of unknown tables now error (see breaking changes).

On the pgwire side, compatibility fixes BI tools care about: comment-only queries no longer error, current_setting('server_version_num') reports PG 16 (Grafana stops complaining), Metabase CSV export returns the full result set, and implicit UTF-8-to-temporal coercion means WHERE created_at > '2026-01-01' does the obvious thing.

We've also modelled much more of the pg_catalog surface (pg_type, pg_namespace, pg_index, pg_attrdef, pg_user, and friends), so the catalog-introspection probes Metabase, Grafana and pgjdbc fire on connect now plan and return sensible results rather than erroring - including cross-catalog joins like Metabase's enum-type discovery.

Garbage collection

XT's primary index is a distributed LSM tree on object storage, which periodically merges ('compacts') its files. 2.2 adds garbage collection: a background process that removes the old files superseded by compaction, keeping read performance and storage costs in check, with trie- and block-level GC metrics for monitoring. For how the LSM tree works, see the "Building a Bitemporal Index" trilogy, part 3.

Operations & observability

With XTDB in production across a number of client sites, 2.2 puts real work into operability:

• Tracing: OpenTelemetry tracing now covers both transactions and queries, toggleable independently so you don't pay for the noise of one while investigating the other.
• Metrics: plenty for your Prometheus dashboards - per-database lag and message-ID, leader/follower status, end-to-end pgwire latency, block-upload times, allocator memory, GC metrics, plus Postgres-source and Kafka-connector metrics.
• EXPLAIN ANALYZE: scan operators now report rows / pages / files pruned vs. used, so you can see at a glance which work the planner elided.

Elsewhere (including breaking changes)

• (breaking) CREATE TABLE now takes explicit columns, and reads of unknown tables error instead of silently returning empty - catches typo'd table names.
• (breaking) Internal-schema tables - pg_catalog, information_schema, xt, and pg_-prefixed tables - now resolve strictly, exactly like user tables (previously some references were silently tolerated). Mostly affects hand-written catalog-introspection queries.
• (breaking) UPDATE skips writes that would produce identical document versions, reducing write amplification on idempotent updaters.
• (breaking) CREATE USER / ALTER USER and the !UserTable authenticator have been removed. Use !UserList for statically-configured users; XTDB_PASSWORD configures the default !SingleRootUser for local dev; OIDC (from 2.1) is the production recommendation.
• GRANT/REVOKE role membership is now expressible as SQL DML - groundwork for role-based authorization, though roles aren't wired end-to-end yet, so there's more to come here in a later release.
• (breaking) Docker images build from a single parameterised Dockerfile - the jar is always xtdb.jar, config always config.yaml. Update custom Dockerfiles / init scripts / volume mounts pointing at old paths.
• (breaking) Logging migrated from Logback to Log4j2, defaulting to structured JSON output.
• (breaking, Kafka log) truncated-log handling: XT now surfaces OffsetOutOfRange on a truncated log, and seeks to the topic beginning when there's no prior anchor, rather than failing opaquely.
• ATTACH DATABASE with credential-less !S3 storage now resolves via the default AWS credentials chain.
• Query correctness fixes: LEFT JOIN no longer drops left rows once the left table's block is finished; unmatched build rows are preserved when the probe schema is empty; ANY/ALL over an array column now resolves correlated references; scans no longer keep pages when no temporal columns are projected.
• WHERE _id IN (...) on non-UUID _id columns is now ~10–60× faster; table-block files are smaller (dropped an unused per-trie iid_bloom and a redundant table_name).

Upgrading from 2.1

Most of 2.2 is a drop-in upgrade, but a handful of the breaking changes above need action before you upgrade:

1. Authentication - CREATE USER / ALTER USER and the !UserTable authenticator are gone. Move statically-configured users to !UserList, set XTDB_PASSWORD for the local-dev root user, or use OIDC (our recommendation for production). See the authentication docs.
2. Logging - we've moved from Logback to Log4j2, defaulting to structured JSON. If you had a custom logback.xml, you'll need to port it to Log4j2. See setting the logging level.
3. Docker - the jar is now always xtdb.jar and the default config always config.yaml. Update any custom Dockerfiles, init scripts or volume mounts pointing at the old variant-specific paths.
4. GENERATE_SERIES is now end-inclusive, matching Postgres. If you relied on the old exclusive-end behaviour, switch those calls to RANGE.
5. CREATE TABLE / unknown tables - tables now need explicit columns, and querying a table that doesn't exist errors rather than returning empty. Make sure your schema's created before you read from it.

If you need to roll back to 2.1, you can - the storage format is unchanged, so a downgrade is safe; just revert the config changes above (logging, Docker paths, auth) alongside it.

---

As always, the milestone has the full list of closed issues and merged PRs. A big thanks to everyone raising issues, contributing reproductions, benchmarks and code - and to our clients and design partners, whose feedback shaped most of the above. It's massively appreciated 🙏

If you've any questions or thoughts, please do get in touch - we'd love to hear from you!

• Discord
• hello@xtdb.com

Cheers!

James & the XT Team