v0.5.1

0.5.1 brings Arrow to libzpq: answers come back as Arrow C Data streams, with typed columns, exact DECIMALs and no
JSON in between, and a Spark DataSource V2 uses them to run Parquet scans and partial aggregates inside zpq. Files
zpq writes now default to Parquet's version 2 encodings, with --parquet-version v1 for older readers.
--max-memory now counts the memory that aggregation, DISTINCT and strings actually use. Six kinds of query are
faster, the largest source files are split into small modules, and two bugs are fixed.

What to do

• Written Parquet files change bytes. zpq query -o now writes --parquet-version v2 by default: the DELTA
  encodings where they fit and RLE_DICTIONARY, with a version 2 footer. The values and their order are unchanged,
  and DuckDB, PyArrow and Polars read every v1 and v2 output the same way. A pipeline whose readers only understand
  version 1 files adds --parquet-version v1 (PLAIN / PLAIN_DICTIONARY and a version 1 footer). Row groups and
  pages copied untouched from the input keep their own encodings either way.

• --max-memory budgets are tighter. A budget now covers aggregation state, DISTINCT sets and string arenas, so
  a query that fit under a small budget in 0.5.0 by not counting them may now refuse, with exit 1 and a message. Give
  it a larger budget, or leave --max-memory unset.

• make_date(x) is a DATE wherever it leaves, as min/max(make_date(col)) already was: Arrow date32,
  Parquet -o INT32 DATE, and CSV ISO text (2024-01-02 where it printed 19724). JSON still prints the stored
  days.

Nothing else changes for scripts: zpq query prints the same rows, types and errors as 0.5.0 (every row output on
the benchmark matrix is byte-identical), and libzpq's existing functions and status codes keep their meanings.

Highlights

• Arrow C Data results from libzpq (experimental). zpq_run_arrow() returns an answer as an
  Arrow C stream instead of JSON: the same SQL or
  structured requests, the same projection, filters, ordering, limits and aggregates, with typed columns. DECIMAL up
  to 38 digits is decimal128 and exact; DATE, TIMESTAMP, TIME, booleans, strings and binary map to their Arrow
  types; nested types and DECIMALs wider than 38 digits are refused with an explicit error. Batches are owned by
  the caller and may outlive the stream and the context. max_result_bytes caps the exported buffers. It is an
  additive symbol in LIBZPQ_1, and neither Arrow C++ nor Arrow Java is linked into libzpq. With PyArrow
  installed, python/zpq_arrow.py reads the stream as a RecordBatchReader; the reader speaks the Arrow
  PyCapsule stream protocol, so duckdb.register("t", reader), DataFusion and pl.DataFrame(reader) take it
  directly. A computed string (regexp_replace, ||, CASE, coalesce, a text cast) is utf8 as a GROUP BY key or
  min/max too, not only as a row. The full contract and type table are in docs/libzpq.md.
• Aggregate -o writes a computed string as STRING. A GROUP BY key or min/max that the query computes
  (regexp_replace, ||, CASE, coalesce, a text cast) was written as unannotated bytes, which DuckDB reads as BLOB
  and PyArrow and Polars as binary; it is now STRING (UTF8), as the same expression written as a row already was. A
  key or min/max of a column keeps that column's type.
• CAST(s AS VARCHAR) (also TEXT, STRING and VARCHAR(n)) of a string. A cast of a number or date to text
  is refused (type mismatch in expression); a cast to another type is still unknown scalar function.
• A Spark DataSource V2 (experimental). spark.read.format("zpq"), in integrations/spark/, runs Parquet
  scans, filter and column pushdown, and partial COUNT, SUM, MIN, MAX, AVG and GROUP BY aggregates in
  libzpq through Java's Foreign Function & Memory API, and takes rows back as Arrow batches (rowMode=arrow, the
  default). Column names and paths are escaped in the SQL it builds. Its correctness suite runs 15 queries (13
  ClickBench queries, plus exact DATE and DECIMAL ones) through both spark.read.parquet and the zpq source and
  compares every cell. It needs Java 22 or later and Spark 4 (tested with Spark 4.2.1 and Arrow Java 19.0.0).
• --parquet-version v1|v2, v2 by default (see What to do). JSON requests to libzpq and the Lambda take
  "parquet_version": "v1" or "v2". Writes are as fast as 0.5.0's on the benchmark machine.

Performance

Measured against 0.5.0 on the same Zen 5 desktop as 0.5.0's benchmarks, a fresh process for every query:

• Queries that print rows use every core. 0.5.0 decoded, filtered and printed row output on one thread. Row
  groups now decode, filter and format in parallel and print in input order, so the bytes are the same at every -j.
  On ClickBench, SELECT SearchPhrase, URL … WHERE URL LIKE '%google%' went from 4.0 s to 0.68 s (−83%), and a
  2.1 GB WHERE SearchPhrase <> '' from 8.8 s to 1.3 s (−86%). It covers NDJSON, JSON, CSV, libzpq JSON and Arrow
  results. Without --max-memory each worker keeps its decode buffers, so peak memory on that LIKE query roughly
  doubles (2.9 to 5.9 GiB, including the mapped file); with a budget, the workers share half of it.
• ClickBench q9 (the top 10 regions by count, each with COUNT(DISTINCT UserID)) ranks the regions by count
  first, then counts distinct users for the 10 winners only: 18.6% faster cold, 27.5% faster hot.
• High-cardinality integer GROUP BY … SUM top-k pre-aggregates each worker's rows into radix partitions before
  the merge: 32–40% faster on the study's zipf11 and nullszipf q14.
• ClickBench q39 and its neighbours split large row groups into subgroups that several workers read ahead in
  parallel: q39 is 48% faster cold and 62% faster hot, and q36 and q37 gained 43–52% in a smaller run. Small or
  highly filtered work stays in one piece, so q38 is unchanged.
• Grouped integer COUNT(DISTINCT) uses exact, bounded bitmaps when the groups and values are known from the
  input's dictionaries: 6.5–7.3× faster on the study's grouped DISTINCT cells (card10 and card1k q09).
• Scalar COUNT(DISTINCT) over strings works on dictionary codes and compares only the distinct strings:
  ClickBench q5 (COUNT(DISTINCT SearchPhrase)) is 17–20% faster.
• Top-k by COUNT over sorted integer keys streams each worker's runs instead of building a full group table:
  2.3–2.5× faster on the study's sorted10m q01.

Across all 464 benchmark queries, hot-fresh, 0.5.1 is 1.6% faster than 0.5.0 by geometric mean (measured on the
release candidate; the 48 queries its last fixes could touch were re-timed on the release build). Against the other
engines, hot-fresh, zpq is faster than DuckDB 2.0.0 on 446 of the 464 queries and faster than Polars 2.1.0 on 456
of the 461 it answered correctly (its answers to 3 were incorrect), and it is never more than 1.5× behind the faster
of the two. Every answer is unchanged: the same rows in the same
order at every -j and memory budget.

Memory: --max-memory

--max-memory now charges aggregation state, DISTINCT sets (including the capacity of their string arenas) and
string storage against the budget, where 0.5.0 left them out. In the accounting work's test, a query with a 256 MiB
budget that peaked at 574 MiB now peaks at about 284 MiB, or refuses cleanly. It is still a working budget, not a
hard cap: the binary, thread stacks and mapped input pages sit outside it, so for a hard ceiling run zpq under a
cgroup memory.max or ulimit -v. Honouring a budget costs time: queries with an explicit budget are 5.7–8% slower
on the benchmark cells, and queries without --max-memory are as fast as before or faster.

Fixed

• A grouped COUNT(DISTINCT) could answer one row for many groups. For a structured request (libzpq, the
  Lambda or the fan-out coordinator), the planner decided "is this grouped, is it filtered" from SQL text, which
  structured requests do not have. One case built a
  single-row answer for a query with 40,000 groups. The planner now reads these from the bound query plan. SQL
  through the CLI was not affected.
• When an allocation was refused while zpq built an answer's typed columns (by --max-memory or by the system),
  the clean-up after the refusal could free memory the column did not own. Each column is now valid before it is
  published, so the query refuses cleanly.

Tests and release tooling

• Flaky tests fixed: the benchmark tool's deadline test (now 20 of 20 under load, from 3 failures in 15), the
  multipart-upload tests' race with the fake server's clock (50 of 50 under load), and the allocation-failure
  sweep's four-worker determinism check (20 consecutive green sweeps).
• An opt-in allocation-failure sweep (ZPQ_OOM_SWEEP=1) runs representative library queries with an allocator that
  fails on a schedule, and checks every refusal's clean-up and every answer against stable results.
• The release workflow resolves its checkers from the invocation directory and names a failed spawn, gives macOS
  runs GNU timeout, exercises the Intel macOS archive on a native runner, has a configurable build timeout, and
  can rehearse a release without publishing it.
• The two failures in 0.5.0's release matrix are fixed: the units-core fixture crash on one Linux test machine,
  and the OS-contract checks on Python 3.14.

Internal

• The four largest source files are split into directories of small modules: src/core/scan/ (from 26,800 lines),
  src/engine/ (13,900), src/core/consumer/ (11,800) and src/core/expr/aggregate/ (8,300); each old file is now
  a root of 85–240 lines. No behaviour changes.

Known

Three small queries are slower than in 0.5.0, each by under 3 ms (fastest of five runs each); they are kept for
0.5.2:

• the study's sorted100k q03, cold: about +10% (21.8 ...

zpq 0.5.0

Read "What to do" first if you script zpq or rely on its answers. 0.5.0 changes what zpq query prints, fixes
answers that 0.4.1 got incorrect without an error, and turns --max-memory into a real ceiling. It adds a C library
(libzpq) with a Python binding, zpq inspect, a much larger SQL surface, Bloom filters, profiling, and Homebrew,
shell-completion and man-page packaging.

Parquet answers to an aggregate or GROUP BY are written like any other zpq output: row groups of 131,072 rows and
pages of 20,000 rows. A large answer file therefore has different bytes than in earlier releases, with the same
values in the same order, and its bytes still never depend on -j or the memory budget.

Top-count and top-sum GROUP BY queries over string and integer keys run on new, faster exact paths, with or without
--unordered. Their answers do not change: NULL groups, full key bytes and first-appearance ties are kept, and the
output is the same at every -j, memory budget and cache size.

What to do

1. Scripts that parse zpq query output must change. Every query now prints its answer as NDJSON rows on stdout,
   and nothing else. The {"ok":true,...,"agg":...} summary that 0.4.1 printed is gone from stdout; take the rows
   instead, or add --summary=json to get a typed summary line on stderr. docs/output-format.md (shipped in
   the release archive) is the contract for the shapes and for how each type prints.

2. BREAKING: avg(x) is now a plain number, in every format. 0.4.1 answered it as a {"sum":..,"count":..}
   object in JSON (a DECIMAL one with an "avg" field too) and as <name>__sum / <name>__count columns in CSV and
   Parquet; the Lambda, libzpq and Python answers carried the same object. It is now one value, NULL over no values:

   • a DOUBLE for integer and floating x, the sum divided once by the count;
   • for DECIMAL(p, s), DECIMAL(q, s + q - p) with q = min(38, p + 4): the exact mean rounded half away from zero,
     with up to four more fractional digits and never fewer integer digits (DECIMAL(15,2) gives DECIMAL(19,6),
     DECIMAL(36,35) gives DECIMAL(38,37), and p = 38 keeps the input type, as Trino does). It never overflows.

   If you merge averages across files or Lambda fan-out, ask for avg_state(x) instead: it answers exactly what
   avg(x) answered in 0.4.1 (minus the DECIMAL "avg" field), and sum(<name>__sum) / sum(<name>__count) over its
   per-file results is the exact overall average. A query that keeps avg(x) and parses __sum / __count or the
   object breaks.

3. If you read zpq's CSV, check dates and empty strings. --format csv works as in 0.4.1, but DATE, TIME and
   TIMESTAMP now print as ISO text (0.4.1 printed integers), and an empty string prints as "" so it differs from
   NULL (0.4.1 printed both as an empty field). Aggregate and GROUP BY answers can now be CSV too.

4. Re-run the queries listed under "Answers that changed" if you relied on their 0.4.1 results: DECIMAL sums and
   filters, UUID filters, NOT IN lists containing NULL, LIKE NULL and LIKE with _, coalesce, integer
   arithmetic near the 64-bit limit, INT96 timestamps, float filters and min/max over columns that hold NaN, BOOLEAN
   GROUP BY keys, and any Parquet
   file 0.4.1 rewrote from a fixed-width binary (UUID), FLOAT16 or BYTE_ARRAY DECIMAL column, or from inputs with
   different DECIMAL scales.

5. Check --max-memory values you pass. It used to bound GROUP BY entries only; it now bounds everything an
   aggregate or a write holds. A query that answered at a given value can now refuse instead, with
   the query needs more memory than --max-memory allows (exit 1). Raise the value, or leave it unset.

6. Expect GROUP BY rows in a different order. Groups now come in the order they first appear in the input, at
   every -j; 0.4.1 sorted them by their serialized key bytes. Add ORDER BY if you need an order.

7. Invalid queries exit 2, not 1. Read, decode and write failures stay 1; a closed stdout pipe is 141. An
   aggregate or a GROUP BY key over a list's elements (a repeated leaf) is now refused while planning, exit 2, with a
   sentence naming the column (aggregate argument column l.list.element is repeated), as a filter on one already
   was.

8. BREAKING for summary readers: rows_kept is gone. It meant rows in the row groups read for an aggregate, but
   rows written for a write, so a filtered aggregate reported every row of the row groups it read as kept. The
   --summary=json line, and the Lambda's and libzpq's responses, now carry rows_in (every input row),
   rows_scanned (rows in the row groups read) and rows_matched (rows the filter kept: what an aggregate folds, a
   write writes, a row output prints before ORDER BY and LIMIT), beside rows (rows in the answer). The Lambda write
   response's rows_kept is rows_matched. A row output's rows_in now counts every input row, not just the row
   groups reached before a LIMIT.

9. BREAKING for summary readers: phase is now wall_ms and worker_ms. wall_ms holds wall-clock phases
   (open, parse, scan, footer; a Lambda write adds mp_await, mp_complete), which run one after another.
   worker_ms holds each scan step summed over every worker (decode, filter, fold for aggregates, encode for
   writes, sink), so with N workers one can reach N times wall_ms.scan. wall_ms.scan is now measured for
   writes and row output too. The field names lost their _ms suffix inside the objects. docs/output-format.md defines every field.

10. Expect a different layout in files zpq re-encodes (any filter, --select or computed column). Output row
    groups no longer follow the input's: they hold up to 131,072 rows or 64 MiB of uncompressed values
    (--row-group-size, --row-group-bytes), so a selective filter over many small row groups writes a few large
    ones. Chunks are split into data pages of at most 1 MiB or 20,000 rows, with a page index. The bytes written depend
    only on the input and the options, never on -j or the machine's memory; an explicit --max-memory caps row
    groups at a sixteenth of it. Plain copies keep their input's row groups.

11. On an older x86-64 CPU, take the -compat archive. The default Linux x86_64 archive (and the .deb, .rpm
    and Homebrew packages built from it) targets x86-64-v3, which needs AVX2. On a CPU without one of its features it
    prints zpq: this CPU lacks <feature>, which this x86-64-v3 build needs; use the x86_64-linux-musl-compat archive, which runs on any x86-64 and exits 3 before doing any work.

Highlights

• Answers on stdout, ready for Unix tools. NDJSON rows by default for every query kind, --format json for an
  array, CSV, Parquet to a file or to stdout. Local outputs are published atomically.
• Exact by default. DECIMAL computes in scaled 128-bit integers, integer arithmetic fails instead of wrapping, and
  every aggregate (float sums included) gives the same bits at every -j and on every run. Floats compare in one
  total order on every path, filters and pruning included, so NaN rows are never dropped. From Python, DECIMAL values
  arrive as decimal.Decimal.
• A memory ceiling that holds. --max-memory is one budget per query that tables, decoding, read-ahead, Top-N, the
  merge, the answer and write buffers all draw from; a query that does not fit is retried on narrower paths and
  otherwise refused, never killed.
• Faster GROUP BY, from compact group state, cached group lookups, private partition runs for high-cardinality
  keys, one shared table for long string keys, and parallel finishes.
• Filters that decode less: an AND runs its cheapest conditions first, and a later condition's column is decoded
  only where the earlier ones kept rows.
• libzpq: the engine as a C library with a Python binding (python/zpq.py) and an HTTP service. SQL through it is
  EXPERIMENTAL.
• All 43 ClickBench queries run, with make_date, epoch_ms, date_trunc, date_part/EXTRACT and
  regexp_replace.
• SQL: ORDER BY, LIMIT/OFFSET, HAVING, count(DISTINCT ...), CASE, strlen, GROUP BY by expression,
  position or alias, BOOLEAN columns as predicates, and parenthesized filters with NOT.
• zpq inspect reports footer metadata without decoding data pages.
• Lossless JSON for binary, UUID, INTERVAL, BSON and GEOMETRY values, and typed BOOLEAN answers.
• Bloom filters read, written and used to prune = and IN.
• Profiling with --profile (a Perfetto trace) and --profile-summary.

Platforms

• CLI release builds: Linux x86_64 and arm64 (static musl), macOS arm64 and x86_64 (macOS 15 or later).
• Linux x86_64 builds target x86-64-v3 (AVX2); the -compat archive runs on any x86-64. On a CPU without a v3
  feature the default build names it and the compat archive, and exits 3, instead of dying with SIGILL.
• Lambda: zpq-lambda for provided.al2023, arm64 and x86_64. The Lambda build has no SQL; a SQL request answers
  SqlNotCompiledIn.
• libzpq: Linux x86_64 and arm64 (glibc 2.28 or later, or musl) and macOS arm64 and x86_64; the macOS libraries are
  cross-built and not run. Windows is not built.
• snappy's NEON paths are enabled on arm64.
• Release binaries are stripped, which makes them much smaller, and they carry no paths from the machine that built
  them. A build is reproducible: the same commit and flags give the same bytes, whatever directory they were built
  in. The Lambda zip is written the same way every time.
  • The x86_64 Linux CLI is 8.1 MB.
  • The Lambda bootstraps are 6.0 MB (arm64) and 7.3 MB (x86_64), against 44.8 and 47.2 MB unstripped.
  • Local builds keep their symbols unless built with -Dstrip=true.
• macOS: the macOS build carries no Linux-only system calls, and its fetch workers wait through libSystem.
• Building from source...

v0.4.1

zpq 0.4.1

Fixes incorrect results from 0.4.0. Upgrade if you use zpq query --query with NOT.

zpq 0.4.0 could return incorrect results, with no error, for a SQL WHERE clause (or an aggregate FILTER (WHERE ...))
that applies NOT to a parenthesized AND/OR. For example, WHERE NOT (x > 5 AND y < 3) was evaluated as
WHERE (NOT x > 5) AND y < 3. The --filter flag was not affected, and neither was SQL without NOT over a group.

What to do: upgrade to 0.4.1, and re-run any 0.4.0 --query whose WHERE or FILTER applies NOT to a parenthesized
group. Until you can upgrade, rewrite such a predicate without the outer NOT (NOT (a AND b) is NOT a OR NOT b;
NOT (a OR b) is NOT a AND NOT b).

In 0.4.1, NOT is pushed down to the comparisons, keeping SQL's NULL semantics. Since the filter grammar is a flat OR
of ANDs, a predicate that still needs grouping after that (e.g. NOT ((a AND b) OR c)) is refused with an error
instead of evaluated differently.

0.4.1 also refuses SQL conditions that 0.4.0 misread and could answer incorrectly. Each condition must now be
column op literal, LIKE (without ESCAPE), IS [NOT] NULL, or IN/BETWEEN over literals. Refused are: NULL
as a value (0.4.0 compared against the string 'NULL', so s NOT IN ('a', NULL) returned rows where SQL returns
none), another column, an expression, CAST or COLLATE as a value (0.4.0 compared against their text), and a string
literal or column name containing a quote, AND, OR or BETWEEN (or a comma, inside an IN list).
x NOT BETWEEN lo AND hi now works in SQL, and a OR (b OR c) is accepted instead of refused.

Full Changelog: v0.4.0...v0.4.1

zpq 0.4.0

zpq 0.4.0

Please upgrade to zpq 0.4.1. zpq 0.4.0 can return incorrect results, with no error, for a SQL WHERE or aggregate FILTER (WHERE ...) that applies NOT to a parenthesized AND/OR, and for a few other predicate shapes (LIKE ... ESCAPE, NULL inside an IN list). Details are in the 0.4.1 notes.

zpq 0.4.0 moves to Zig 0.17.0. It adds macOS builds, fixes TLS hostname verification, and makes many correctness fixes. On AWS Lambda it is built around one use: cold, bursty invocations.

Built for cold starts on AWS Lambda

zpq-lambda is a single static binary for the provided.al2023 runtime, and starting cold is what it is designed for. We measured on 2026-10-05 in us-west-2, with every function at 3008 MB:

• Init time. zpq 0.4.0's median Lambda Init Duration was 18–22 ms on arm64 and 26–35 ms on x86_64 across eleven read, aggregate and S3-to-S3 write scenarios, three forced cold starts each. DuckDB 1.5.6, run as a Python 3.12 container image, measured 448–832 ms. Polars 1.44.2, packaged the same way, measured 478–806 ms.
• Total cold time. zpq's cold Init Duration + Duration was lower than DuckDB's in every scenario. On x86_64, DuckDB took 2.4× to 5.6× as long. Two examples: a range query on TPC-H lineitem (SF1) took 26 + 204 ms against 535 + 762 ms, and filtering an 868 MB Overture Places file and writing the result to S3 took 35 + 1,726 ms against 738 + 4,063 ms.
• Memory. zpq 0.4.0's highest Max Memory Used per scenario was 38–449 MB on x86_64. DuckDB's was 189–709 MB.

The full tables and method are in docs/measurements/lambda_cold_start_2026-10-05.md.

Because initialisation costs so little, the intended pattern is cold, bursty and fanned out: one invocation per file, partition or compaction unit, fired when the work arrives. Keeping sandboxes warm, whether with keep-warm pings or with provisioned concurrency bought to hide initialisation, is not the intended use. Lambda still reuses sandboxes on its own, and 0.4.0 handles that better:

• pooled S3 connections idle for more than 4 s are closed rather than reused, because they do not survive a frozen sandbox;
• a reused connection that turns out to be dead is retried at once on a new one;
• a saturated worker pool delays read workers instead of failing the query with ConcurrencyUnavailable;
• at 3008 MB, zpq now uses both vCPUs.

New: S3_NO_SIGN_REQUEST=1 sends unsigned requests, so zpq can read buckets that allow public access without credentials (a region is still required).

Behaviour changes to know about

• NULL results. sum, avg, min and max over no non-null values return NULL, not 0 or ""; count stays 0. In -o output, sum, avg, min and max columns are now OPTIONAL.
• Stricter CLI.
  • Unknown or empty --columns is an error.
  • Duplicate output column names are an error.
  • Arithmetic and functions over a UINT64 column are rejected rather than wrapped, since values of 2^63 and up do not fit.
• Copies. A plain copy keeps each chunk's codec and reports it, or MIXED when chunks differ. --codec lz4 means LZ4_RAW.
• Lambda requests. Events are parsed as strict JSON. The newly rejected shapes are listed in docs/lambda_requests.md. Aggregate responses now include row_groups_full_match and a pool statistics object; the write response's pool gained idle-eviction and retry counters.
• Endpoints.
  • Dotted AWS bucket names use path-style addressing.
  • IPv6 endpoints and non-canonical IPv4 spellings (such as 127.1) are rejected.
  • Custom endpoints need a certificate that names the host.
• Stricter files. Footers that no valid writer produces are rejected. So are DELTA miniblocks whose size is not a multiple of 32.
• Platform. Building requires Zig 0.17.0. The macOS binaries require macOS 15 or later.

Security

• TLS hostname verification. Earlier releases accepted any valid certificate, whatever name it was issued for. Now:
  • certificates must name the host;
  • a wildcard matches exactly one label;
  • IP endpoints must match an IP SAN;
  • the certificate's common name is never used.
• Related hardening.
  • SNI carries the bare host name, and none is sent for IP literals.
  • Truncated or corrupt TLS streams are always errors.
  • Lost multipart Create and Complete responses are cleaned up, or confirmed by ETag.

macOS

• The CLI now ships for macOS on Apple silicon and Intel.
• File I/O and sockets are portable, and the macOS binaries make no raw Linux system calls.
• DNS lookups for S3 now work on macOS.

Correctness

• Decoding. FIXED_LEN_BYTE_ARRAY on V2 data pages now decodes, and so does the deprecated LZ4 codec, whether a writer framed it the Hadoop way, as a bare block or as an LZ4 frame.
• Statistics.
  • Unsigned integers filter, prune and print as unsigned.
  • Deprecated min/max statistics are used only where signed order is correct.
  • Non-standard column orders turn pruning off.
  • Float != respects NaN.
  • A page index that contradicts the footer is ignored.
  • IS NULL no longer undercounts on all-null pages.
• Query semantics.
  • Temporal literals compare exactly at any precision.
  • BETWEEN binds like the comparison operators.
  • Mixed integer/float arithmetic in GROUP BY keys is correct.
  • A grouped -o write produces one row per group and keeps column types.
• Names. Bare names bind to top-level columns, and dotted paths to nested leaves. Colliding output names are disambiguated or rejected.
• Output. --codec is honoured on byte-copied chunks. Lambda JSON is escaped, and invalid UTF-8 becomes U+FFFD.
• Robustness.
  • Malformed files produce clean errors instead of crashes.
  • Page value counts are capped, so a crafted file cannot exhaust memory.
  • Varint reads are bounded everywhere.

Performance

We measured against 0.3.2 on 2026-10-05:

• Local files. On the same workstation, using 16 CPUs on warm local files, 0.4.0's engine time was 0.38× to 0.80× of 0.3.2's across 14 read-and-aggregate scenarios (median of 14 interleaved rounds). The gains come from:
  • per-worker buffer reuse;
  • skipping filter-only columns in row groups whose statistics show that every row matches;
  • a faster DELTA_BINARY_PACKED decoder.
• Lambda, warm after an idle gap. This is not the intended pattern, but it is where 0.3.2 struggled most. On x86_64 at 3008 MB in us-west-2, 0.3.2's median Duration was 3.5–5.7 s; 0.4.0's was 0.1–1.9 s on the same scenarios. Peak memory fell from 336–910 MB to 38–449 MB.

The cold-start comparison with DuckDB above is the figure we would point to first. Two caveats:

• A single Lambda sandbox's network throughput limits full-file scans for every engine.
• Warm, Polars was faster than zpq x86_64 in 9 of the 11 scenarios.

v0.3.2

What's Changed

• v0.3.1 — aggregate scan perf, bounded remote read workers by @chicagobuss in #12
• v0.3.2 — correctness hardening, test coverage, and benchmark protocol by @chicagobuss in #13

Full Changelog: v0.3.0...v0.3.2

v0.3.0

Full Changelog: https://github.com/chicagobuss/zpq/commits/v0.3.0