Repository navigation
Releases
v0.3.1 — dependency security fixes
Dependency fixes only; no behaviour changes. Upgrading from v0.3.0 is a binary swap.
What changed
- A lockfile you can build from. The lockfile had drifted from the manifests since August, so
cargo --lockedrefused it, and v0.3.0's binaries were built with dependencies resolved at build time. v0.3.1 is the first release built from a committed, checked lockfile. - rustls 0.23.46 fixes four TLS advisories:
- TLS 1.3 handshake messages accepted across encryption levels;
- secrets in a session's
Debugoutput; - unbounded memory in brotli certificate decompression;
- a ServerHello that poisons the per-name key-exchange record.
- arrow and parquet 58.4.1, which moves thrift from 0.17 to 0.23. quinn-proto 0.11.16. Together they clear both open Dependabot alerts.
- docs/arrow-versions.md: parket uses Arrow 58 throughout. The doc explains why moving to deltalake 1.x, which is on Arrow 59, waits on connector_arrow.
Downloads
- Static binaries (musl) and glibc builds for x86_64 and aarch64 are attached below, with
checksums-sha256.txt. - Container image:
ghcr.io/oonid/parket:0.3.1, which is also:latest.
Pull requests
- #7 Bring Cargo.lock in sync and update rustls to 0.23.46
- #8 Fix the quinn-proto and thrift advisories; record the Arrow version split
- #9 Release v0.3.1
Full changelog: v0.3.0...v0.3.1
v0.3.0 — archive mode
Archive mode, and safety checks that change how some runs end. A minor version: archive mode is new, and a two-stream sync that used to commit an inconsistent rewrite now stops before committing.
Highlights
-
Archive mode. A table set to
TABLE_ARCHIVE_<t>=<scope>is never synced. It is copied to Delta one unit at a time, so the Delta copy can outlive the source rows. Three commands work on units:--archive-unitsappends each unit in one commit, stamped with_archive_unit. Timestamps are stored as UTC, with the source's zone as metadata.--verify-archive-unitscompares a unit with the source row by row.--replace-archive-unitsrepairs a unit by replacing exactly its files.
A library API serves callers that choose the units themselves. A scope may use the unit id more than once (
id >= ? * 10000 AND id < (? + 1) * 10000), so a table without a unit column can be archived in ranges of its primary key. See config.md and cli.md. -
Two-stream upsert overwrites check their row accounting before committing. Kept plus replaced rows must equal the rows removed, and the staged rows must equal kept plus new rows. Otherwise the run stops and the live table is untouched.
-
A Drift verdict compares values per key bucket and names the buckets that differ. It reports only.
-
Every remedy that offers a full refresh or
TABLE_RECONCILEsays what it costs: the Delta table is replaced by the source's current rows, dropping rows the source has deleted or purged. -
--vacuumtells files the log names apart from files present in storage. Its byte totals were already right.
Upgrading from v0.2.6
- Run
parket --checkwith the new binary. - Run one supervised sync.
- Keep v0.2.6 beside it for a one-step rollback.
A sync that now stops on the row-accounting check has left the table as it was. Its message says what did not add up.
Downloads
- Static binaries (musl) and glibc builds for x86_64 and aarch64 are attached below, with
checksums-sha256.txt. - Container image:
ghcr.io/oonid/parket:0.3.0, which is also:latest.
Pull requests
- #1 Pre-archive safety: overwrite warnings, upsert row accounting, bucketed Drift value check
- #2 Describe tooling and review in neutral terms
- #3 Archive mode: append, verify and repair units of archive-only tables
- #4 Use checked_div for the per-row memory estimate
- #5 Let a scope use the unit id more than once
- #6 Release v0.3.0
Full changelog: v0.2.6...v0.3.0