Skip to content

sluice v0.99.251

Choose a tag to compare

View all tags
@github-actions github-actions released this 15 Jul 18:40
· 2056 commits to main since this release
v0.99.251
0fc7f65

New: sluice backup export-as-parquet — the analytics exit surface (ADR-0164, roadmap item 63). One-shot, read-only transcode of an existing backup into one zstd Parquet file per table plus a parquet_index.json export manifest — for ANY source engine's chain (MySQL, PlanetScale, Vitess, Postgres, SQLite, D1 — the export reads the engine-neutral chunk format, not any engine), with restore-grade integrity end to end and a faithful-or-loud type-mapping contract. DuckDB is the companion cookbook recipe (docs/cookbook/duckdb-on-sluice-backups.md), never a dependency. Purely additive drop-in: one new subcommand, one new coded refusal, one new direct dependency; no change to backup/restore formats or any existing command, and nothing in this release reaches published versions as a fix.

Added

  • sluice backup export-as-parquet — transcode an existing backup chain into Parquet with restore-grade integrity (ADR-0164). The exporter rides the restore-side machinery wholesale: SHA-256-verified chunk reads, encrypted chains via --encrypt with GCM AAD binding and the plaintext-splice refusal, signed chains via --verify-key/--require-signature (the ADR-0154 policy), and the same per-chunk/per-table layer-2 row-count backstops including the Bug-183 tamper refusals — every integrity property restore enforces holds for the export. Exit-only by design: sluice never reads its Parquet output back, restore keeps the JSON-Lines path, and the Parquet files are written plaintext even from an encrypted chain (the analytics destination's encryption posture is the operator's separate choice). CLI follows the house shape: --from-dir/--from + --output-dir/--output, --include-table/--exclude-table, and re-running into the same destination refuses without --force-overwrite (the index manifest, written last, is both completion marker and overwrite sentinel).

  • Faithful-or-loud type mapping — exact where Parquet can hold it, coded refusal where it can't, never a silent narrow. DECIMAL stays exact over three physical tiers (INT32/INT64/FLBA16 by precision, refuse-don't-round parsing); unsigned integers keep the full uint64 range; temporals land as microsecond instants (timestamptz UTC-adjusted, as the IR already carries it); geometry is raw WKB with a GeoParquet geo footer block; enum/set value universes ride in footer metadata. Values Parquet cannot represent refuse with the new coded SLUICE-E-EXPORT-UNREPRESENTABLE naming the column and the remedy (--exclude-table, or query the JSON-Lines chunks directly): multi-dimensional array values (PG's type system doesn't declare dimensionality, so the derived LIST<element> schema is 1-D — the silent alternative is exactly Bug 74's flatten), MySQL TIME beyond the 24h day, NUMERIC NaN/Infinity, precision/scale overflow, and sub-microsecond temporals. Exactly two documented string downgrades — unbounded/oversized NUMERIC and TIMETZ — carry the exact value text and WARN with operator-visible notes. The whole encoder is pinned by the Bug-74 family×shape matrix (every IR family × value/NULL/empty/extreme/refusal shape, ground-truthed by parquet-go's own reader at raw-value level) plus a real-Postgres integration ground truth over a wide-type corpus.

  • Row groups align 1:1 with source chunks; provenance in every footer. The operator-visible chunk concept survives into the file: each Parquet file's footer records the full source-chunk list with SHA-256s and row counts (sluice:source_chunks — cross-reference backup verify), plus sluice:backup_id, sluice:source_engine, sluice:backup_created_at, and schema/table keys; parquet_index.json maps tables → files → rows/row-groups/source-chunks/type-notes.

  • Snapshot granularity, never a silent fold. The export represents ONE snapshot: the chain's latest full by default, or an earlier segment's full via --backup-id. Incremental change-windows after the selected full are NOT folded in — a loud WARN names the excluded count, and naming an incremental's id (or an unknown id) refuses listing the exportable fulls. Point-in-time row state composes honestly: restore the chain, re-export.

  • DuckDB cookbook recipe + the dependency posture (docs/cookbook/duckdb-on-sluice-backups.md). The recipe covers read_parquet locally and straight from S3, footer-metadata inspection, GeoParquet via the spatial extension, and the zero-export path (DuckDB reads the JSON-Lines chunks directly, with its caveats named). DuckDB is deliberately a recipe, never a dependency or subcommand — embedding it means CGO plus the arrow-go tail. The writer is parquet-go/parquet-go v0.30.1: pure Go, CGO-free (Windows builds and the goreleaser cross-compile matrix intact), ~6 small transitive modules, no arrow-go, no gRPC.

Fixed

  • Pre-land value-fidelity review — verdict SHIP (no silent loss); the coverage findings were folded before release and never reached any published version. The headline catch is the parquet-go zero-value-as-NULL wart, closed at design time and now tripwire-pinned: parquet-go's map-row deconstruction treats every Go ZERO value in an optional column as parquet NULL, so false, 0, ±0.0, and "" would all silently export as NULL — invisible to naive round-trip tests because a null reads back as the zero value. boxLeafValue pointer-wraps every scalar leaf at one chokepoint (a non-nil pointer is never "zero"), pinned by explicit non-null assertions on every zero-shaped value per family plus TestPin_UnboxedZeroCollapsesToNull, which proves the upstream wart is real in the pinned parquet-go version and fires loudly if an upgrade changes upstream's semantics. The review's other findings, all folded pre-release: a nested []string (2-D text array via blobcodec's list_str tag) now gets the multi-dimensional refusal with the --exclude-table remedy instead of a misleading "upstream codec bug" message; the array-element pin matrix gained the reviewer's gap set (FLBA16-tier decimal elements — the closest Bug-74 structural analog — int64-tier decimals, []byte empty-vs-NULL, JSON, time-of-day incl. the out-of-day refusal, full-range unsigned); and a JSON body of null (a PRESENT SQL value) is pinned to export as the "null" bytes, never SQL NULL — the tripwire against a future json.RawMessage refactor, which parquet-go special-cases to NULL.

Compatibility

  • Purely additive; no breaking changes. One new subcommand under backup, one new coded refusal (SLUICE-E-EXPORT-UNREPRESENTABLE), and one new direct dependency (parquet-go, pure Go — no CGO, no arrow-go, no gRPC); no flag, default, or behavior change to any existing command, and no change to the backup or restore formats — existing chains export as-is, and restore is untouched. The export is read-only against the backup store.

Who needs this — action required

Nothing to re-verify anywhere: this release contains no fix reaching any published version, and the review findings above were fixed before release. Pick this up if you want your existing backups feeding an analytics stack: sluice backup export-as-parquet --from-dir ./backups/prod --output-dir ./lake/prod (add --encrypt + the chain's key material for encrypted chains, --verify-key/--require-signature for signed ones), then point DuckDB, Snowflake COPY INTO, Spark, or pandas at the Parquet files — docs/cookbook/duckdb-on-sluice-backups.md is the walkthrough. If a column refuses as unrepresentable, the refusal names it and the remedy; the JSON-Lines chunks remain queryable directly for that table.

Install / upgrade: see the README. Verify downloads against checksums.txt.

Assets 15
Loading