v0.7.8
Proves Parquet type fidelity end-to-end with four independent readers
(DuckDB, ClickHouse, pyarrow, BigQuery), emits native Parquet logical
types for UUID/JSON, and closes two validation-surface bugs uncovered
while walking the documented golden paths.
Types — native Parquet logical types + round-trip proof
feat(types)— UUID columns now emit native Parquet
LogicalType::UuidasFixedSizeBinary(16)carrying the Arrow canonical
arrow.uuidextension type; JSON/JSONB carryarrow.json. Downstream
engines (DuckDB → nativeUUID, ClickHouse →Nullable(UUID),
BigQuery) load these without a cast. Enabled via the parquet
arrow_canonical_extension_typesfeature. Seesrc/types/mapping.rs.test(types)— four-reader validator matrix: every PG/MySQL type
round-trips through Parquet and is read back by DuckDB, ClickHouse,
pyarrow, and (live) BigQuery to pin field metadata + row-group stats
(tests/type_roundtrip/).fix(types/mysql)—UNSIGNED BIGINT(UINT64) overflowsINT64;
now mapped toDecimal128so the full range survives.
Bug fixes — validation surface
fix(pipeline/chunked)— the sequential checkpoint path ran
--validateon every chunk file but never recorded the result, so
mode: chunked+chunk_checkpoint: true+ defaultparallel: 1runs
storedvalidated = NULLinexport_metricsand dropped the
validated: passline from the run summary. It now sets the flag like
the other three export paths (regression test in
tests/live_chunked_recovery.rs).fix(preflight/doctor)—rivet doctordrops a.rivet_doctor_probe
writability test object at the destination and never removes it. A
subsequentrivet run --validateflagged it as anuntracked_object
and downgraded the run tovalidated: FAIL. The probe is now a
recognised Rivet sidecar (manifest::DOCTOR_PROBE_FILENAME) and skipped
by the manifest-aware--validatepass (regression test in
src/pipeline/validate_manifest.rs).
Preflight + UX
fix(preflight)— chunked / incremental exports on an indexed
cursor / chunk column no longer report a falseDEGRADEDverdict. A
catalogbtreeprobe replaces theEXPLAIN-on-base-query heuristic, so
an indexed PK reads asACCEPTABLEwith an(indexed)scan-type
suffix.polish(ux)—rivet initexplains its mode choice inline
(# auto: ~N rows ≥ 100K threshold and chunk column 'id' is available)
and scaleschunk_sizeto the row estimate; skipped incremental runs
printstatus: skipped (no new rows since cursor 'X'); the plaintext-
password and TLS warnings fire fromdoctor/check; the retry-safe
WARN is demoted to DEBUG for local destinations.
Docs + assets
docs— validated every command in the user-facing guides against
the binary; fixed drift (thefile_logstate table, realrivet doctor
output, the pilot walkthrough's missingdecimal(10,2)override).docs(gifs)— regenerated all instructional GIFs against current
behavior (card UI,(indexed)scan type,validated: pass).
Dependencies
- Bumped
mach20.4 → 0.6,tikv-jemallocator0.6 → 0.7,
criterion0.5 → 0.8 (dev),brotli8.0.2 → 8.0.3.