Skip to content

Timelake 0.3.0

Choose a tag to compare

@olekslitus olekslitus released this 14 Oct 16:49
· 327 commits to main since this release

Added

  • Table format version: tables record timelake.format-version (2 for tables created now; tables without it are
    format 1). Timelake reads tables of any older format and refuses to write to a table of a newer format than it
    knows, with a message to upgrade.
  • timelake upgrade (also Table.upgrade() in Python and Rust) brings tables to the current format: records the
    format version, builds a missing last-value index, rewrites unsorted or small files and writes the head. Each step is
    a commit, and no snapshot expires for 7 days (--keep-snapshots), so timelake rollback <table> <snapshot>
    (Table.rollback()) undoes it. --dry-run shows the plan. New table property timelake.keep-snapshots-until.

Fixed

  • After a rollback, a batch id from the abandoned snapshots no longer counts as already committed.

Changed

  • Faster first queries: after each commit the writer also writes metadata/timelake-head, which holds the table
    metadata, the live files, and (while they fit in 1 MiB) the last-value index and the Parquet footers of the newest
    files. A cold query reads it together with a listing of the table's metadata files that confirms it is current, so a
    first latest query takes one round trip to storage instead of three, and a first filtered scan two instead of six.
    A process that already has a table only checks for a newer version. Measured from a laptop, first queries take
    245–545 ms instead of 510–1,030 on S3, and 210–520 instead of 760–1,190 on Tigris. The TypeScript client reads the
    head too and checks for a newer version while its query runs (TableInfo.manifests is null for a table read from
    its head). Other engines see no change. New table properties timelake.head-format (arrow, json, a codec
    registered with Lake::register_head_codec, or none) and timelake.head-inline-bytes; new storage options
    timelake_version_lookup (list or hint: no LIST, one more round trip), timelake_freshness_check (head or list,
    list by default on Tigris) and timelake_warm_connections (off by default). The requests each operation makes,
    and what they cost, are in the storage docs. If you write a table with an older Timelake after this one, delete its
    metadata/timelake-head.