Skip to content

v0.1.0

Choose a tag to compare

@github-actions github-actions released this 02 Nov 20:24
· 57 commits to main since this release
v0.1.0
b24dc23

[0.1.0] - 2026-10-30

Fixed

  • tpch_q1 and the other few-group aggregates no longer update each aggregate row by row (#160). Perfect-hash keys are encoded one column at a time. COUNT and float SUM/AVG without nulls fold a whole batch in one pass into per-group locals, and the domain check reads string offsets instead of every cell. On the hash path, a single string key is dictionary-encoded per batch, so groups resolve once per distinct value. Join output builds its take index and unmatched-row mask once per batch rather than once per column. At SF1 with 10 workers, tpch_q1 is about 28 ms from 148 and tpcds_store_sales about 37 ms from 43. docs/bench/results.md is re-timed at SF1 and SF10. tpch_q3 at SF10 remains below the 3x target (#168).
  • tpch_q3 uses the extra workers again (#165). Its unique-key join fold with one float SUM ran on one thread; workers now fold contiguous runs of probe morsels into their own per-build-row sums and merge them in worker order, and the result is built with vectorised take instead of per-row scalars. The fused join-then-aggregate path also collects a scan-pipeline probe side on all workers, since it collects that side anyway. At SF1, tpch_q3 is about 2.4× faster at 4 workers than at one.
  • Runtime join-key filters and scan pruning no longer slow down single-worker queries (#164). The per-morsel key-range check read every key row through a generic per-row accessor, twice for a range and once per value of an IN-list; it now takes one vectorised min/max per morsel and key. A runtime range filter only skips morsels it rules out and no longer filters rows, so the probe side is not copied. At SF1 one worker, tpch_q3 is back to about 120 ms from 172, tpcds_store_sales to about 51 ms from 74, and tpch_q6 to about 33 ms from 86.
  • Join queries no longer slow down as workers increase (#159). A streamed hash-join probe spawned one thread per worker for every 4,096-row morsel and copied each thread's rows, so tpch_q3 and tpcds_store_sales got slower at 2, 4 and 8 workers. The probe now gathers about 16,384 rows per worker and spreads whole morsels across workers without copying, and build sides under 131,072 rows stay one table with their dense key index. A morsel much larger than one worker's share is sliced zero-copy so it still spreads across workers. scripts/check-worker-scaling.sh compares a 1-worker and an N-worker run and fails any query that is more than 25% slower with more workers; the bench workflow runs it at SF1 with 4 workers.
  • Grouped aggregation uses a perfect hash when integer, boolean, or single-byte string key ranges fit a dense index, so those groups do not allocate Scalar keys or probe a hash table. Wider domains radix-partition by hash bits, flush small thread-local tables, and merge partitions in parallel.
  • Unbounded inner joins scatter build morsels into hash partitions on worker threads, finalise each partition without an insert lock, and probe left morsels as they arrive so the probe side is not collected first.
  • Memory-limited and spilled inner joins hash-partition both sides and join each partition once, so a hash table is not rebuilt for every left×right chunk pair.
  • Unbounded inner join plus aggregate (run_then_agg) reuses hash_join_streamed when unique-key fold does not apply: one build table, then parallel probe morsels. After morsels stayed unconcatenated, the batch-pair fallback rebuilt a hash table per left×right pair and made tpcds_store_sales hundreds of times slower with workers > 1. scripts/bench-gb.sh and docs/bench/baseline.json time the suite with workers > 1; scripts/check-bench-regression.sh rejects a 1-worker run.
  • Query API bearer comparison hashes both tokens and compares the digests in constant time, so response timing does not leak a prefix of the configured token.

Changed

  • execute_sql always uses the morsel scheduler (ADR 0015), including the default of one worker. Pipelines split at join build, aggregate, and sort. try_parallel is gone. Fused inner-join-plus-aggregate stays. The SQL logic tests and the tiny decision-support suite run at 1 and 2 workers with identical results. EXPLAIN ANALYZE still times the pull operator tree.
  • The library is a layered workspace (ADR 0013): mizuchi-core, mizuchi-plan, mizuchi-io, mizuchi-postgres, mizuchi-exec, and the mizuchi facade. Embedder crate-root re-exports are unchanged. parse_sql no longer re-exports sqlparser::ast::Statement. Query API JSON Schema lives in crates/mizuchi-exec/schemas/query-api-v1.json. Flight stays behind --features flight in mizuchi-flight.

Added

  • Tagged v* releases publish a GitHub Release with crate packages, wheels, checksums, and build provenance, and push ghcr.io/sannrox/mizuchi:<version> (never latest). deploy/tenkai.toml is the Tenkai product entry. crates.io and PyPI stay unpublished while sekai-keys is a git pin (#146).

  • A standby coordinator takes over through a fenced lease (#161, ADR 0016). LeaseStore has an in-memory backend and a shared-directory backend (DirLeaseStore). Every takeover raises the fencing token, every Flight assignment carries it (x-mizuchi-fencing-token), and workers refuse a lower token and re-check it before publishing a result, registration or shuffle write; workers that mount the lease directory (--lease-dir) also check the directory's current token, so a restart cannot admit a superseded leader's work. A coordinator that lost the lease stops its query with Error::Fenced and does not lead again under a new token. RemoteCluster::with_leadership fences a cluster and leader_status reports leadership, lease age and token; mizuchi coordinator --lease-dir waits as a standby. The compose and Kubernetes layouts run a standby coordinator and mount the shared lease directory on every coordinator and worker. serve_worker and spawn_worker take the lease directory as a new argument.

  • Table files and query spills encrypt with caller-supplied keys (EncryptionSession + sekai-keys). Parquet modular encryption wraps per-file DEKs in footer metadata; Delta and Iceberg record key ids only. A revoked unwrap grant fails the read. Spill files use an ephemeral per-query DEK (MZS1) dropped with the pool. Encrypted writes refuse Hive partition paths. Session object-store Delta and Iceberg methods honor the same keys. Schema evolution keeps existing lake encryption identifiers. Unkeyed writes to an encrypted lake table fail closed, as do keyed appends onto an unencrypted table. Appends must use the table's tenant and KEK. Encrypted sessions refuse disk shuffle, streaming queries, and JSON/Avro/ORC writes. Keyed Parquet reads refuse plaintext files. Spill readers refuse unauthenticated frame lengths above 64 MiB before allocating. Large spill batches split into frames under that cap, encoded one frame at a time. Flight shuffle IPC is unchanged.

  • Query policy context and HS256 JWT/JWKS verification, off by default. SessionConfig.governed refuses execute without a PolicyContext. The context names tenant, principal, and actor, lists allowed tables, injects row filters, and strips masked columns from the planning catalog so SELECT *, joins, and aggregates cannot name them. Tables bound with Session::bind_table_tenant reject a different tenant. Query reports record principal and actor. The query API accepts a Bearer JWT next to the static token (QueryApiConfig.jwt); JWKS is embedder-supplied oct keys, with no network fetch. PolicyContext can attach opaque table and column marking labels; the planner unions them through projections, expressions, joins, aggregates, window functions, filters, CTEs and subqueries. Results and write reports list output columns with those labels. In governed mode a write whose destination labels do not cover the output is refused. Mizuchi does not interpret labels.

  • Hash tables pack Utf8 grouping and join keys of at most 16 bytes into a fixed-width cell, including null and empty keys, and unpack only when emitting. Dictionary-encoded Utf8 stays dictionary through filter; a comparison against a Utf8 literal does not decode the stored column. gb_bench profile on live SF1/SF10 is unchanged here (no bench-data/ in this change).

  • A per-query memory budget covers join, aggregate, and sort. Session::with_system_memory_budget caps at one quarter of machine memory and spills under the process temp directory. Hash join already spills per partition; aggregate merge reloads one spill file at a time; sort spills runs. QueryResult reports peak_bytes and spilled_bytes. EXPLAIN ANALYZE prints query peak_bytes= and spilled_bytes=. Session::new stays unbounded so embedders keep policy.

  • After an inner hash-join build, integer equality keys become a min/max or small IN-list filter on the probe-side scan. EXPLAIN ANALYZE prints runtime_filter=[...] and the scan's rows= count is the rows that survive the filter. In-memory scans skip morsels that cannot match. Parquet footer prune treats IN-list as OR. The filter is optional: Session::with_runtime_join_filters(false) is still correct.

  • Query history and a versioned HTTP metrics API (mizuchi.query-api/v1, ADR 0012). Off until SessionConfig.query_api is set. /healthz lists the contract id; /v1/queries lists, filters, searches, and paginates completed queries; /v1/queries/{id} returns plan, read/write report, and per-operator metrics; /v1/queries/running and POST .../cancel cover in-flight work. Schema: crates/mizuchi-exec/schemas/query-api-v1.json. History persists to a caller-provided JSONL file and survives restart. Loopback binds do not require allow_network; non-loopback binds do. Optional bearer token is supplied by the embedder. QueryApiConfig.ui_dir serves an unpacked mizuchi-console zip at /ui/ (ADR 0014); default is off.

  • Flight coordinator and worker processes (--features flight). TCP binds only with allow_network / MIZUCHI_ALLOW_NETWORK. Workers heartbeat; a failed worker is marked dead and the task retries on a live peer. Shuffle is a Flight DoGet stream with a shared spill-file fallback. deploy/compose.yaml runs three workers. The optional .github/workflows/flight.yml job kills a worker mid-query and still expects a correct result.

  • Join and sort keep a single incoming batch as-is and concatenate only when two or more morsels must share one buffer. An unbounded join no longer concatenates the probe side. Contiguous gather/take is a zero-copy slice. EXPLAIN ANALYZE and Session::query_metrics report copied_bytes (bytes written by concat/take; slices add 0). Before: both join sides were concatenated on the unbounded path. After: only the smaller build side is concatenated, and only when it arrives as more than one morsel.

  • Decision-support SF1 target: grouping and join maps use a cheap hasher, integer equi-keys in a modest span use a compact range map, unique i64 keys probe with a direct index, and an aggregate over a unique-key inner join folds into the build slots. Unused columns are dropped through joins and aggregates before a scan reads them. The scheduled GB bench job publishes ratios and fails when any query is more than 20% slower than docs/bench/baseline.json. Every SF1 DuckDB ratio in docs/bench/results.md is at least 0.33.

  • Parquet scan pushdown: projection to column chunks, row-group skip from footer min/max (Int64 and Float64) and bloom filters, page-index skip, and late materialization of remaining columns. read_parquet_reported returns kept groups and compressed bytes of projected columns. Clustered q6-style skip ratios in docs/bench/results.md.

  • Parallel morsel execution: combinable aggregates (AVG keeps sum and count until the merge), inner hash-join probe, and wrapping sort/limit run on Session::with_workers threads. SF1 scaling at 1/2/4/8 workers in docs/bench/results.md.

  • Hash join and hash aggregate: equality keys hash in place (a single integer column uses the value as the map key). Inner joins build the smaller collected side; the optimizer places the smaller estimated input on the build side from table stats. Micro-benchmarks: cargo run --release -p mizuchi --example kernel_micro. SF1 q1/q3/store_sales times in docs/bench/results.md.

  • Columnar aggregate updates: ungrouped COUNT/SUM/AVG/MIN/MAX fold whole Arrow arrays (float SUM/AVG through compute kernels; integer SUM stays checked so overflow fails closed). Grouped lookup hashes primitive, Utf8, and dictionary-string cells in place and allocates Scalar keys only for new groups. Micro-benchmarks: cargo run --release -p mizuchi --example kernel_micro. SF1 q1/q6 times in docs/bench/results.md.

  • Workspace skeleton with a placeholder library crate.

  • ADR 0001 (record architecture decisions) and ADR 0002 (own query engine as a side project).

  • Arrow record-batch API: schema round-trip for supported types, owned and borrowed batches that wrap standard Arrow arrays without a private memory format, and a configurable max-rows policy (default 4096).

  • Scalar expressions evaluated column-at-a-time with Arrow compute kernels (arithmetic, comparison, boolean, cast, CASE, null handling, string basics).

  • SQL parser: sqlparser-rs with the PostgreSQL dialect; syntax errors and unimplemented SQL fail closed (ADR 0003).

  • Logical plan bound from SQL: scan, filter, project, aggregate, join, sort, limit, union; EXPLAIN text; fail-closed analysis errors.

  • Pull-based vectorized scan, filter, and projection over in-memory tables; end-to-end SELECT ... WHERE (ADR 0004).

  • Hash aggregate: grouped and ungrouped COUNT, SUM, MIN, MAX, AVG, and COUNT DISTINCT over Arrow batches, including empty input and null grouping keys.

  • Hash join: inner, left, right, full, semi, and anti over Arrow batches. Null keys never match; CROSS JOIN is a nested loop.

  • Sort, top-k, and limit/offset over Arrow batches. ORDER BY honors ASC/DESC and NULLS FIRST/LAST; LIMIT after ORDER BY sorts only the kept prefix.

  • Parquet and CSV readers that yield Arrow batches. Parquet projection decodes only requested columns; row-group skip is a footer-stats hook. CSV can infer types or keep Utf8.

  • Rule-based optimizer: constant folding, boolean simplification, filter merge, limit pushdown through projection, and redundant projection removal.

  • Predicate and projection pushdown into scans and inner joins. Parquet readers skip row groups using footer min/max.

  • Table and column statistics (row count, NDV, Int64 min/max) and cost-based inner-join reordering. Dynamic programming for up to 12 tables; greedy above that.

  • Stage planning: hash-partition exchanges at equi-joins and grouped aggregates, shuffle-map and result stages, and a single-process worker pool.

  • Shuffle blocks written as Arrow IPC files in a caller-provided directory; job subdirectories are removed when the query finishes.

  • Per-query memory pool with reservations. Sort, join, and aggregate spill Arrow IPC runs to a caller-provided directory and fail closed when the limit cannot be honoured.

  • Morsel work queue across a bounded thread pool for scan/filter/project and combinable aggregates, plus cooperative cancellation checked between morsels.

  • SQLLogicTest runner and corpus under tests/slt. CI runs the adopted subset. Window, CTE, subquery, set operator, DECIMAL, and date/time coverage is recorded in docs/sql-coverage.md; gaps are skipif mizuchi records plus follow-up issues.

  • TPC-H and TPC-DS style decision-support harness: a tiny in-memory generator, a fixed query set, and JSON with per-query time and memory. Larger scales stay out of the default cargo test run.

  • Decision-support comparison against DuckDB and DataFusion: same SQL and tables, JSON with result diffs and timing ratios. CI uses committed goldens so the default test run stays offline.

  • Python DataFrame API via PyO3: lazy select, filter, group_by, join, sort, and SQL, with a pip-installable wheel built in CI.

  • LibFuzzer targets for SQL parse, plan, execute, and a supported-SQL grammar. CI runs them for a bounded time; crashes land as regression tests.

  • Engine owner shares: reserved memory and worker slices so concurrent owners cannot starve each other.

  • Stage tasks retry after a worker failure and reuse committed shuffle files. Partial writes from a failed attempt are discarded.

  • Adaptive skew handling: after shuffle-map, oversized join partitions are split by grouping map outputs and the matching other-side partition is replicated. EXPLAIN ANALYZE via Session::explain_analyze reports the splits.

  • Python scalar and aggregate UDFs over Arrow arrays, plus zero-copy interchange through the Arrow C Data Interface (register_arrow / to_arrow).

  • ADR 0005: incremental execution as micro-batches on the same hash aggregate. MicroBatchAgg folds append-only batches and emits the complete group table.

  • Query result cache on execute_sql: identical plans reuse stored Arrow batches while scanned table generations match. Re-registering a table invalidates entries that used it.

  • ADR 0006: GPU operators are a no-go for this phase. The default decision-support suite is too small for device copy to pay off; revisit when inputs are gigabyte-scale and a runner grants a device.

  • Gigabyte-scale decision-support baseline: SF1 and SF10 TPC-H/TPC-DS-style queries versus DuckDB and DataFusion on the same Snappy Parquet. Reproduce with scripts/bench-gb.sh; committed times and method in docs/bench/. Optional workflow_dispatch smoke at --sf tiny; default cargo test is unchanged.

  • Physical operators implement a pull Operator trait. Session::with_lowering selects an implementation per operator; CPU scan/filter/project/limit stay the default. Dispatch is once per batch.

  • Partitioned Parquet writer: Hive-style directories, append or overwrite, staging then rename so a failed or cancelled job leaves committed files unchanged. Part files round-trip through read_parquet.

  • Object-store datasets: the runner supplies an ObjectStore. Hive-style partition paths are pruned before fetch; Parquet footer stats prune row groups after. Default tests use in-memory storage. An ignored s3 feature talks to a local emulator.

  • EXPLAIN prints logical and physical plans. EXPLAIN ANALYZE and Session::query_metrics add per-operator rows, batches, elapsed time, exclusive time, and peak pool bytes as text and JSON. Ordinary execute_sql does not wrap operators.

  • Decision-support limiter profiles: exclusive operator time and pool high-water for every suite query, ranked under docs/bench/limiters.md. Reproduce Parquet profiles with scripts/profile-suite.sh; default tests drive query_metrics on the tiny in-memory tables.

  • ADR 0007: crate-root re-exports are the public API. 0.y may break; after 1.0, deprecations last at least one minor. Tag v* dry-runs cargo publish and builds Linux/macOS/Windows wheels. The crate package includes the repository README; 0.1 stays experimental until a maintainer tags v0.1.0 after that dry-run is green.

  • Embedder API: SessionConfig sets memory, workers, spill, and allowed read/write paths. Session plans and executes SQL or a logical plan, registers Arrow batches, files, and C streams, and writes Parquet through those limits. Example: crates/mizuchi/examples/embed.rs.

  • Query reports: Session::execute_sql_reported and write_parquet_reported return inputs (dataset, path, pruned Hive partitions, columns, mtime) and outputs (path, partitions, mode, rows, bytes, commit id) plus engine version and plan text. Store reads expose the same scan record.

  • Sandbox compatibility: path denials are Error::Io, spill stays in the configured directory, and a memory cap without spill is Error::Memory. The crate has no sandbox backend. Optional workflow_dispatch job runs Linux filesystem-limit tests; default pull-request CI is unchanged.

  • Cluster coordinator: Cluster / ClusterConfig manage in-process worker slots with CPU, memory, and spill grants, dynamic scale-up/down, and speculative copies of straggler stage tasks (first commit wins). Standalone and Kubernetes layouts live under deploy/. Optional workflow_dispatch job runs the tiny decision-support suite on two workers.

  • Delta Lake and Apache Iceberg tables: snapshot writes of Parquet data files, time travel by version or timestamp, schema evolution (add, rename, widen), and MERGE with atomic create_new commits. A concurrent writer that races the same next log or metadata file gets Error::Io instead of replacing a commit. Session::merge_sql applies MERGE INTO to registered tables; merge_delta / merge_iceberg commit a new snapshot.

  • JSON (newline-delimited), Avro Object Container Files, and uncompressed ORC: readers with schema inference and projection, plus writers. Nested struct and list types round-trip. PostgreSQL table and query scans push projection, filters, and integer-range partitions into SELECT; TCP requires SessionConfig.allow_network (ADR 0008). Optional postgres feature (TLS when PostgresSource.ssl is set) and workflow_dispatch job; default cargo test stays offline.

  • Continuous queries: StreamQuery runs micro-batches over an unbounded source with event-time watermarks, tumbling/sliding/session windows, complete or append output, and a checkpointed Parquet sink (ADR 0009). A trigger that writes the sink but not the commit marker is replayed with the same batch id. Offset, commit, state, and sink files appear only after a durable write, so a killed process can resume. A default-CI integration test kills a child at random points and checks the sink against a batch reference, including late rows.

  • Pluggable catalog: three-part names (catalog.namespace.table), default mizuchi.public, memory and file backends, and a CatalogProvider trait for an embedder-supplied store (ADR 0010). Partition metadata on TableStats drives Hive-path pruning. Session::execute_ddl runs CREATE / DROP / ALTER TABLE; SHOW TABLES is a query.

  • Scalar function library (string, math, nullif, ARRAY / array_length, EXTRACT on Date/Timestamp columns) in SQL and DataFrame fragments. Combinable user-defined aggregates (CombineAgg, Session::register_udaf, Python register_udaf) merge under parallel workers and spill. Vectorized Python UDFs remain Arrow-array batched. Coverage: docs/functions.md; compare vectorized vs row with scripts/bench_udf.py.

  • Window functions: ROW_NUMBER, RANK, DENSE_RANK, and COUNT/SUM with OVER (PARTITION BY, ORDER BY, ROWS/RANGE frames, named WINDOW). Evidence: window.slt.

  • Common table expressions: non-recursive WITH (later names see earlier) and WITH RECURSIVE that unions an anchor with a step over a working table until empty. Evidence: cte.slt.

  • Subqueries: IN of literals and IN (SELECT ...), NOT IN (SELECT ...) (null-aware anti: a null in the subquery yields unknown), EXISTS / NOT EXISTS (including correlation as a semi or anti join), uncorrelated scalar subqueries, and LATERAL derived tables that project outer columns. Evidence: subquery.slt.

  • Set operators: UNION / UNION DISTINCT / UNION ALL, INTERSECT / INTERSECT ALL, EXCEPT / EXCEPT ALL. Left column names; matching column counts; integer/float/null widen; nulls compare equal; first-seen left-then-right order. Evidence: setops.slt.

  • DECIMAL / NUMERIC SQL types map onto Arrow Decimal32 / Decimal64 / Decimal128. CAST and arithmetic use Arrow kernels and fail closed when precision exceeds 38 or a value does not fit the chosen width. Evidence: decimal.slt.

  • SQL DATE, TIMESTAMP, and INTERVAL 'n' DAY map onto Arrow Date32, Timestamp(Microsecond), and Interval(DayTime). CAST, EXTRACT, DATE_TRUNC('day', …), CURRENT_DATE, and NOW() evaluate; the session clock pins CURRENT_DATE / NOW(). Evidence: datetime.slt.

  • Delta Lake and Apache Iceberg on a caller-supplied object store: the same snapshot, time travel, schema evolution, and MERGE path as local tables, with exclusive PutMode::Create commits. Iceberg REST catalog implements CatalogProvider; TCP requires SessionConfig.allow_network. Optional workflow_dispatch job talks to an S3 emulator; default cargo test uses in-memory storage.

  • CI runs object-store S3 emulator tests against MinIO on pull requests and main. The job fails if those ignored tests are skipped. Default cargo test stays offline.