Repository navigation
v0.1.0
[0.1.0] - 2026-10-30
Fixed
tpch_q1and the other few-group aggregates no longer update each aggregate row by row (#160). Perfect-hash keys are encoded one column at a time. COUNT and float SUM/AVG without nulls fold a whole batch in one pass into per-group locals, and the domain check reads string offsets instead of every cell. On the hash path, a single string key is dictionary-encoded per batch, so groups resolve once per distinct value. Join output builds its take index and unmatched-row mask once per batch rather than once per column. At SF1 with 10 workers,tpch_q1is about 28 ms from 148 andtpcds_store_salesabout 37 ms from 43.docs/bench/results.mdis re-timed at SF1 and SF10.tpch_q3at SF10 remains below the 3x target (#168).tpch_q3uses the extra workers again (#165). Its unique-key join fold with one floatSUMran on one thread; workers now fold contiguous runs of probe morsels into their own per-build-row sums and merge them in worker order, and the result is built with vectorisedtakeinstead of per-row scalars. The fused join-then-aggregate path also collects a scan-pipeline probe side on all workers, since it collects that side anyway. At SF1,tpch_q3is about 2.4× faster at 4 workers than at one.- Runtime join-key filters and scan pruning no longer slow down single-worker queries (#164). The per-morsel key-range check read every key row through a generic per-row accessor, twice for a range and once per value of an IN-list; it now takes one vectorised min/max per morsel and key. A runtime range filter only skips morsels it rules out and no longer filters rows, so the probe side is not copied. At SF1 one worker,
tpch_q3is back to about 120 ms from 172,tpcds_store_salesto about 51 ms from 74, andtpch_q6to about 33 ms from 86. - Join queries no longer slow down as workers increase (#159). A streamed hash-join probe spawned one thread per worker for every 4,096-row morsel and copied each thread's rows, so
tpch_q3andtpcds_store_salesgot slower at 2, 4 and 8 workers. The probe now gathers about 16,384 rows per worker and spreads whole morsels across workers without copying, and build sides under 131,072 rows stay one table with their dense key index. A morsel much larger than one worker's share is sliced zero-copy so it still spreads across workers.scripts/check-worker-scaling.shcompares a 1-worker and an N-worker run and fails any query that is more than 25% slower with more workers; the bench workflow runs it at SF1 with 4 workers. - Grouped aggregation uses a perfect hash when integer, boolean, or single-byte string key ranges fit a dense index, so those groups do not allocate
Scalarkeys or probe a hash table. Wider domains radix-partition by hash bits, flush small thread-local tables, and merge partitions in parallel. - Unbounded inner joins scatter build morsels into hash partitions on worker threads, finalise each partition without an insert lock, and probe left morsels as they arrive so the probe side is not collected first.
- Memory-limited and spilled inner joins hash-partition both sides and join each partition once, so a hash table is not rebuilt for every left×right chunk pair.
- Unbounded inner join plus aggregate (
run_then_agg) reuseshash_join_streamedwhen unique-key fold does not apply: one build table, then parallel probe morsels. After morsels stayed unconcatenated, the batch-pair fallback rebuilt a hash table per left×right pair and madetpcds_store_saleshundreds of times slower with workers > 1.scripts/bench-gb.shanddocs/bench/baseline.jsontime the suite with workers > 1;scripts/check-bench-regression.shrejects a 1-worker run. - Query API bearer comparison hashes both tokens and compares the digests in constant time, so response timing does not leak a prefix of the configured token.
Changed
execute_sqlalways uses the morsel scheduler (ADR 0015), including the default of one worker. Pipelines split at join build, aggregate, and sort.try_parallelis gone. Fused inner-join-plus-aggregate stays. The SQL logic tests and the tiny decision-support suite run at 1 and 2 workers with identical results.EXPLAIN ANALYZEstill times the pull operator tree.- The library is a layered workspace (ADR 0013):
mizuchi-core,mizuchi-plan,mizuchi-io,mizuchi-postgres,mizuchi-exec, and themizuchifacade. Embedder crate-root re-exports are unchanged.parse_sqlno longer re-exportssqlparser::ast::Statement. Query API JSON Schema lives incrates/mizuchi-exec/schemas/query-api-v1.json. Flight stays behind--features flightinmizuchi-flight.
Added
-
Tagged
v*releases publish a GitHub Release with crate packages, wheels, checksums, and build provenance, and pushghcr.io/sannrox/mizuchi:<version>(neverlatest).deploy/tenkai.tomlis the Tenkai product entry. crates.io and PyPI stay unpublished whilesekai-keysis a git pin (#146). -
A standby coordinator takes over through a fenced lease (#161, ADR 0016).
LeaseStorehas an in-memory backend and a shared-directory backend (DirLeaseStore). Every takeover raises the fencing token, every Flight assignment carries it (x-mizuchi-fencing-token), and workers refuse a lower token and re-check it before publishing a result, registration or shuffle write; workers that mount the lease directory (--lease-dir) also check the directory's current token, so a restart cannot admit a superseded leader's work. A coordinator that lost the lease stops its query withError::Fencedand does not lead again under a new token.RemoteCluster::with_leadershipfences a cluster andleader_statusreports leadership, lease age and token;mizuchi coordinator --lease-dirwaits as a standby. The compose and Kubernetes layouts run a standby coordinator and mount the shared lease directory on every coordinator and worker.serve_workerandspawn_workertake the lease directory as a new argument. -
Table files and query spills encrypt with caller-supplied keys (
EncryptionSession+sekai-keys). Parquet modular encryption wraps per-file DEKs in footer metadata; Delta and Iceberg record key ids only. A revoked unwrap grant fails the read. Spill files use an ephemeral per-query DEK (MZS1) dropped with the pool. Encrypted writes refuse Hive partition paths. Session object-store Delta and Iceberg methods honor the same keys. Schema evolution keeps existing lake encryption identifiers. Unkeyed writes to an encrypted lake table fail closed, as do keyed appends onto an unencrypted table. Appends must use the table's tenant and KEK. Encrypted sessions refuse disk shuffle, streaming queries, and JSON/Avro/ORC writes. Keyed Parquet reads refuse plaintext files. Spill readers refuse unauthenticated frame lengths above 64 MiB before allocating. Large spill batches split into frames under that cap, encoded one frame at a time. Flight shuffle IPC is unchanged. -
Query policy context and HS256 JWT/JWKS verification, off by default.
SessionConfig.governedrefuses execute without aPolicyContext. The context names tenant, principal, and actor, lists allowed tables, injects row filters, and strips masked columns from the planning catalog soSELECT *, joins, and aggregates cannot name them. Tables bound withSession::bind_table_tenantreject a different tenant. Query reports record principal and actor. The query API accepts a Bearer JWT next to the static token (QueryApiConfig.jwt); JWKS is embedder-suppliedoctkeys, with no network fetch.PolicyContextcan attach opaque table and column marking labels; the planner unions them through projections, expressions, joins, aggregates, window functions, filters, CTEs and subqueries. Results and write reports list output columns with those labels. In governed mode a write whose destination labels do not cover the output is refused. Mizuchi does not interpret labels. -
Hash tables pack Utf8 grouping and join keys of at most 16 bytes into a fixed-width cell, including null and empty keys, and unpack only when emitting. Dictionary-encoded Utf8 stays dictionary through filter; a comparison against a Utf8 literal does not decode the stored column.
gb_bench profileon live SF1/SF10 is unchanged here (nobench-data/in this change). -
A per-query memory budget covers join, aggregate, and sort.
Session::with_system_memory_budgetcaps at one quarter of machine memory and spills under the process temp directory. Hash join already spills per partition; aggregate merge reloads one spill file at a time; sort spills runs.QueryResultreportspeak_bytesandspilled_bytes.EXPLAIN ANALYZEprintsquery peak_bytes=andspilled_bytes=.Session::newstays unbounded so embedders keep policy. -
After an inner hash-join build, integer equality keys become a min/max or small IN-list filter on the probe-side scan.
EXPLAIN ANALYZEprintsruntime_filter=[...]and the scan'srows=count is the rows that survive the filter. In-memory scans skip morsels that cannot match. Parquet footer prune treats IN-list as OR. The filter is optional:Session::with_runtime_join_filters(false)is still correct. -
Query history and a versioned HTTP metrics API (
mizuchi.query-api/v1, ADR 0012). Off untilSessionConfig.query_apiis set./healthzlists the contract id;/v1/querieslists, filters, searches, and paginates completed queries;/v1/queries/{id}returns plan, read/write report, and per-operator metrics;/v1/queries/runningandPOST .../cancelcover in-flight work. Schema:crates/mizuchi-exec/schemas/query-api-v1.json. History persists to a caller-provided JSONL file and survives restart. Loopback binds do not requireallow_network; non-loopback binds do. Optional bearer token is supplied by the embedder.QueryApiConfig.ui_dirserves an unpacked mizuchi-console zip at/ui/(ADR 0014); default is off. -
Flight coordinator and worker processes (
--features flight). TCP binds only withallow_network/MIZUCHI_ALLOW_NETWORK. Workers heartbeat; a failed worker is marked dead and the task retries on a live peer. Shuffle is a FlightDoGetstream with a shared spill-file fallback.deploy/compose.yamlruns three workers. The optional.github/workflows/flight.ymljob kills a worker mid-query and still expects a correct result. -
Join and sort keep a single incoming batch as-is and concatenate only when two or more morsels must share one buffer. An unbounded join no longer concatenates the probe side. Contiguous gather/take is a zero-copy slice.
EXPLAIN ANALYZEandSession::query_metricsreportcopied_bytes(bytes written by concat/take; slices add 0). Before: both join sides were concatenated on the unbounded path. After: only the smaller build side is concatenated, and only when it arrives as more than one morsel. -
Decision-support SF1 target: grouping and join maps use a cheap hasher, integer equi-keys in a modest span use a compact range map, unique i64 keys probe with a direct index, and an aggregate over a unique-key inner join folds into the build slots. Unused columns are dropped through joins and aggregates before a scan reads them. The scheduled GB bench job publishes ratios and fails when any query is more than 20% slower than
docs/bench/baseline.json. Every SF1 DuckDB ratio indocs/bench/results.mdis at least 0.33. -
Parquet scan pushdown: projection to column chunks, row-group skip from footer min/max (Int64 and Float64) and bloom filters, page-index skip, and late materialization of remaining columns.
read_parquet_reportedreturns kept groups and compressed bytes of projected columns. Clustered q6-style skip ratios indocs/bench/results.md. -
Parallel morsel execution: combinable aggregates (AVG keeps sum and count until the merge), inner hash-join probe, and wrapping sort/limit run on
Session::with_workersthreads. SF1 scaling at 1/2/4/8 workers indocs/bench/results.md. -
Hash join and hash aggregate: equality keys hash in place (a single integer column uses the value as the map key). Inner joins build the smaller collected side; the optimizer places the smaller estimated input on the build side from table stats. Micro-benchmarks:
cargo run --release -p mizuchi --example kernel_micro. SF1 q1/q3/store_sales times indocs/bench/results.md. -
Columnar aggregate updates: ungrouped COUNT/SUM/AVG/MIN/MAX fold whole Arrow arrays (float SUM/AVG through compute kernels; integer SUM stays checked so overflow fails closed). Grouped lookup hashes primitive, Utf8, and dictionary-string cells in place and allocates
Scalarkeys only for new groups. Micro-benchmarks:cargo run --release -p mizuchi --example kernel_micro. SF1 q1/q6 times indocs/bench/results.md. -
Workspace skeleton with a placeholder library crate.
-
ADR 0001 (record architecture decisions) and ADR 0002 (own query engine as a side project).
-
Arrow record-batch API: schema round-trip for supported types, owned and borrowed batches that wrap standard Arrow arrays without a private memory format, and a configurable max-rows policy (default 4096).
-
Scalar expressions evaluated column-at-a-time with Arrow compute kernels (arithmetic, comparison, boolean, cast, CASE, null handling, string basics).
-
SQL parser: sqlparser-rs with the PostgreSQL dialect; syntax errors and unimplemented SQL fail closed (ADR 0003).
-
Logical plan bound from SQL: scan, filter, project, aggregate, join, sort, limit, union; EXPLAIN text; fail-closed analysis errors.
-
Pull-based vectorized scan, filter, and projection over in-memory tables; end-to-end SELECT ... WHERE (ADR 0004).
-
Hash aggregate: grouped and ungrouped COUNT, SUM, MIN, MAX, AVG, and COUNT DISTINCT over Arrow batches, including empty input and null grouping keys.
-
Hash join: inner, left, right, full, semi, and anti over Arrow batches. Null keys never match; CROSS JOIN is a nested loop.
-
Sort, top-k, and limit/offset over Arrow batches. ORDER BY honors ASC/DESC and NULLS FIRST/LAST; LIMIT after ORDER BY sorts only the kept prefix.
-
Parquet and CSV readers that yield Arrow batches. Parquet projection decodes only requested columns; row-group skip is a footer-stats hook. CSV can infer types or keep Utf8.
-
Rule-based optimizer: constant folding, boolean simplification, filter merge, limit pushdown through projection, and redundant projection removal.
-
Predicate and projection pushdown into scans and inner joins. Parquet readers skip row groups using footer min/max.
-
Table and column statistics (row count, NDV, Int64 min/max) and cost-based inner-join reordering. Dynamic programming for up to 12 tables; greedy above that.
-
Stage planning: hash-partition exchanges at equi-joins and grouped aggregates, shuffle-map and result stages, and a single-process worker pool.
-
Shuffle blocks written as Arrow IPC files in a caller-provided directory; job subdirectories are removed when the query finishes.
-
Per-query memory pool with reservations. Sort, join, and aggregate spill Arrow IPC runs to a caller-provided directory and fail closed when the limit cannot be honoured.
-
Morsel work queue across a bounded thread pool for scan/filter/project and combinable aggregates, plus cooperative cancellation checked between morsels.
-
SQLLogicTest runner and corpus under
tests/slt. CI runs the adopted subset. Window, CTE, subquery, set operator, DECIMAL, and date/time coverage is recorded indocs/sql-coverage.md; gaps areskipif mizuchirecords plus follow-up issues. -
TPC-H and TPC-DS style decision-support harness: a tiny in-memory generator, a fixed query set, and JSON with per-query time and memory. Larger scales stay out of the default
cargo testrun. -
Decision-support comparison against DuckDB and DataFusion: same SQL and tables, JSON with result diffs and timing ratios. CI uses committed goldens so the default test run stays offline.
-
Python DataFrame API via PyO3: lazy select, filter, group_by, join, sort, and SQL, with a pip-installable wheel built in CI.
-
LibFuzzer targets for SQL parse, plan, execute, and a supported-SQL grammar. CI runs them for a bounded time; crashes land as regression tests.
-
Engine owner shares: reserved memory and worker slices so concurrent owners cannot starve each other.
-
Stage tasks retry after a worker failure and reuse committed shuffle files. Partial writes from a failed attempt are discarded.
-
Adaptive skew handling: after shuffle-map, oversized join partitions are split by grouping map outputs and the matching other-side partition is replicated.
EXPLAIN ANALYZEviaSession::explain_analyzereports the splits. -
Python scalar and aggregate UDFs over Arrow arrays, plus zero-copy interchange through the Arrow C Data Interface (
register_arrow/to_arrow). -
ADR 0005: incremental execution as micro-batches on the same hash aggregate.
MicroBatchAggfolds append-only batches and emits the complete group table. -
Query result cache on
execute_sql: identical plans reuse stored Arrow batches while scanned table generations match. Re-registering a table invalidates entries that used it. -
ADR 0006: GPU operators are a no-go for this phase. The default decision-support suite is too small for device copy to pay off; revisit when inputs are gigabyte-scale and a runner grants a device.
-
Gigabyte-scale decision-support baseline: SF1 and SF10 TPC-H/TPC-DS-style queries versus DuckDB and DataFusion on the same Snappy Parquet. Reproduce with
scripts/bench-gb.sh; committed times and method indocs/bench/. Optionalworkflow_dispatchsmoke at--sf tiny; defaultcargo testis unchanged. -
Physical operators implement a pull
Operatortrait.Session::with_loweringselects an implementation per operator; CPU scan/filter/project/limit stay the default. Dispatch is once per batch. -
Partitioned Parquet writer: Hive-style directories, append or overwrite, staging then rename so a failed or cancelled job leaves committed files unchanged. Part files round-trip through
read_parquet. -
Object-store datasets: the runner supplies an
ObjectStore. Hive-style partition paths are pruned before fetch; Parquet footer stats prune row groups after. Default tests use in-memory storage. An ignoreds3feature talks to a local emulator. -
EXPLAINprints logical and physical plans.EXPLAIN ANALYZEandSession::query_metricsadd per-operator rows, batches, elapsed time, exclusive time, and peak pool bytes as text and JSON. Ordinaryexecute_sqldoes not wrap operators. -
Decision-support limiter profiles: exclusive operator time and pool high-water for every suite query, ranked under
docs/bench/limiters.md. Reproduce Parquet profiles withscripts/profile-suite.sh; default tests drivequery_metricson the tiny in-memory tables. -
ADR 0007: crate-root re-exports are the public API. 0.y may break; after 1.0, deprecations last at least one minor. Tag
v*dry-runscargo publishand builds Linux/macOS/Windows wheels. The crate package includes the repository README; 0.1 stays experimental until a maintainer tagsv0.1.0after that dry-run is green. -
Embedder API:
SessionConfigsets memory, workers, spill, and allowed read/write paths.Sessionplans and executes SQL or a logical plan, registers Arrow batches, files, and C streams, and writes Parquet through those limits. Example:crates/mizuchi/examples/embed.rs. -
Query reports:
Session::execute_sql_reportedandwrite_parquet_reportedreturn inputs (dataset, path, pruned Hive partitions, columns, mtime) and outputs (path, partitions, mode, rows, bytes, commit id) plus engine version and plan text. Store reads expose the same scan record. -
Sandbox compatibility: path denials are
Error::Io, spill stays in the configured directory, and a memory cap without spill isError::Memory. The crate has no sandbox backend. Optionalworkflow_dispatchjob runs Linux filesystem-limit tests; default pull-request CI is unchanged. -
Cluster coordinator:
Cluster/ClusterConfigmanage in-process worker slots with CPU, memory, and spill grants, dynamic scale-up/down, and speculative copies of straggler stage tasks (first commit wins). Standalone and Kubernetes layouts live underdeploy/. Optionalworkflow_dispatchjob runs the tiny decision-support suite on two workers. -
Delta Lake and Apache Iceberg tables: snapshot writes of Parquet data files, time travel by version or timestamp, schema evolution (add, rename, widen), and
MERGEwith atomiccreate_newcommits. A concurrent writer that races the same next log or metadata file getsError::Ioinstead of replacing a commit.Session::merge_sqlappliesMERGE INTOto registered tables;merge_delta/merge_icebergcommit a new snapshot. -
JSON (newline-delimited), Avro Object Container Files, and uncompressed ORC: readers with schema inference and projection, plus writers. Nested struct and list types round-trip. PostgreSQL table and query scans push projection, filters, and integer-range partitions into
SELECT; TCP requiresSessionConfig.allow_network(ADR 0008). Optionalpostgresfeature (TLS whenPostgresSource.sslis set) andworkflow_dispatchjob; defaultcargo teststays offline. -
Continuous queries:
StreamQueryruns micro-batches over an unbounded source with event-time watermarks, tumbling/sliding/session windows, complete or append output, and a checkpointed Parquet sink (ADR 0009). A trigger that writes the sink but not the commit marker is replayed with the same batch id. Offset, commit, state, and sink files appear only after a durable write, so a killed process can resume. A default-CI integration test kills a child at random points and checks the sink against a batch reference, including late rows. -
Pluggable catalog: three-part names (
catalog.namespace.table), defaultmizuchi.public, memory and file backends, and aCatalogProvidertrait for an embedder-supplied store (ADR 0010). Partition metadata onTableStatsdrives Hive-path pruning.Session::execute_ddlrunsCREATE/DROP/ALTER TABLE;SHOW TABLESis a query. -
Scalar function library (string, math,
nullif,ARRAY/array_length,EXTRACTon Date/Timestamp columns) in SQL and DataFrame fragments. Combinable user-defined aggregates (CombineAgg,Session::register_udaf, Pythonregister_udaf) merge under parallel workers and spill. Vectorized Python UDFs remain Arrow-array batched. Coverage:docs/functions.md; compare vectorized vs row withscripts/bench_udf.py. -
Window functions:
ROW_NUMBER,RANK,DENSE_RANK, andCOUNT/SUMwithOVER(PARTITION BY,ORDER BY,ROWS/RANGEframes, namedWINDOW). Evidence:window.slt. -
Common table expressions: non-recursive
WITH(later names see earlier) andWITH RECURSIVEthat unions an anchor with a step over a working table until empty. Evidence:cte.slt. -
Subqueries:
INof literals andIN (SELECT ...),NOT IN (SELECT ...)(null-aware anti: a null in the subquery yields unknown),EXISTS/NOT EXISTS(including correlation as a semi or anti join), uncorrelated scalar subqueries, andLATERALderived tables that project outer columns. Evidence:subquery.slt. -
Set operators:
UNION/UNION DISTINCT/UNION ALL,INTERSECT/INTERSECT ALL,EXCEPT/EXCEPT ALL. Left column names; matching column counts; integer/float/null widen; nulls compare equal; first-seen left-then-right order. Evidence:setops.slt. -
DECIMAL/NUMERICSQL types map onto ArrowDecimal32/Decimal64/Decimal128. CAST and arithmetic use Arrow kernels and fail closed when precision exceeds 38 or a value does not fit the chosen width. Evidence:decimal.slt. -
SQL
DATE,TIMESTAMP, andINTERVAL 'n' DAYmap onto Arrow Date32, Timestamp(Microsecond), and Interval(DayTime). CAST,EXTRACT,DATE_TRUNC('day', …),CURRENT_DATE, andNOW()evaluate; the session clock pinsCURRENT_DATE/NOW(). Evidence:datetime.slt. -
Delta Lake and Apache Iceberg on a caller-supplied object store: the same snapshot, time travel, schema evolution, and
MERGEpath as local tables, with exclusivePutMode::Createcommits. Iceberg REST catalog implementsCatalogProvider; TCP requiresSessionConfig.allow_network. Optionalworkflow_dispatchjob talks to an S3 emulator; defaultcargo testuses in-memory storage. -
CI runs object-store S3 emulator tests against MinIO on pull requests and
main. The job fails if those ignored tests are skipped. Defaultcargo teststays offline.