SnowQ

SnowQ is a single-process C++20 analytical query engine built around typed, nullable columnar batches. It loads schema-defined CSV data and executes manually constructed scan, filter, projection, and hash-aggregation pipelines. The project focuses on execution internals and measurable row-versus-column tradeoffs rather than SQL parsing or storage.

Features

• Int64, Double, String, and Bool logical types with typed nulls
• contiguous TypedColumn<T> storage and row-aligned null bytes
• schema-checked Batch and Table containers
• schema-driven CSV loading with quoted fields and explicit null parsing
• batch-at-a-time Volcano operators for scan, filter, and projection
• blocking hash aggregation with multi-column grouping
• COUNT(*), COUNT(column), SUM, MIN, MAX, and AVG
• literal, column, comparison, and boolean expressions
• independent map-per-row reference engine
• optional DuckDB result comparison
• deterministic benchmark and batch-size profiling tools
• Apache Arrow C Data Interface copy adapter for the four logical types

Architecture

CSV file
   |
   v
CsvLoader -> Table<Batch>
                  |
                  v
              ScanOperator
                  |
             FilterOperator
                  |
             ProjectOperator
                  |
          HashAggregateOperator
                  |
                  v
             Result Batch

Operators implement one pull interface:

virtual Result<std::optional<Batch>> next() = 0;

Scan, filter, and projection stream one batch at a time. Hash aggregation is blocking: its first pull consumes the complete child stream before materializing the result.

Requirements

• CMake 3.24 or newer
• a C++20 compiler
• GoogleTest
• Python 3 for reference-tool and benchmark integration tests

DuckDB and Google Benchmark are optional. Neither is linked into the core snowq library.

On macOS with Homebrew:

brew install cmake googletest

On Ubuntu:

sudo apt-get install cmake g++ libgtest-dev python3

Build And Test

cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure

Equivalent Make targets are available:

make configure
make build
make test

The test suite covers column and batch invariants, CSV parsing, expression semantics, streaming behavior, aggregation edge cases, row-reference parity, Arrow import/export, benchmark reporting, and fixed reference queries. The DuckDB comparison is reported as skipped when the Python duckdb package is not installed.

Reference Queries

Generate a deterministic quote/trade CSV:

python3 data/generate_data.py \
  --output /tmp/snowq_quotes.csv \
  --rows 1000 \
  --symbols 10 \
  --seed 42

Run a fixed grouped query through SnowQ:

./build/snowq_reference_queries \
  --input /tmp/snowq_quotes.csv \
  --query group_by \
  --batch-size 1024

The reference tool supports filter_project and group_by. It is deliberately not a SQL parser.

When DuckDB is installed, compare both fixed query results:

python3 scripts/compare_duckdb.py \
  --engine ./build/snowq_reference_queries \
  --data /tmp/snowq_quotes.csv

The comparison uses an explicit DuckDB schema, canonical row ordering, duplicate preservation, typed-null normalization, and floating-point tolerance.

Benchmarks

Configure a Release build with the benchmark runner:

cmake -S . -B build-bench \
  -DCMAKE_BUILD_TYPE=Release \
  -DSNOWQ_BUILD_BENCHMARKS=ON
cmake --build build-bench --target snowq_bench

Run the full deterministic suite:

python3 scripts/run_benchmarks.py \
  --binary build-bench/snowq_bench \
  --suite full \
  --output-dir bench/results \
  --warmups 3 \
  --iterations 20

Representative results from the checked-in report:

Workload	Columnar p50	Row p50	p50 ratio
Scan, 1M rows	2.04 ms	80.01 ms	39.30x
Filter, 1M rows, 50% selected	76.42 ms	274.10 ms	3.59x
Group by, 1M rows, 1K groups	134.03 ms	702.65 ms	5.24x
Filter + group by, 1M rows	77.01 ms	216.69 ms	2.81x

These are local measurements, not general performance guarantees. The complete methodology, p50/p95/p99 values, host/compiler metadata, and binary hash are in bench/results/latest.md. Batch-size sensitivity is reported separately in bench/results/cache_profile.md; it is a cache-sensitive proxy, not a hardware cache-counter measurement.

Data And Null Semantics

• every column in a batch has the same row count
• nulls retain their logical type
• comparisons involving null evaluate to false
• boolean expressions treat null operands as false
• aggregates ignore null inputs except COUNT(*)
• integer SUM reports overflow instead of invoking signed-overflow behavior
• null group keys group together
• grouped aggregate output order is unspecified
• global aggregation over empty input emits one row
• grouped aggregation over empty input emits zero rows with a valid schema

Repository Layout

include/snowq/   public engine interfaces
src/             engine implementation
tests/           C++ and Python correctness tests
tools/           fixed reference-query executable
bench/           benchmark runners and checked-in reports
scripts/         DuckDB comparison and benchmark drivers
data/            deterministic generator and small CSV fixtures

Scope

SnowQ does not implement:

• a SQL parser, binder, planner, or optimizer
• joins
• persistent storage, transactions, or a catalog
• Parquet or Arrow IPC
• predicate pushdown or dictionary encoding
• SIMD or multithreaded execution
• distributed query execution

See DESIGN.md for data-layout decisions, operator behavior, aggregation state, Arrow ownership, correctness references, and benchmark methodology.