---

Embeddable vectorised query engine for fixed schema integer data that parses a restricted SQL dialect into a relational MLIR, performs logical optimisation before lowering through the SCF, MemRef, Vector and LLVM dialects, emits LLVM IR, and executes fully native kernels through ORC JIT

My goal with this was to keep it intentionally narrow because predictable specialisation is more valuable than broad SQL compatibility for the workloads the engine is intended to serve. Typical use cases include latency triage, risk limit aggregation, market data quality validation and compact batch analytics embedded into OCaml or C++ services where minimising execution overhead is more important than supporting the full relational feature set. The fixed schema contract makes kernel specialisation deterministic, while the combination of a stable C ABI and the Arrow C Data Interface allows existing columnar producers to invoke compiled kernels directly without any input serialisation/intermediate data transformation

It currently operates over a single in memory columnar table containing signed i32/i64 columns and implements relational operators including filter, project, sum, count and grouped aggregation over a single key. Joins, NULL semantics, strings, floating point types, storage, and transactional behaviour are deliberately excluded, not because they are fundamentally difficult to implement, but because omitting them keeps the execution model simple the optimisation space deterministic and the generated kernels aggressively specialised for the workloads the engine is designed to execute

---

Build

cmake --preset release \
  -DMLIR_DIR=/usr/lib/llvm-19/lib/cmake/mlir \
  -DLLVM_DIR=/usr/lib/llvm-19/lib/cmake/llvm
cmake --build --preset release -j

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DVEIN_BUILD_OCAML=ON \
  -DMLIR_DIR=/usr/lib/llvm-19/lib/cmake/mlir \
  -DLLVM_DIR=/usr/lib/llvm-19/lib/cmake/llvm
cmake --build build -j

cmake --install build --prefix "$HOME/.local"

The install exports a versioned CMake package. The exact target names follow;

Vein::Query
Vein::RelDialect
Vein::C

---

Query semantics

The schema token uses a table name + ordered column names and scalar types

requests:status:i32,latency_us:i64,service:i32

The SQL grammar supports projection expressions, signed comparison operators, boolean predicates and wrapping integer arithmetic. Aggregate queries support SUM and COUNT with filtering through WHERE while grouped aggregation is expressed using GROUP BY. Arithmetic expressions promote operands to wrapping i64 whereas projecting a bare column preserves its original i32 representation. Global aggregates over empty inputs produce a single row containing zero valued aggregate results while grouped aggregates over empty inputs produce no rows. Grouped output is emitted in ascending order of the signed grouping key to provide deterministic results

The grammar deliberately enforces several restrictions to simplify optimisation and code generation. The grouping key must appear as a plain projected column, SUM accepts only a direct input column rather than an arbitrary expression and joins and subqueries are not supported. Features including ORDER BY, HAVING, DISTINCT and grouping over multiple keys are also intentionally excluded from the supported grammar

---

Example(s)

The OCaml example passes Bigarrays through the stable ABI:

LIBRARY_PATH="$PWD/build${LIBRARY_PATH:+:$LIBRARY_PATH}" \
LD_LIBRARY_PATH="$PWD/build" \
ORIGIN='$ORIGIN' \
dune exec examples/request.exe

----

Compiler inspection

Run the following command to inspect the complete pass pipeline:

./build/veinq explain \
  --schema 't:x:i32,p:i32' \
  --rows 35 \
  --query 'SELECT SUM(x), COUNT(*) FROM t WHERE p > 0' \
  --mode vector \
  --dump all

The compilation begins with relational MLIR followed by its logically optimised form. Physical planning then lowers into fused SCF and MemRef loops with Vector dialect IR emitted whenever vectorisation legality succeeds. The final stages consist of LLVM dialect IR and textual LLVM IR before native code generation. Constant folding and predicate pushdown are applied prior to physical fusion to maximise optimisation opportunities

Vector width is determined directly from the referenced column types. i32 columns map to eight lane SIMD vectors while i64 columns use four lanes. Runtime dispatch selects the vectorised execution path only when the input satisfies the row count and alignment assumptions encoded by the compiled kernel. Filter masks are combined with tail masks before reductions or compressed stores to preserve correctness without introducing scalar cleanup paths

---

C ABI

The C ABI exposes opaque handles for schemas, queries, results and errors. Every API call returns vein_status_t preventing C++ exceptions from crossing the ABI boundary. VEIN_ABI_VERSION defines the binary compatibility contract while all extensible configuration structures begin with a struct_size so new members can be appended without breaking existing callers

The shared library exports only the versioned C API declared by the public headers. All compiler impl details remain hidden behind the ABI boundary so that a stable external interface independent of internal implementation changes

---

Arrow C data

vein/Arrow.h accepts struct arrays from Arrow C Data with signed int32 and int64 children that are nonnullable. Input value buffers stay in place so execution copies no values. Separate entry points distinguish borrowing from ownership transfer bc release callback semantics must stay explicit. Primitive child offsets are supported. Root struct offsets are rejected explicitly

Result export produces independently owned Arrow C Data objects with release callbacks. Exported values are copied so their lifetime does not depend on the Vein result handle. Child move semantics remain valid after parent release

Arrow IPC and Parquet decoding are not implemented in the compiler core. Use an Arrow library adapter to decode those formats. Then pass its C Data objects to Vein

---

OCaml

The optional Dune package provides OCaml bindings over the C ABI mapping Bigarray values directly onto the columnar interface. Input Bigarray views are borrowed for the duration of execution to allow queries to consume existing buffers without introducing copy overhead. Result accessors materialise managed Bigarray values for safe ownership transfer. Compilation options expose execution mode, SIMD policy, optimisation level and optional resource limits

The installed package ships with both bytecode and native artefacts alongside the dynamic C stub. At runtime the stub resolves the installed Vein shared library directly, avoiding any dependency on the original build tree

When installing to a custom prefix both the OCaml toolchain and the platforms native dynamic loader must be able to locate the installation. The compiles the installed sample application through Findlib:

OCAMLPATH="$HOME/.local/lib/ocaml" \
CAML_LD_LIBRARY_PATH="$HOME/.local/lib/ocaml/stublibs" \
LIBRARY_PATH="$HOME/.local/lib${LIBRARY_PATH:+:$LIBRARY_PATH}" \
LD_LIBRARY_PATH="$HOME/.local/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
ocamlfind ocamlopt -package vein -linkpkg \
  "$HOME/.local/share/vein/examples/request.ml" \
  -o request
./request