clink v0.8.0

clink v0.8.0 is the launch release: the production-qualification programme
run across the engine's guarantee surface, and the repository reshaped for
public use.

Ten qualification campaigns, published green. After v0.7.0's QUAL-01,
ten more campaigns ran under continuous fault injection, judged by
independent oracles, and were published only once green with evidence
retained: PostgreSQL two-phase commit, S3 staged multipart commits, 29 GiB
of keyed state on a disaggregated backend, bounded state through declared
retention, wide job graphs at 147 operators and 292 subtasks, content-level
agreement with an independent reference engine on 19 of 19 queries, a
rolling engine upgrade with exactly-once continuity, infrastructure faults
from ENOSPC to stepped clocks, a running job's keyed-state type changed and
migrated at restore, and a declared security-refusal matrix. Every result:
https://orhaugh.github.io/clink/qualification/

Engine fixes. The campaigns and the sanitizer matrix forced out a long
list of defects, fixed in this release - among them: savepoints pinned
against retention for the life of their job; a committing sink held open
until its final commit lands; a job whose checkpoints fail persistently
fails instead of crashlooping; a terminal job's HA manifest retired so
recovery cannot resurrect it; channel closes carrying a reason so a cancel
never reads as end-of-input; SQL sinks writing the table's declared schema;
declared state_ttl bounding DISTINCT and set operations; a wire decoder's
reserve bounded by the frame's bytes (one corrupt frame could otherwise
cost a 22 GB allocation); and a worker connection kept alive across an
off-lock send (a use-after-free in the coordinator under same-id worker
re-registration). The full list is in the CHANGELOG.

Runtime image. jemalloc is the image's default allocator with Arrow's
pool routed to it: under repeated recovery, allocator retention kept
gigabytes of freed memory resident, and the paired defaults return it -
measured, not inferred. The image is published for amd64 and arm64, and one
command runs a complete example, data included:

docker run --rm ghcr.io/orhaugh/clink-runtime:0.8.0 run /opt/clink/examples/sql/hello.sql

Compatibility. No REST API breaks. Wire protocol v2 unchanged in
negotiation (CommitCheckpoint gained a backwards-compatible tail field).
Snapshot format unchanged. Plugin ABI v1 unchanged.

Documentation: https://orhaugh.github.io/clink/

Artefacts

• ghcr.io/orhaugh/clink-runtime:0.8.0 - runtime image, linux/amd64 and linux/arm64.
• clink-v0.8.0-linux-x86_64-ubuntu24.04.tar.gz - relocatable SDK (CLI, headers, libraries); checksum published alongside it.
• pyclink-0.8.0-py3-none-macosx_14_0_arm64.whl - macOS arm64 wheel,
  sha256 b2d74f478d1cf355f1b08a9bf3c2a3917b99d43689c8597f2d42a0dc96164b5e.
  Other platforms: build from source, or point CLINK_LIB at an SDK library
  and pip install "git+https://github.com/orhaugh/clink@v0.8.0#subdirectory=python".

clink v0.7.0 - the qualification release

The qualification release. The centre of gravity across 318 commits is one question: do the guarantees hold when processes die at the worst possible instant? The answer is now published evidence rather than an architecture claim.

Highlights

QUAL-01: Kafka exactly-once, qualified and published. A windowed aggregation from Kafka through the transactional sink ran two hours on a multi-host rig under continuous fault injection - kills armed inside the two-phase-commit protocol's own windows, coordinator SIGKILLs, broker restarts and outages, network partitions - and an independent seeded oracle judged 755/755 windows byte-exact: zero missing, zero duplicates, zero foreign. Full report: https://orhaugh.github.io/clink/qualification/qual-01-kafka-exactly-once/

The exactly-once machinery behind it. Commit-confirmed restores, prepared-transaction resume over the Kafka wire protocol (SASL/PLAIN, SCRAM-SHA-256 with server verification, TLS), durable commit receipts with replay suppression, in-doubt resolution with receipt materialisation and unresolved-orphan markers, and the sink's pre-fence describe - which refuses to open a producer while its predecessor's transaction is unknowable.

Hot rescale. An eligible operator's parallelism changes as an in-place cutover at a checkpoint barrier; only the rescaled operator's subtasks cycle. Graceful stop-at-savepoint lands alongside.

Production-hardening round closed (F1-F101) - per-operator key-group slices, real TTL on collection/CEP state, SQL refusing clauses it used to silently drop, protocol-corrupt receives failing loudly, and more.

Compatibility made explicit. Wire protocol v2 negotiated (v1 peers retained for rolling upgrades); snapshot format version enforced; frozen-bytes fixtures pin each encoding; plugin ABI v1 unchanged. No REST API breaks.

Observability. Real OTLP export (metrics + lifecycle spans), checkpoint staleness, per-job state size, a shipped Grafana dashboard and alert runbook.

Testing surface. Source/sink contract suites where a capability claim is a test obligation, libFuzzer targets whose findings become regression tests, and the SQL differential oracle.

Full changelog: https://github.com/orhaugh/clink/blob/main/CHANGELOG.md

Artifacts

• Python wheel pyclink-0.7.0 (attached; sha256 8072195032778fdc7205f6032eaf7d50597c7cff53ee04e80e7187a1d4cff084)
• Runtime image ghcr.io/orhaugh/clink-runtime:0.7.0
• Linux SDK clink-0.7.0-linux-x86_64-ubuntu24.04.tar.gz (relocatable, attached by release-binaries.yml)

clink v0.6.0

Two engine improvements to the SQL-native AI surface, both surfaced by driving it
with a real downstream consumer. No REST API or state-format breaks; plugin ABI
unchanged (v1).

• Metadata pre-filter on VECTOR_SEARCH. A trailing
  filter_eq='query_col:corpus_col,...' option scopes each query to the corpus
  rows whose named columns equal the query's (a null query value imposes no
  constraint). It is a genuine per-query pre-filter - the operator scores only the
  matching corpus subset - so restricting a similarity search by metadata (a
  document's system, tenant, and so on) does not lose recall the way post-filtering
  a top-k would.
• clink replay reconstructs linked-impl operators. The replay command now
  registers the linked impls the same way clink run does, so a captured job that
  uses an impl operator (VECTOR_SEARCH, ML_PREDICT, a connector) replays and
  --verifys with no plugin - any job clink can run, it can replay.

Full notes: CHANGELOG.md.

Consume a release without a compiler

Every release carries a prebuilt Linux x86_64 SDK:

curl -fsSLO https://github.com/orhaugh/clink/releases/download/v0.6.0/clink-v0.6.0-linux-x86_64-ubuntu24.04.tar.gz
tar xzf clink-v0.6.0-linux-x86_64-ubuntu24.04.tar.gz
./clink-v0.6.0-linux-x86_64-ubuntu24.04/bin/clink run pipeline.sql

A relocatable prefix - the clink CLI and clink_node daemon, static libraries,
headers, and the CMake package, with the pinned Arrow bundled and $ORIGIN
rpaths - built at an Ubuntu 24.04 glibc floor. Point CMAKE_PREFIX_PATH at it and
find_package(clink) resolves. Scope is the dependency-free impl set (SQL,
file/Parquet, RocksDB state, HTTP, TLS, WebSocket, Avro, vector search); anything
needing an external client library or object-store SDK is a from-source build.

clink v0.5.0

Cluster-hardening release. A class of jobs the earlier releases never exercised
end to end - a bounded source with checkpointing on, side-output sinks, and a
fan-out topology, deployed as a compiled job plugin and recovered across a hard
worker kill - now completes and recovers cleanly. Every fix was found by driving
that shape with a real downstream consumer. No REST API or state-format
breaks; plugin ABI unchanged (v1).

Highlights:

• Checkpoint barriers and watermarks now reach side-output channels, so a
  checkpointed bounded job with a side-output sink no longer hangs at its
  end-of-stream final checkpoint.
• Worker-loss recovery rolls the whole job back instead of leaving surviving
  upstreams holding stale bridges to relocated peers.
• Cluster-built sinks get a stable, unique identity, so sibling 2PC sinks no
  longer collide on one PREPARE TRANSACTION gid.
• clink replay works on plugin-typed operators, not just SQL Row
  (--plugin=<so> --verify / --emit-test).
• A fluent CEP timed-out side output (PatternStream::select_with_timed_out)
  for alerting on the absence of a match, and the runtime image now ships as a
  job SDK.

Full notes: CHANGELOG.md.

Consume a release without a compiler

Every release carries a prebuilt Linux x86_64 SDK:

curl -fsSLO https://github.com/orhaugh/clink/releases/download/v0.5.0/clink-v0.5.0-linux-x86_64-ubuntu24.04.tar.gz
tar xzf clink-v0.5.0-linux-x86_64-ubuntu24.04.tar.gz
./clink-v0.5.0-linux-x86_64-ubuntu24.04/bin/clink run pipeline.sql

A relocatable prefix - the clink CLI and clink_node daemon, static
libraries, headers, and the CMake package, with the pinned Arrow bundled and
$ORIGIN rpaths - built at an Ubuntu 24.04 glibc floor. Point
CMAKE_PREFIX_PATH at it and find_package(clink) resolves. Scope is the
dependency-free impl set (SQL, file/Parquet, RocksDB state, HTTP, TLS,
WebSocket, Avro, vector search); anything needing an external client library or
object-store SDK is a from-source build.

clink v0.4.0

A small, focused release: a new connector, prebuilt Linux binaries, the
coordinator serving the real ops console same-origin, and one crash fix
that lean builds of v0.3.0 need. No REST API or format breaks.

Full notes: CHANGELOG.md.

Consume a release without a compiler

Every release now carries a prebuilt Linux x86_64 SDK:

curl -fsSLO https://github.com/orhaugh/clink/releases/download/v0.4.0/clink-v0.4.0-linux-x86_64-ubuntu24.04.tar.gz
tar xzf clink-v0.4.0-linux-x86_64-ubuntu24.04.tar.gz
./clink-v0.4.0-linux-x86_64-ubuntu24.04/bin/clink run pipeline.sql

A relocatable prefix - the clink CLI and clink_node daemon, static
libraries, headers and the CMake package, with the pinned Arrow bundled and
$ORIGIN rpaths - so both the CLI and find_package(clink) work straight
out of the tarball (-DCMAKE_PREFIX_PATH=<extracted dir>). Built at an
honest Ubuntu 24.04 glibc floor (the name says so) and smoke-tested
from a clean directory as both a CLI and a linked consumer before upload.
Scope is the dependency-free impl set: SQL, file/Parquet, RocksDB state,
HTTP, TLS, WebSocket, Avro, vector search. Object stores (s3://, ...) and
broker connectors remain source-build territory, where they are on by
default.

WebSocket source

connector='websocket' connects to a ws:// or wss:// push feed - the
delivery mechanism of most market-data and event APIs - sends the venue's
subscribe message, and emits each text message as a record. A declared
format='json' schema rides the columnar JSON decode exactly as a Kafka
table does. RFC 6455 is implemented in-tree over POSIX sockets, pinned in
tests to the RFC's own worked examples, adding zero dependencies (wss
uses OpenSSL when present). Delivery is stated plainly: a push stream has
no offsets, so at-most-once across restarts - bridge to Kafka for
durability, or pair with the flight recorder to make an unreplayable feed
locally replayable. Verified against a real venue: one inline clink run
statement pulls live trades off a public exchange stream through TLS into a
file - the one-liner is in
docs/connectors/websocket.md.

The coordinator serves the real console

The embedded dashboard SPA - a hand-rolled page predating the
clink-fe ops console - is removed.
In its place, clink_node --http-static-dir=<dir> serves any built console
bundle (clink-fe's dist/) same-origin at / beside the JSON API: one
port, no CORS setup, no separate web server, with SPA deep-link fallback
and a traversal-guarded, unit-tested resolver. Without the flag, /
answers with a JSON signpost. Every /api/v1 route, /metrics, the SSE
stream, auth and CORS are untouched.

Crash fix for lean builds

v0.3.0 crashed at CLI startup (In not registered) in any build without
the Iceberg impl: vector_search registered its Row-channel operator without
registering the Row type, which Iceberg's install had been masking in full
builds. Fixed by idempotent self-registration; found by
market-pulse, the downstream
consumer showcase, whose CI builds exactly that subset. The Iceberg impl
also now skips itself cleanly against an Arrow built without S3 instead of
poisoning every consumer link.

Artifacts

• Linux SDK: clink-v0.4.0-linux-x86_64-ubuntu24.04.tar.gz (+ .sha256), attached below.
• pyclink wheel (macOS 14+, Apple silicon, self-contained): pyclink-0.4.0-py3-none-macosx_14_0_arm64.whl, sha256: 8ff797acf364fc76e2b694c290211b14799519e7e81f2371cd5d1c90ea1dbac3. The v0.2.0 caveats carry over (no object-store filesystems, no HTTPS in the HTTP subsystem).
• Runtime image: ghcr.io/orhaugh/clink-runtime:0.4.0 (also :latest).

clink v0.3.0

Ninety-nine commits of engine, benchmark and correctness work since v0.2.0.
No public header was removed or renamed, and snapshot and savepoint
encodings are unchanged - the GROUP BY accumulator's byte layout was audited
for this release. Operators that previously lost state at a restore now
persist it, so a v0.2.0 snapshot still restores and simply carries none of
that newly persisted state. One behavioural note: the keyed-shuffle routing
fix below means a savepoint restored across the upgrade can move keys between
subtasks, the same class of movement as a rescale.

Full notes: CHANGELOG.md.

The benchmark suite is published, and every query is gated

The full 17-query nexmark suite now runs on a five-node cluster against
canonical data, each query correctness-gated against an independent oracle at
parallelism 4 and every window kind gated cross-engine. Headline, measured:
clink processes an event for 1.9x to 5.3x less CPU (median 2.45x) than a
JVM stream processor producing identical output - method, caveats and raw
per-run data at Benchmarks,
priced out at Cost and footprint.
Two earlier q18/q19 figures were found unsound, withdrawn and re-measured.

Making that gate honest found real defects, now fixed

The row and columnar carriers of a keyed shuffle disagreed about which
subtask a key belongs to (the row side read the key through a double; 74.5%
of FNV-folded keys misrouted at parallelism 4, silently splitting group
state). A data batch larger than the send-credit window was silently
dropped; a failed send vanished instead of failing the task; an
unpartitioned stateful operator was fanned out to produce N answers. All
fixed, each with a test that failed against the original code.

A restore now preserves what was open

An open window survives a restore (tumbling, hopping, session and cumulate -
previously every open window was silently lost), and so do top-N retained
rows, LAST_N state, the OVER aggregate's sync path, and the upsert and
netting sinks' compaction view. Window arithmetic is floored rather than
truncated, event-time reads are exact, and window arguments fail at bind
time.

Also in this release

• Performance: projection pushdown reaches the columnar JSON bridge
  (45% off the shuffle split, 27% off decode); per-group aggregate state
  fell by a third (AggState 264 -> 104 bytes); the windowed fire stopped
  scanning every group per watermark (4.4x at 200k groups); the Kafka
  source fetches in batches; parallel pipeline instances co-locate, taking
  forward edges off the network.
• SQL: WHERE accepts expression operands; a MATCH_RECOGNIZE DEFINE
  predicate with an expression operand (price < PREV(price) * 0.997) now
  matches instead of silently never firing; declared FLOAT and DECIMAL are
  honoured at columnar JSON decode; doubles are no longer truncated to six
  significant digits.
• Embedded: the engine configures logging on first open and honours
  CLINK_LOG_LEVEL, so clink run and pyclink stop printing registry
  chatter that could not be turned off.
• Packaging: installed binaries carry an rpath to the pinned toolchain -
  a host cmake --install previously produced a binary macOS refused to
  load; CLINK_ISA_BASELINE and opt-in CLINK_WITH_JEMALLOC build knobs.

The three consumer-facing fixes (rpath, MATCH_RECOGNIZE expression
operands, embedded logging) were found by
market-pulse, a downstream
consumer showcase built against this release, on its first day running
against an installed clink.

Artifacts

pyclink wheel (macOS 14+, Apple silicon)

pip install pyclink-0.3.0-py3-none-macosx_14_0_arm64.whl

Self-contained: it bundles libclink with a static Arrow and vendors no
dylibs at all. The v0.2.0 caveats carry over unchanged: no object-store
filesystems (s3://, gs://, abfs://) and no HTTPS in the HTTP
subsystem; a source build keeps both.

sha256: 3743f309fc34940e5f311a7258447499d6b9c6889173d86ec5a344e393ebfcd8

Runtime image (linux/amd64)

ghcr.io/orhaugh/clink-runtime:0.3.0 (also :latest, :main,
:sha-<short>), used by docker-compose, the Helm chart and the Kubernetes
operator.

clink v0.2.0

Forty-four commits of engine, build and CI work since v0.1.0. No API or format
breaks: state written by 0.1.0 restores unchanged, and every new behaviour is
either on by default with a documented opt-out or off by default.

Full notes: CHANGELOG.md.

Distribution is the headline

clink now ships artifacts rather than only source. The runtime image is published,
and the Helm chart, the Kubernetes operator and its samples default to it - before
this release they referenced an image tag that existed only on the author's
machine, so the Kubernetes path was undeployable by anyone else. Prebuilt pyclink
wheels build on a release tag again. Documentation publishes to
orhaugh.github.io/clink.

Columnar execution now covers both ends of a pipeline

Columnar JSON decode is the default for Kafka tables (columnar_decode='false'
opts out), with an adaptive damper so systematically unfaithful data pays ~1.6%
instead of 2x, and the keyed shuffle splits columnar batches on the cluster path
with zero row decode. Operators can now also emit born-columnar output: an
append-only INNER join and the windowed fire build typed Arrow columns directly
instead of a name-keyed row per emission, enabled by the planner only where a
consumer can ingest columnar.

Gated nexmark q12: sustained slope 1.08M -> 1.83M rec/s (+69%), CPU 113s ->
54s. Born-columnar emission adds -11% CPU on a join under a filter.

Also in this release

• Performance: scratch keys and transparent state probes removed the
  per-record key-string build from the window, session, aggregate (-31% on the
  q12 row path) and equi-join (-30%) operators; a pre-sorted bulk join-output
  build took the join to -54% cumulative.
• State: opt-in ForSt backend (CLINK_WITH_FORST=ON) with forst://,
  changelog+forst://, s3+forst:// and s3sst+forst:// for live remote data
  files, plus a deferred-read mode that puts hot-path state in the engine.
• Correctness: an input is closed only when closed AND drained, fixing an
  end-of-stream data-loss race; a worker registration is installed before it is
  acked.

Artifacts

pyclink wheel (macOS 14+, Apple silicon)

pip install pyclink-0.2.0-py3-none-macosx_14_0_arm64.whl

Self-contained: it bundles libclink with a static Arrow and vendors no
dylibs at all, so it depends on nothing but macOS system libraries. Verified by
installing into a clean venv and running a windowed GROUP BY through the bundled
library.

sha256: 6a94099b06cb0a2e0e9db745138fb5f2c97f58a6c26ba3598219e0d6c5b05b8c

Two deliberate reductions, both of which are what let the wheel keep a macOS
14 floor rather than requiring the build host's much newer version:

• No object-store filesystems. s3://, gs:// and abfs:// paths do not
  work from the wheel.
• No HTTPS in the HTTP subsystem. It serves and fetches plain HTTP only.

Local files, SQL, the embedded engine and everything else are unaffected. A build
from source keeps both capabilities - they are on by default there.

Linux wheels are not published: the pinned toolchain image is Debian trixie, whose
glibc floor would make them installable only on very new distributions.

Runtime image (linux/amd64)

docker pull ghcr.io/orhaugh/clink-runtime:0.2.0
helm install my-clink deploy/helm/clink   # defaults to this image

Also tagged :latest and :sha-9929808d6655. Anonymous pull verified.

Prebuilt toolchain archives

Prebuilt pinned-toolchain archives (Arrow + Parquet + iceberg-cpp) consumed by scripts/fetch-deps.sh to skip the from-source bootstrap. Not a clink release: these are build-dependency artifacts, keyed by the version pins in scripts/versions.env and verified against the sha256 pins committed in scripts/deps-checksums.txt. Rebuilt via the deps-artifacts workflow (Linux) or scripts/package-deps.sh (macOS).

clink v0.1.0

Initial public release: the engine as described in the README.

Naming, settled for 1.0: the cluster roles are the coordinator (control
plane) and workers (subtask hosts), clink_node --role=coordinator|worker;
the fluent API entry point is clink::api::Pipeline; the in-process test
cluster is clink::test::TestCluster. Domain vocabulary (watermarks, windows,
checkpoints, savepoints, keyed state, key groups, slots) is unchanged.

In brief:

• Typed operator DAG and fluent API on a local runtime and a distributed
  Coordinator/Worker runtime (TLS/mTLS, HA, HTTP API + dashboard).
• Event time end to end: watermarks, tumbling/sliding/session windows,
  interval joins, CEP.
• Keyed and broadcast state over in-memory, file-backed, RocksDB, and
  changelog backends; rescale, schema evolution, savepoints.
• Exactly-once checkpointing with true 2PC sinks (file, Kafka, Parquet,
  S3, Postgres) and an effectively-once upsert family.
• Arrow-native columnar wire format and columnar operator fast paths.
• SQL frontend: embedded (clink run, libclink C ABI, pyclink, Flight
  SQL) or submitted to a cluster.
• Deterministic incident replay: flight recorder, state-diff,
  replay --verify, frozen regression bundles.
• State as data: snapshots are Arrow IPC; export to Parquet/Iceberg;
  queryable live state.
• Connector suite across messaging, object storage, table formats,
  databases, and HTTP endpoints (see docs/connectors/).
• Public testing framework (clink::test), Kubernetes Helm chart and
  operator, reproducible benchmark harnesses.