Skip to content

clink v0.2.0

Choose a tag to compare

@orhaugh orhaugh released this 25 Jul 15:33
· 588 commits to main since this release

Forty-four commits of engine, build and CI work since v0.1.0. No API or format
breaks
: state written by 0.1.0 restores unchanged, and every new behaviour is
either on by default with a documented opt-out or off by default.

Full notes: CHANGELOG.md.

Distribution is the headline

clink now ships artifacts rather than only source. The runtime image is published,
and the Helm chart, the Kubernetes operator and its samples default to it - before
this release they referenced an image tag that existed only on the author's
machine, so the Kubernetes path was undeployable by anyone else. Prebuilt pyclink
wheels build on a release tag again. Documentation publishes to
orhaugh.github.io/clink.

Columnar execution now covers both ends of a pipeline

Columnar JSON decode is the default for Kafka tables (columnar_decode='false'
opts out), with an adaptive damper so systematically unfaithful data pays ~1.6%
instead of 2x, and the keyed shuffle splits columnar batches on the cluster path
with zero row decode. Operators can now also emit born-columnar output: an
append-only INNER join and the windowed fire build typed Arrow columns directly
instead of a name-keyed row per emission, enabled by the planner only where a
consumer can ingest columnar.

Gated nexmark q12: sustained slope 1.08M -> 1.83M rec/s (+69%), CPU 113s ->
54s
. Born-columnar emission adds -11% CPU on a join under a filter.

Also in this release

  • Performance: scratch keys and transparent state probes removed the
    per-record key-string build from the window, session, aggregate (-31% on the
    q12 row path) and equi-join (-30%) operators; a pre-sorted bulk join-output
    build took the join to -54% cumulative.
  • State: opt-in ForSt backend (CLINK_WITH_FORST=ON) with forst://,
    changelog+forst://, s3+forst:// and s3sst+forst:// for live remote data
    files, plus a deferred-read mode that puts hot-path state in the engine.
  • Correctness: an input is closed only when closed AND drained, fixing an
    end-of-stream data-loss race; a worker registration is installed before it is
    acked.

Artifacts

pyclink wheel (macOS 14+, Apple silicon)

pip install pyclink-0.2.0-py3-none-macosx_14_0_arm64.whl

Self-contained: it bundles libclink with a static Arrow and vendors no
dylibs at all, so it depends on nothing but macOS system libraries. Verified by
installing into a clean venv and running a windowed GROUP BY through the bundled
library.

sha256: 6a94099b06cb0a2e0e9db745138fb5f2c97f58a6c26ba3598219e0d6c5b05b8c

Two deliberate reductions, both of which are what let the wheel keep a macOS
14 floor rather than requiring the build host's much newer version:

  • No object-store filesystems. s3://, gs:// and abfs:// paths do not
    work from the wheel.
  • No HTTPS in the HTTP subsystem. It serves and fetches plain HTTP only.

Local files, SQL, the embedded engine and everything else are unaffected. A build
from source keeps both capabilities - they are on by default there.

Linux wheels are not published: the pinned toolchain image is Debian trixie, whose
glibc floor would make them installable only on very new distributions.

Runtime image (linux/amd64)

docker pull ghcr.io/orhaugh/clink-runtime:0.2.0
helm install my-clink deploy/helm/clink   # defaults to this image

Also tagged :latest and :sha-9929808d6655. Anonymous pull verified.