Sink v0.22.0

Sink v0.22.0 adds unified, bounded Store admission for ready batches and Native Store operations.

execution.queue.max_tasks limits ready tasks (a ready batch counts as one task); execution.queue.max_bytes limits shared waiting bytes across batch collection and admission. Defaults are 10,000 tasks and 128 MiB. batching.queue remains the per-method collection bound, while stores.<name>.max_concurrent controls active Store work. Healthy requests wait cancellably for capacity; a full bounded queue still reports overload.

Full changes: v0.21.0...v0.22.0

Qualification: #129

Installation artifacts

Docker image:

ghcr.io/batchstream/sink:0.22.0

docker pull ghcr.io/batchstream/sink:0.22.0

Immutable image: ghcr.io/batchstream/sink@sha256:a52a4a8807d30bb96cdb811f53152a9e1d27a3f57086c694f5ae89b670c4a2b2

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Sink v0.21.0

Sink v0.21.0 moves the public sink.v1 Go protocol to the shared sink-protocol module and updates per-Store concurrency configuration.

Set the optional Store max_concurrent to override the adaptive execution ceiling for each Engine or Worker process. When omitted, MongoDB defaults to 64 and Elasticsearch/OpenSearch to 128. The former role-level execution.store_max_concurrent is no longer accepted; move an explicit value to the Store configuration.

Full changes: v0.20.0...v0.21.0

Installation artifacts

Docker image:

ghcr.io/batchstream/sink:0.21.0

docker pull ghcr.io/batchstream/sink:0.21.0

Immutable image: ghcr.io/batchstream/sink@sha256:24065ca4eb6948d7371f6ad43ce5df056555a1aa948038b097906ed5f74bb227

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Sink v0.20.0

This release introduces adaptive per-Store backend backpressure and simplifies the document write execution path. It also preserves mutation progress when backend work slows and adds deterministic handling for conditional-write storage failures.

Use sink-go v0.10.1 with the canonical module path github.com/batchstream/sink-go.

Full changes: v0.19.0...v0.20.0

Installation artifacts

Docker image:

ghcr.io/batchstream/sink:0.20.0

docker pull ghcr.io/batchstream/sink:0.20.0

Immutable image: ghcr.io/batchstream/sink@sha256:f21be92eb164235ad220c3fcc8ad8f7803731dac1c1e8a0b94704014e9335caa

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Sink v0.19.0

This release redesigns role configuration, streams read and native page responses, and keeps record revisions internal to storage. It also replaces allocation accounting with process memory watermarks, shares Kafka topic policy and MongoDB settings, and removes obsolete MongoDB concurrency limits.

This is a breaking release. Use sink-go v0.10.0 for the matching streaming protocol and typed request API, and Sink chart v0.9.0 for the new values layout. Recreate configuration from the current examples; older configuration and wire contracts are not supported.

The release commit passed all 15 main-branch CI jobs, including race tests, fuzzing, backend conformance, SDK integration and isolated role qualification. All 15 release qualification and publication checks passed. Published Linux/macOS archives were downloaded and their checksums verified; the Linux amd64/arm64 image was inspected and its arm64 binary returned v0.19.0.

Full changes: v0.18.0...v0.19.0

Installation artifacts

Docker image:

ghcr.io/batchstream/sink:0.19.0

docker pull ghcr.io/batchstream/sink:0.19.0

Immutable image: ghcr.io/batchstream/sink@sha256:e6c429dc4f6f96d677b4d64e6fe32ac16964b43ddbacf102b8be42a9a90977b8

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Sink v0.18.0

Sink v0.18.0 adds bounded structured diagnostic logs across Gateway, Engine, and Worker, with optional direct OTLP export to an existing OpenTelemetry Collector.

Improvements

• Emit warn-level JSON diagnostics on stderr by default, with per-component levels, stable diagnostic labels, bounded messages, and rate-limited repeated failures.
• Support OTLP over gRPC or HTTP/protobuf with TLS verification, bounded queues, explicit loss reporting, and bounded shutdown flushing. Collector outages do not block startup or affect readiness.
• Preserve RPC application-failure diagnostics when responses use the memory-managed protocol wrapper. Request bodies and backend error text remain excluded from ordinary RPC summaries; supported failure bodies require explicit opt-in.
• Expand logging, backend, fault-recovery, and coverage qualification. Component unit tests, input fuzzers, and unit microbenchmarks remain in Sink; integration and workload qualification is maintained in the production suite with immutable CI pins.

Upgrade

The public Sink protocol is unchanged and sink-go v0.8.0 remains compatible. Existing configurations use the default console logging; configure logging.otlp only when exporting to a Collector. See the logging reference for output limits, labels, TLS, and shutdown settings.

When upgrading from Sink 0.16.x or earlier, advance all Engines before Gateways for the private forwarding protocol introduced in 0.17. Roll back Gateways before Engines. See the rolling upgrade guide.

Full changelog

Installation artifacts

Docker image:

ghcr.io/batchstream/sink:0.18.0

docker pull ghcr.io/batchstream/sink:0.18.0

Immutable image: ghcr.io/batchstream/sink@sha256:9a5cef79198cbda78e6c4b596746079c7633233393e20c9f70928c078a6429e2

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Sink v0.17.0

Sink v0.17.0 introduces demand-based memory admission across Gateway, Engine, and Worker. Small requests acquire capacity for their actual retained data instead of reserving maximum-size responses. A shared, automatically sized process pool reserves 10% of managed capacity for completion and bounds response-growth waits.

Improvements and fixes

• Account for input, document copies, response encoding, and forwarding buffers through their full lifetime, including cancellation and batching.
• Release discarded search buffers, canceled write candidates, and consumed MongoDB count pages; bound Lua-expanded Kafka payloads and preserve safe retries before forwarding.
• Keep local memory pressure separate from endpoint health and preserve search read budgets after validation failures.
• Prevent hidden search mutation replays through HTTP retry headers or redirects.
• Reuse search connections and Kafka retry buffers, and reduce redundant read, queue, routing, and batching allocations.
• Add memory pressure, waiting, rejection, and driver-reservation metrics with KEDA examples; update Go dependencies and CI tooling.

Upgrade notes

Upgrade all Engines before Gateways. New Gateways use the private streaming forwarding protocol; old Gateways remain compatible with new Engines. Roll back Gateways before Engines. See the rolling upgrade guide.

Replace legacy admission sizing with memory.max_bytes, memory.burst_percent, and memory.wait_timeout. Legacy gateway.max_bytes, gateway.max_requests, gateway.max_requests_per_store, service.execution, and service.publish admission settings remain accepted and validated but no longer control CLI admission. Migrate autoscaling to sink_memory_* metrics. Managed capacity is not an RSS limit; runtime, drivers, Kafka, Lua, and garbage collection still require headroom.

The public Sink protocol is unchanged; sink-go v0.8.0 remains compatible.

Full changelog

Installation artifacts

Docker image:

ghcr.io/batchstream/sink:0.17.0

docker pull ghcr.io/batchstream/sink:0.17.0

Immutable image: ghcr.io/batchstream/sink@sha256:8f27feaeeb1d2ceb9b470f57427522bb71d56f031d79f97953a1e5c0c47259b6

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Sink v0.16.0

Store credentials can now be read from mounted Kubernetes Secret files. Configure storage.mongodb.uri_file or search username_file, password_file, and api_key_file with absolute paths instead of placing credentials in ordinary runtime YAML.

Files are bounded to 1 MiB, preserve exact UTF-8 bytes, and are read once at startup. Missing, unreadable, empty, invalid or conflicting references fail validation without printing file contents. Kubernetes projected-volume symlinks are supported. Rotation requires a rolling restart with overlapping backend credentials until every old Pod exits; see credential handling.

This is an additive runtime configuration feature. The public protocol is unchanged and sink-go v0.8.0 remains compatible. Inline server configuration is still supported; Helm chart 0.3 uses per-field Secret references.

Installation artifacts

Docker image:

ghcr.io/batchstream/sink:0.16.0

docker pull ghcr.io/batchstream/sink:0.16.0

Immutable image: ghcr.io/batchstream/sink@sha256:0b66a1ec7099bfd42228115ea43d36aaa800bdb3df6c930a60ac2394e95efdcf

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Sink v0.15.0

This release introduces Store-isolated Gateway, Engine and Worker roles and the URI-only protocol. Pair it with sink-go v0.8.0.

Breaking deployment change: deploy a new cluster with new Kafka topics and consumer groups, then cut clients over together. Existing mutation envelopes and scan checkpoints are not migration inputs. Follow the new-cluster migration guide.

• Gateway owns static Store routing and stable record affinity. Each Engine and Worker binds to one Store, isolating its backend and processing resources.
• Canonical record/native resource URIs replace old address fields; Store names are the sole routing identity, and MongoDB native operations support collection-bound resources.
• HTTP health uses a dedicated port independently of opt-in Prometheus. Shutdown now withdraws HTTP readiness before background work and accepted RPCs drain.
• Gateway removes unnecessary single-owner hashing and single-group goroutine fanout while preserving affinity mapping across versions.
• Adds real DNS withdrawal/cache/outage counterexamples, graceful and forced drain regressions, Worker 1→3→0→1 qualification, MongoDB election/majority-loss tests and a fixed-resource workload matrix.

The rollout guide explains why DNS refresh alone is insufficient: cache lifetime, discovery failure, retained requests and the complete process shutdown budget all matter. The backend environment guide records the MongoDB/kernel compatibility constraint reproduced during qualification.

Installation artifacts

Docker image:

ghcr.io/batchstream/sink:0.15.0

docker pull ghcr.io/batchstream/sink:0.15.0

Immutable image: ghcr.io/batchstream/sink@sha256:b240d56f3df686e9e42d863deb7ac7f52bddf321397b43ba919f665ddfbf21a7

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Qualification evidence: full report, fixed-resource performance data, and release artifact verification. The report separates successful reconciliation from offered-load targets that were not sustained, and records the Docker-only scope and deployment limits.

Sink v0.14.0

Upgrade

This release changes the YAML configuration schema. Migrate server, worker and CLI configuration together using the configuration migration guide, then run sink config check --config /etc/sink/config.yaml before starting the new binary. Duration strings and readable byte sizes are supported. Stored documents, Kafka mutation encoding and topic/group identities are unchanged.

Changes

• Bound synchronous admission by queued request count, retained input bytes, per-store count and wait time. Short bursts can wait for capacity; cancellation releases queue entries and waiting consumes the request deadline.
• Wake eligible waiters individually and reserve known returning-Put sizes, improving useful CPU utilization while preserving per-RPC response budgets and store isolation.
• Add Scan field projection, per-store execution-byte limits and bounded Scan admission. Pair with sink-go v0.7.0 for projected pages and bounded retries of explicitly marked pre-execution admission rejections.
• Reduce Lua environment allocations and preserve cancellation after admission wakeups.
• Align CI, release and sustained qualification to the same immutable production suite with current SDK coverage and historical regression proofs.

Performance and qualification

In the documented synthetic MongoDB test with one Sink container limited to 1 CPU and 256 MiB, returning Put improved from 2,662 to 17,531 successful RPC/s (6.6x); exact Count improved by 29%. A three-minute mixed workload completed 1.71 million RPCs with zero errors and sampled peak RSS of 33.4 MiB. These are measured workload results, not a production sizing guarantee. See the full report.

The final main CI and current seven-store production suite passed against release commit 005a056c89e7540b5a882657267fe0b7f9bfda0c. Coverage includes current Scan projection/retry behavior, bounded synchronous admission, returning-Put reservations, ordinary race tests, and historical regression sensitivity.

The sustained fault qualification ran its workload for 2h 2s with 16 concurrent workers: 67,686 completed cycles and 406,116 logical operations, with 65 reconciled transients and 65 mutation retries. Worker termination, OpenSearch outages and Kafka restarts were followed by readiness recovery, zero ordinary consumer lag/dead letters, and explicit permanent-failure DLQ repair/replay checks. This run used 5c667ad7f34d55cda05b619f0be3c8ebe2c2cf2d; its runtime source is identical to the release commit, which adds only a deterministic test fixture correction. These disposable-backend gates do not substitute for deployment-specific topology and recovery validation.

Installation artifacts

Docker image:

ghcr.io/liran/sink:0.14.0

docker pull ghcr.io/liran/sink:0.14.0

Immutable image: ghcr.io/liran/sink@sha256:f777d1d00ad590c7aa0a23d0707c3d16da36adfb567dd9b9efb7284756bee1cc

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Sink v0.13.0

Sink v0.13.0 hardens batching, Lua execution, backend boundaries, and asynchronous processing, and adds per-store operational metrics.

Upgrade requirements

• Remove service.batching.enabled from both server and worker configurations before upgrading. Batching is always enabled, and the strict configuration parser rejects the removed field for either value.
• Search Kafka partition keys now use store, dataset, and record key without the logical namespace. When upgrading an existing search Kafka deployment, stop publishing, drain the old topic, and switch publishers and workers together. Mixing key schemes can route the same record to different partitions. Apply the same coordination when rolling back.
• Merge now has one create-or-merge behavior. The former missing_document_mode field is reserved and ignored, including for older clients and queued records. Review callers relying on MissingDocumentFail and Lua scripts handling a missing current document. Upgrade callers to sink-go v0.6.0, whose wire format preserves create-or-merge compatibility with older servers.
• Operational metrics now include a bounded store label. Review dashboard aggregations and recording rules when adopting per-store metrics.

Changes

• Make bounded request batching mandatory and improve read batching, caller isolation, cancellation, and returned-document budget accounting.
• Bound Lua allocation, Unicode conversion, pattern matching, table operations, and failure responses; preserve JSON/BSON types and validate UTF-8.
• Correct MongoDB numeric record identities, conditional writes, duplicate/conflict classification, Count/Scan behavior, and query validation.
• Harden Elasticsearch and OpenSearch query bounds, pagination sizing, response identities, endpoint recovery, and write retry behavior.
• Split worker batches rejected by admission capacity without replaying successfully applied work, and preserve physical-record ordering across search namespaces.
• Attribute request, write, batching, publisher, and worker metrics to configured stores.
• Update the immutable public qualification suite with regression and per-store metric assertions.

Validation

The release candidate passed the full server CI, including race tests, lint, mutation-envelope fuzzing, MongoDB ReplicaSet integration, Elasticsearch/OpenSearch integration, public API conformance on OpenSearch 2.17 and 3.8, client protocol checks, Docker Compose quickstart, and release artifact builds. Local full race tests, vet, and staticcheck also passed.

Release binaries and container images are gated by the pinned public production qualification suite.

Full changes: v0.12.1...v0.13.0

Installation artifacts

Docker image:

ghcr.io/liran/sink:0.13.0

docker pull ghcr.io/liran/sink:0.13.0

Immutable image: ghcr.io/liran/sink@sha256:c0ac1379313ff64da773f4186cc1411f76421a0edfa07a0b2852fb6e0369ab9b

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.