Skip to content

Sink v0.17.0

Choose a tag to compare

View all tags
@liran liran released this 18 Sep 15:13
v0.17.0
2dc6503
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Sink v0.17.0 introduces demand-based memory admission across Gateway, Engine, and Worker. Small requests acquire capacity for their actual retained data instead of reserving maximum-size responses. A shared, automatically sized process pool reserves 10% of managed capacity for completion and bounds response-growth waits.

Improvements and fixes

  • Account for input, document copies, response encoding, and forwarding buffers through their full lifetime, including cancellation and batching.
  • Release discarded search buffers, canceled write candidates, and consumed MongoDB count pages; bound Lua-expanded Kafka payloads and preserve safe retries before forwarding.
  • Keep local memory pressure separate from endpoint health and preserve search read budgets after validation failures.
  • Prevent hidden search mutation replays through HTTP retry headers or redirects.
  • Reuse search connections and Kafka retry buffers, and reduce redundant read, queue, routing, and batching allocations.
  • Add memory pressure, waiting, rejection, and driver-reservation metrics with KEDA examples; update Go dependencies and CI tooling.

Upgrade notes

Upgrade all Engines before Gateways. New Gateways use the private streaming forwarding protocol; old Gateways remain compatible with new Engines. Roll back Gateways before Engines. See the rolling upgrade guide.

Replace legacy admission sizing with memory.max_bytes, memory.burst_percent, and memory.wait_timeout. Legacy gateway.max_bytes, gateway.max_requests, gateway.max_requests_per_store, service.execution, and service.publish admission settings remain accepted and validated but no longer control CLI admission. Migrate autoscaling to sink_memory_* metrics. Managed capacity is not an RSS limit; runtime, drivers, Kafka, Lua, and garbage collection still require headroom.

The public Sink protocol is unchanged; sink-go v0.8.0 remains compatible.

Full changelog

Installation artifacts

Docker image:

ghcr.io/batchstream/sink:0.17.0

docker pull ghcr.io/batchstream/sink:0.17.0

Immutable image: ghcr.io/batchstream/sink@sha256:8f27feaeeb1d2ceb9b470f57427522bb71d56f031d79f97953a1e5c0c47259b6

Standalone binaries for Linux and macOS on amd64 and arm64 are attached to this release.
Verify an archive with checksums.txt before installation.

Assets 7
Loading