v0.9.0

Major Features

multi-partitioned buffers for higher throughput

• We can now support > 10K TPS (up to 30K TPS)
• To enable multi-partition, please read this doc

Other features

• Support to run the map vertex in streaming mode with batch size set to 1. doc
• UI improvements
  - More accurate processing rates in the UI using pod metrics.
  - Support for multi-partitioned edges.
  - better performance.
• Introduce an optional Validating Admission Webhook to prevent operations such as updating InterStepBufferService immutable fields.
• Capability to increase max message size upto 64mb.

Major Bug Fixes

• Fix intermittent failure from Kafka to get consumer offsets. issue
• Fix pods not rescheduled after eviction. issue
• Fixed the message size limitation issue (only messages with size < 32K were allowed).

Breaking Changes and Upgrading Guide

• ISBSVC with Redis 6.0.6 is not supported any more, using 7.0.11. Anyone who is using old version, please delete the old object, update the version and recreate.

• The buffer and bucket definitions are revised, therefore we updated the buffer and bucket implementation, which requires recreating the pipeline during upgrade (1. Delete the pipelines; 2. Upgrade; 3. Recreate the pipelines). Meanwhile, we removed the following properties:

  1. Removed spec.edges[].parallelism, use spec.vertices[].partitions.
  2. Removed spec.edges[].limits.*, use spec.vertices[].limits.*, * includes bufferMaxLength and bufferUsageLimit.

• If upgrading from v0.8.0, please delete and recreate the pipeline.

Pull Requests

v0.8.0...v0.9.0

Contributors

@KeranYang
@vigith
@yhl25
@juliev0
@chromevoid
@dpadhiar
@veds-g
@whynowy
@kohlisid
@xdevxy

Installation

kubectl create namespace numaflow-system
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v0.9.0/install.yaml