v1.5.0

Major Features 🔥

• Introducing Serving
• Accumulator
• Contextual Debugging

Recommend SDK

• Please use 0.9 and above. Refer Compatibility Matrix
• Contextual debugging only available for MonoVertex

🚨 Breaking Change 🚨

• None

Experimental 🚀

• Async data movement in Numaflow backed by Rust

What's Changed

• feat: add sdk infomation metrics by @whynowy in #2208
• feat: use sidecar for ud containers [Breaking k8s < v1.29] by @whynowy in #2230
• feat: Built-in Pulsar source by @BulkBeing in #2237
• feat: Asynchronous Streaming for Source and Sink in Numaflow Core by @yhl25 in #2251
• feat: counter metrics visualizer for pipeline by @adarsh0728 in #2238
• feat: Introducing Tracker to track completeness of the Message by @yhl25 in #2264
• feat: Asynchronous Map Implementation for Pipeline by @yhl25 in #2295
• feat: Added support for kafka OAuth authentication by @SzymonZebrowski in #2257
• feat: contextual flow for metrics by @veds-g in #2345
• feat: rows/page dropbox. Fixes #1764 by @SaniyaKalamkar in #2337
• feat: adaptive lookback for monovertex by @kohlisid in #2373
• feat: add sqs source by @cosmic-chichu in #2355
• feat: Accumulator by @yhl25 in #2475
• feat: validating webhook for MonoVertex by @whynowy in #2497
• feat: surface application errors by @adarsh0728 in #2499
• feat: errors tab for application errors by @veds-g in #2473
• feat: [experimental] start golang vs. rust image based on env (#2537) by @vigith in #2538
• feat: persist error in case of partial/no responses from udf by @adarsh0728 in #2539
• feat: Builtin serve sink by @BulkBeing in #2551
• feat: Support for pending count with Adaptive lookback by @RohanAshar in #2486
• feat: Improves Serving Latency and Throughput by @yhl25 in #2564

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0/validating-webhook-install.yaml

New Contributors

• @SzymonZebrowski made their first contribution in #2257
• @SaniyaKalamkar made their first contribution in #2337
• @jacque1ine made their first contribution in #2360
• @sesame0224 made their first contribution in #2457
• @t-wata made their first contribution in #2466
• @ccpeng made their first contribution in #2504
• @borkox made their first contribution in #2526

Full Changelog: v1.4.6...v1.5.0

v1.5.0-rc5

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc5/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc5/validating-webhook-install.yaml

v1.5.0-rc4

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc4/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc4/validating-webhook-install.yaml

v1.5.0-rc3

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc3/validating-webhook-install.yaml

v1.4.6

What's Changed

• chore: calculate pending in replica 0 only
  Full Changelog: v1.4.5...v1.4.6

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.6/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.6/validating-webhook-install.yaml

v1.4.5

What's Changed

• fix: broken retry strategy in pipeline sink by @yhl25 in #2625
• feat: expose metrics for pipeline/mvtx desired and current phase by @whynowy in #2563

Full Changelog: v1.4.4...v1.4.5

v1.5.0-rc2

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc2/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc2/validating-webhook-install.yaml

v1.5.0-rc1

Major Features 🔥

• Introducing Serving
• Accumulator
• Contextual Debugging

Recommend SDK

• Please use 0.9 and above. Refer Compatibility Matrix
• Contextual debugging only available for MonoVertex

🚨 Breaking Change 🚨

• None

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc1/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.5.0-rc1/validating-webhook-install.yaml

Other Changes

Features

• feat: add sdk infomation metrics by @whynowy in #2208
• feat: use sidecar for ud containers [Breaking k8s < v1.29] by @whynowy in #2230
• feat: unify metrics ( cleanup and add missing metrics ) by @adarsh0728 in #2207
• feat: Built-in Pulsar source by @BulkBeing in #2237
• feat: Asynchronous Streaming for Source and Sink in Numaflow Core by @yhl25 in #2251
• feat: counter metrics visualizer for pipeline by @adarsh0728 in #2238
• feat: Introducing Tracker to track completeness of the Message by @yhl25 in #2264
• feat: Conditional Forwarding In Asynchronous Pipeline by @yhl25 in #2270
• feat: mvtx gauge metrics support by @veds-g in #2259
• feat: counter metrics visualizer for mono-vertex by @adarsh0728 in #2256
• feat: pipeline gauge metrics by @veds-g in #2284
• feat: Asynchronous Map Implementation for Pipeline by @yhl25 in #2295
• feat: enhance time range selector for metrics - include presets by @adarsh0728 in #2292
• feat: Implement Sourcer traits for serving source by @BulkBeing in #2301
• feat: make unit and display name configurable by @adarsh0728 in #2269
• feat: Added support for kafka OAuth authentication by @SzymonZebrowski in #2257
• feat: expose pending messages metric in async pipeline by @yhl25 in #2330
• feat: logs enhancements by @veds-g in #2320
• feat: Enhance Serving To Support Flatmap Operations by @yhl25 in #2324
• feat: contextual flow for metrics by @veds-g in #2345
• feat: implement callbacks from vertices when serving is used as source by @BulkBeing in #2311
• feat: rows/page dropbox. Fixes #1764 by @SaniyaKalamkar in #2337
• feat: Watermark Implementation for Async Data Movement by @yhl25 in #2376
• feat: adaptive lookback for monovertex by @kohlisid in #2373
• feat: moving ud containers logs as default in pods view by @veds-g in #2378
• feat: Controller changes to support Serving as a builtin source by @BulkBeing in #2357
• feat: cpu memory metrics by @adarsh0728 in #2332
• feat: contextual flow for remaining metrics by @veds-g in #2379
• feat: updating chart lables and tooltip by @veds-g in #2387
• feat: add sqs source by @cosmic-chichu in #2355
• feat: Idle Watermark Implementation Inside Async Data Movement for Source and ISB by @yhl25 in #2385
• feat: Serving as a builtin source with Monovertex by @BulkBeing in #2382
• feat: attach emptyDir volume for runtime info by @adarsh0728 in #2431
• feat: serving feature complete by @vigith in #2422
• feat: Accumulator by @yhl25 in #2475
• feat: validating webhook for MonoVertex by @whynowy in #2497
• feat: surface application errors by @adarsh0728 in #2499
• feat: Jetstream source implementation in Rust by @BulkBeing in #2484
• feat: vertex error endpoints by @veds-g in #2519
• feat: errors tab for application errors by @veds-g in #2473
• feat: scaffold ServingPipeline by @whynowy in #2468
• feat: Serving - Expose metrics by @BulkBeing in #2533
• feat: [experimental] start golang vs. rust image based on env (#2537) by @vigith in #2538
• feat: persist error in case of partial/no responses from udf by @adarsh0728 in #2539
• feat: enhancing metrics tab and charts by @veds-g in #2544
• feat: Builtin serve sink by @BulkBeing in #2551
• feat: Serving - option to specify container template by @BulkBeing in #2554
• feat: Support for pending count with Adaptive lookback by @RohanAshar in #2486
• feat: expose metrics for pipeline/mvtx desired and current phase by @whynowy in #2563
• feat: add missing metrics for mvtx by @adarsh0728 in #2542
• feat: generic Write Ahead Log framework by @vigith in #2571
• feat: Improves Serving Latency and Throughput by @yhl25 in #2564
• feat: integrate sqs source by @cosmic-chichu in #2403

Fixes

• fix: init container logs by @veds-g in #2245
• fix: honor lookbackSeconds in monovertex and rust pipeline by @kohlisid in #2258
• fix: Sink Config to respect Fallback by @yhl25 in #2265
• fix: "Loading Pods View" and "failed to get pod details" state on UI by @adarsh0728 in #2248
• fix: Numaflow serving sink by @yhl25 in #2103
• fix: set max decode size of proto message, add mvtx metrics by @yhl25 in #2283
• fix: Initialize rustls's CryptoProvider early in the code by @BulkBeing in #2312
• fix: lastScaledAt not updated during autoscaling down by @whynowy in #2323
• fix: clean up metrics for deleted mvtx by @whynowy in #2338
• fix: more criteria to detect lifecycle changes, fixes: #2331 by @whynowy in #2346
• fix: units and tooltip content for metric charts by @adarsh0728 in #2333
• fix: error handling in case of udsink failures by @adarsh0728 in #2372
• fix: get status of init containers in pods-info API call by @adarsh0728 in #2371
• fix: timestamp and level filter fix for logs by @veds-g in #2374
• fix: error state & search/negate search in case of prev terminated logs by @adarsh0728 in #2391
• fix: filter out deamon pod and unnecessary containers from metrics output by @adarsh0728 in #2394
• fix: sorting metrics chart tooltip values by @veds-g in #2401
• fix: error handling and grace shutdown during udf failures by @yhl25 in #2406
• fix: legends overlapping metrics charts by @veds-g in #2453
• fix: Recreate KV watcher if it never returns any events by @BulkBeing in #2474
• fix: concurrent invocations to update active pods list in rater by @adarsh0728 in #2478
• fix: main is broken by @vigith in #2490
• fix: typo, use timeout instead of ttl by @vigith in #2494
• fix: add active pod function for runtime errors by @adarsh0728 in #2515
• fix: create a new bidirectional stream for every partition to avoid race condition by @yhl25 in #2521
• fix: sidecar health check for long running queries in async data movement by @yhl25 in #2541
• fix: format stacktrace by @veds-g in #2548
• fix: start persisting errors after first active pods u...

v1.4.4

Bug fixes 🐞

• Sink stops when we have multiple partitions #2516 by @yhl25 in #2517
• Pod tracker going out of the tick interval by @adarsh0728 in #2478

Full Changelog: v1.4.3...v1.4.4

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.4/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.4/validating-webhook-install.yaml

v1.4.3

What's Changed

• chore: publish read count metric when no messages are read by @yhl25 in (#2433)
• refactor: introducing lifecycle to Vertex spec by @whynowy in (#2419)
• fix(controller): replicas calculation for paused pipeline by @whynowy in (#2412)
• fix: honor lookbackSeconds in monovertex and rust pipeline by @kohlisid in (#2258)
• feat: Added support for kafka OAuth authentication by @SzymonZebrowski in (#2257)
• fix(controller): multiple issues related to min/max of a vertex by @whynowy in (#2398)
• fix: more criteria to detect lifecycle changes, fixes: #2331 by @whynowy in (#2346)
• fix: clean up metrics for deleted mvtx by @whynowy in (#2338)
• chore: expose metrics for min and max replicas by @whynowy in (#2335)
• chore: skip publishing the last empty struct when generator stops by @KeranYang in (#2327)
• fix: lastScaledAt not updated during autoscaling down by @whynowy in (#2323)
• chore: use domain-qualified finalizer name by @whynowy in (#2304)
• chore: use same metric to expose binary build info by @whynowy in (#2293)
• chore: always publish watermark to sink OT by @whynowy in (#2288)
• fix: sorting metrics chart tooltip values by @veds-g in (#2401)
• fix: filter out deamon pod and unnecessary containers from metrics output by @adarsh0728 in (#2394)
• fix: error state & search/negate search in case of prev terminated logs by @adarsh0728 in (#2391)
• feat: updating chart lables and tooltip by @veds-g in (#2387)
• feat: contextual flow for remaining metrics by @veds-g in (#2379)
• feat: cpu memory metrics by @adarsh0728 in (#2332)
• feat: moving ud containers logs as default in pods view by @veds-g in (#2378)
• fix: timestamp and level filter fix for logs by @veds-g in (#2374)
• fix: get status of init containers in pods-info API call by @adarsh0728 in (#2371)
• feat: rows/page dropbox. Fixes #1764 by @SaniyaKalamkar in (#2337)
• fix: units and tooltip content for metric charts by @adarsh0728 in (#2333)
• feat: contextual flow for metrics by @veds-g in (#2345)
• chore: display error mssg based on api error response for metric charts by @adarsh0728 in (#2298)
• feat: logs enhancements by @veds-g in (#2320)
• feat: make unit and display name configurable by @adarsh0728 in (#2269)
• feat: enhance time range selector for metrics - include presets by @adarsh0728 in (#2292)
• chore: e2e tests for metrics APIs by @adarsh0728 in (#2277)
• feat: pipeline gauge metrics by @veds-g in (#2284)
• feat: counter metrics visualizer for mono-vertex by @adarsh0728 in (#2256)
• feat: mvtx gauge metrics support by @veds-g in (#2259)
• fix: "Loading Pods View" and "failed to get pod details" state on UI by @adarsh0728 in (#2248)
• feat: counter metrics visualizer for pipeline by @adarsh0728 in (#2238)
• fix: init container logs by @veds-g in (#2245)
• feat: unify metrics ( cleanup and add missing metrics ) by @adarsh0728 in (#2207)
• chore: infinite retry for connecting to UDF server, avoid publishing metrics when there is no data by @yhl25 in (#2399)

New Contributors

• @SaniyaKalamkar made their first contribution in #2337
• @SzymonZebrowski made their first contribution in #2257

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.3/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.4.3/validating-webhook-install.yaml

Full Changelog: v1.4.2...v1.4.3