v1.9.0

Major Features 🔥

• Pods view: The Pods view is revamped for pipeline vertices and MonoVertex, with a golden signals strip for Latency (processing-time P99), Traffic, Backlog, Message Outcomes(drop, nack), and Errors, plus Focus logs for a larger pod-log view.
• Deep links (Copy View): Copy View copies a link to the current screen, including vertex details, spec, logs, and metrics, so the same view can be opened again or shared.
• numa stays up when a UDF crashes: The monitor container is removed and its work runs inside numa. A source, transformer, map, or sink failure reconnects and rebuilds the forwarder instead of restarting the pod.
• Per-message nack: User code can nack a single message and request redelivery, with an optional delay, max deliveries, reason. This works on the source, source transformer, map, and sink.
• Retry strategy for map and source transformers: retryStrategy now applies to map UDFs and source transformers. A message marked as failed is retried with the same backoff settings as the sink.

Installation

kubectl create namespace numaflow-system

Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.9.0/install.yaml

Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.9.0/validating-webhook-install.yaml

New Contributors

• @meichuanyi made their first contribution in #3473
• @AyushSrivastava1818 made their first contribution in #3502
• @devsarvesh92 made their first contribution in #3558
• @callumc34 made their first contribution in #3572
• @DoTuanAnh2k1 made their first contribution in #3573
• @eshawadher made their first contribution in #3595
• @will-flemmer made their first contribution in #3611
• @rahulsgh21 made their first contribution in #3613
• @chiliec made their first contribution in #3644
• @niyaz000 made their first contribution in #3646
• @sbbuilds made their first contribution in #3657
• @akarazhev made their first contribution in #3660
• @chriscarter90 made their first contribution in #3656
• @HamdaanAliQuatil made their first contribution in #3686
• @mchoy made their first contribution in #3687
• @nirajkumarbarot made their first contribution in #3665
• @madsbv made their first contribution in #3719

Full Changelog: v1.8.0...v1.9.0

What's Changed

• fix: option to skip main image rebuilds during make Test% by @vaibhavtiwari33 in #3446
• docs: updated CHANGELOG.md by @github-actions[bot] in #3447
• chore(deps): bump launch-editor from 2.8.0 to 2.14.1 in /ui by @dependabot[bot] in #3454
• fix: remove the trailing & from the Install k3d step by @adarsh0728 in #3460
• fix: wait for deployments during e2e startup by @adarsh0728 in #3465
• chore(deps): bump shell-quote from 1.8.1 to 1.8.4 in /ui by @dependabot[bot] in #3467
• feat: add udf shared helpers and resilient error persistence by @adarsh0728 in #3459
• feat: support micro batching alternative at source by @vaibhavtiwari33 in #3450
• feat(UI): jetstream summary & raft details by @suryapratap-01 in #3466
• docs: fix typo in users list by @meichuanyi in #3473
• fix: pod logs streaming blocked by cleanResponseMiddleware buffering by @suryapratap-01 in #3471
• feat: expose udsink retry count in the system metadata by @yhl25 in #3464
• fix: cleanup wal on eof in accumulator by @yhl25 in #3461
• docs: updated CHANGELOG.md by @github-actions[bot] in #3478
• chore(deps): bump @babel/core from 7.24.9 to 7.29.7 in /ui by @dependabot[bot] in #3477
• chore(deps): bump form-data from 3.0.4 to 3.0.5 in /ui by @dependabot[bot] in #3476
• chore(deps): bump go.mongodb.org/mongo-driver from 1.15.0 to 1.17.7 by @dependabot[bot] in #3479
• feat: keep numa alive across source, transformer, and sink UDF failures by @adarsh0728 in #3475
• chore(deps): bump http-proxy-middleware from 2.0.9 to 2.0.10 in /ui by @dependabot[bot] in #3486
• feat: end to end wiring for per msg nack option by @vaibhavtiwari33 in #3455
• feat: isb stream consumer kv debug info by @suryapratap-01 in #3472
• chore(deps): bump opentelemetry_sdk from 0.31.0 to 0.32.1 in /rust by @dependabot[bot] in #3487
• feat: surface OOM (and other) reasons as free-form text by @vaibhavtiwari33 in #3481
• chore: update docs for accumulator by @vaibhavtiwari33 in #3500
• chore(deps): bump golang.org/x/net from 0.51.0 to 0.55.0 by @dependabot[bot] in #3495
• chore: remove deprecated Go Report badge with Rust Card by @vigith in #3501
• doc: introduce monovertex in getting-started by @vigith in #3506
• chore: upgrade mkdocs and its dependencies by @vigith in #3507
• fix: return detailed vertex status message when unhealthy by @AyushSrivastava1818 in #3502
• feat: show mvtx udf metrics on UI by @adarsh0728 in #3503
• fix(UI): dynamic monovertex dimension for cleaner UI by @suryapratap-01 in #3505
• chore: introduce additional unhealthy waiting statuses by @vaibhavtiwari33 in #3517
• fix(isbsvc): use quorum-aware health threshold for JetStream by @Koalk in #3484
• feat: add securityContext to monitor container by @vigith in #3519
• chore: update roadmap by @yhl25 in #3525
• fix: drain closed accumulator window on re-opened key by @vaibhavtiwari33 in #3494
• fix(UI): source transformer node in pipeline by @suryapratap-01 in #3508
• chore(deps): bump golang.org/x/crypto from 0.51.0 to 0.52.0 by @dependabot[bot] in #3527
• feat(pulsar): log unsupported per-message nack options by @AyushSrivastava1818 in #3529
• feat: metrics to track nacks by @vaibhavtiwari33 in #3534
• fix: remove streaming gate for monovertex metrics in source by @vaibhavtiwari33 in #3535
• feat(pulsar): support consumer dead letter policy and redelivery delay by @AyushSrivastava1818 in #3532
• docs: steps for mem profiling numa and ud containers by @vaibhavtiwari33 in #3536
• feat(UI): move metrics beside pods view by @suryapratap-01 in #3538
• chore(deps): bump websocket-driver from 0.7.4 to 0.7.5 in /ui by @dependabot[bot] in #3541
• chore: Rust 1.97, faster local image build by @BulkBeing in #3542
• fix: apply retention gate when deleting frozen gc files after compaction by @vaibhavtiwari33 in #3533
• feat: introduce a generic map for per message nack options by @vaibhavtiwari33 in #3539
• chore: new k8s events api group by @whynowy in #3553
• feat: Generic ISB by @BulkBeing in #3551
• chore(deps): bump shell-quote from 1.8.4 to 1.10.0 in /ui by @dependabot[bot] in #3561
• chore(deps): bump fast-uri from 3.1.2 to 3.1.4 in /ui by @dependabot[bot] in #3559
• feat: reconnect on map(udf) failures by @adarsh0728 in #3488
• feat(sqs): utilize nack options for SQS by @AyushSrivastava1818 in #3557
• docs: updated CHANGELOG.md by @vaibhavtiwari33 in #3563
• feat: remove monitor container by @adarsh0728 in #3524
• chore: bump google.golang.org/grpc by @vigith in #3565
• chore(deps): bump quinn-proto from 0.11.14 to 0.11.16 in /rust by @dependabot[bot] in #3564
• feat: cron-based MonoVertex autoscaling spec by @devsarvesh92 in #3558
• doc: update email address by @vigith in #3566
• chore(docs): update nats docs by @adarsh0728 in #3568
• fix: Ensure batch consumes it's own EOT by @callumc34 in #3572
• refactor(rust): box nack_options in Message by @DoTuanAnh2k1 in #3573
• refactor(ui): split PodLogs helpers and fix log stream lifecycle by @suryapratap-01 in #3567
• feat(testing): redrive-e2e tests to cover UDF failure scenarios by @adarsh0728 in #3562
• feat(ui): add virtualized rendering for pod logs by @suryapratap-01 in #3575
• chore(deps): bump fast-uri from 3.1.4 to 3.1.5 in /ui by @dependabot[bot] in #3581
• doc: link monovertex bypass with conditional forward by @vigith in #3586
• feat(ui): restyle pod logs toolbar and layout with MUI by @suryapratap-01 in #3584
• feat(ui): add pod log search navigation and visual improvement for logs search by @suryapratap-01 in #3583
• feat: support cron-based autoscaling for source vertices by @AyushSrivastava1818 in #3569
• docs: updated CHANGELOG.md by @github-actions[bot] in #3589
• doc: add Rancher Desktop to container runtime quick-start options by @eshawadher in #3595
• doc: update fallback sink documentation to clarify message lifecycle by @vaibhavtiwari33 in #3593
• feat(ui): N-lines selector for pod logs by @suryapratap-01 in #3591
• doc: fallback doc update based on review comment by @vaibhavtiwari33 in #3597
• doc: add MonoVertex usage guide for HTTP source by @eshawadher in #3596
• chore: upgrade lychee (network certificate fixes) by @adarsh0728 in #3599
• fix: avoid unnecessary calls to SQS during nack by @vaibhavtiwari33 in #3601
• feat: introduce retry config for transformer and map by @vaibhavtiwari33 in #3570
• feat(ui): add Focus logs modal for larger pod log viewing by @suryapratap-01 in #3602
• fix(doc): match docs readme with root readme by @adarsh0728 in #3606
• chore: email update by @whynowy in #3608
• chore(deps): bump github.com/gorilla/websocket from 1.5.0 to 1.5.3 by @dependabot[bot] in #3609
• fix: bound pipeline buffer clean up job lifetime by @will-flemmer in #3611
• fix: stricter health check for pipeline/mvtx in UI by @suryapratap-01 in #3600
• feat: introduce endpoint field for http source by @vaibhavtiwari33 in #3605
• fix: vertex pod recreation on scale min/max change by @rahulsgh21 in #3613
• feat: add JetStream max payload exceeded metric by @adarsh0728 in #3612
• chore(deps): bump postcss-selector-parser from 6.1.1 to 6.1.4 in /ui by @dependabot[bot] in #3618
• chore(deps): bump google.golang.org/grpc from 1.82.1 to 1.83.1 by @dependabot[bot] in #3620
• chore(deps): bump browserslist from 4.23.2 to 4.28.8 in /ui by @dependabot[bot] in #3621
• chore(deps): bump nanoid from 3.3.8 to 3.3.18 in /ui by @dependabot[bot] in #3622
• test: use unique stream name for different js writer tests by @vaibhavtiwari33 in #3624
• chore(deps): bump fast-uri from 3.1.5 to 3.1.7 in /ui by @dependabot[bot] in #3626
• chore(deps): bump @xmldom/xmldom from 0.8.13 to 0.8.15 in /ui by @dependabot[bot] in #3627
• ci: add weekly grouped dependabot config for go, npm, cargo, and github-actions by @vigith in #3630
• test: e2e tests for retry strategies for non-sink components by @vaibhavtiwari33 in #3619
• feat: e2e testing for cron based autoscaling by @rahulsgh21 in #3623
• feat: SQS source can consume multiple queues by @suryapratap-01 in #3615
• fix(docs): UI metrics, pods view and logs tab by @suryapratap-01 in #3634
• fix: spec level validation for conditional forwarding's tag values by @vaibhavtiwari33 in #3636
• docs: updated CHANGELOG.md by @github-actions[bot] in #3641
• chore: upgrade rust to 1.98.1 by @vigith in #3642
• feat: Introduce a KVStoreFactory trait by @vaibhavtiwari33 in #3640
• docs: use content tabs for session reduce examples. Fixes #3030 by @chiliec in #3644
• fix: log NATS server errors and disconnects at ERROR level by @niyaz000 in #3646
• feat: use dns resolver to detect active replicas from daemon by @adarsh0728 in #3643
• fix: remove scale.max dependency and fix broken replica count in mvtx health status by @adarsh0728 in #3649
• chore(deps): bump the go-dependencies group across 1 directory with 2 updates by @dependabot[bot] in #3655
• fix: daemon should not be recreated on scale changes by @adarsh0728 in #3638
• Add TLS caCertSecret support to Pulsar source and sink by @sbbuilds in #3657
• docs: correct when an accumulator UDF must respond to keep the watermark moving by @akarazhev in #3660
• feat(ui): add core observability URL state by @suryapratap-01 in #3652
• feat(ui): add Spec deep links and Copy View UX by @suryapratap-01 in #3663
• feat: allow a startupProbe on the numa container by @chriscarter90 in #3656
• fix(ui): stabilize metrics and logs deep links by @suryapratap-01 in #3664
• docs: refresh readme intro, use cases, and add case studies by @syayi in #3671
• fix: avoid retry count resets across batchMap redrives by @vaibhavtiwari33 in #3661
• feat(server): add API v2 capabilities discovery by @suryapratap-01 in #3658
• docs: retry strategy docs for udf and transformer by @vaibhavtiwari33 in #3676
• feat: propagate ISB type as env var by @vaibhavtiwari33 in #3677
• feat(server): add API v2 pipeline vertex summary endpoint by @suryapratap-01 in #3679
• feat: multi-partition support for pulsar source by @vaibhavtiwari33 in #3682
• chore(deps): bump moment from 2.30.1 to 2.31.0 in /ui in the ui-dependencies group across 1 directory by @dependabot[bot] in #3691
• chore: update to Rust 1.99 by @vigith in #3692
• chore(deps): bump the rust-dependencies group across 1 directory with 3 updates by @dependabot[bot] in #3693
• feat(server): add API v2 pipeline vertex status endpoint by @suryapratap-01 in #3695
• feat: rebuild forwarder on runtime errors by @adarsh0728 in #3673
• fix: store the pending future back in jetstreamwatcher by @vaibhavtiwari33 in #3683
• feat(server): add API v2 MonoVertex summary and status endpoints by @suryapratap-01 in #3696
• fix: serve sidecar-livez during reduce fence wait by @chriscarter90 in #3674
• Add keep alive to daemon connection by @callumc34 in #3703
• fix: release scaler lock before dispatching to workers by @callumc34 in #3706
• fix(server): scope pod log access to Numaflow resources by @HamdaanAliQuatil in #3686
• fix: remove default replicas value from mvtx/vtx CRD by @vaibhavtiwari33 in #3709
• chore: increase the default isb message size limit by @yhl25 in #3694
• fix: resume pipelines unpaused before pause is recorded by @will-flemmer in #3707
• chore(deps): bump rustls from 0.23.31 to 0.23.45 in /rust in the rust-dependencies group across 1 directory by @dependabot[bot] in #3714
• chore(deps): bump the ui-dependencies group across 1 directory with 5 updates by @dependabot[bot] in #3715
• fix(monovertex): keep spec.replicas in sync with real replica count by @mchoy in #3687
• fix(ui): unify pod view dark mode by @suryapratap-01 in #3721
• feat(sqs): validate native dead-letter queue policies by @JagjotB in #3662
• chore(deps): bump github.com/getkin/kin-openapi from 0.144.0 to 0.149.0 in the go-dependencies group across 1 directory by @dependabot[bot] in #3724
• fix(UI): Display Messages/API Requests counts in compact (k/M) format on ISB Service Details page by @adarsh0728 in #3726
• fix(UI): sections in buffers tab in dark theme by @adarsh0728 in #3728
• chore(deps): bump follow-redirects from 1.16.0 to 1.16.1 in /ui in the ui-dependencies group across 1 directory by @dependabot[bot] in #3729
• chore(deps): bump github.com/klauspost/compress from 1.18.4 to 1.18.7 in the go-dependencies group across 1 directory by @dependabot[bot] in #3730
• chore(deps): bump launch-editor from 2.14.1 to 2.14.2 in /ui in the ui-dependencies group across 1 directory by @dependabot[bot] in #3732
• feat: add schedulerName to pod template by @syayi in #3725
• feat: cleanup redrive/reconnect flow by @adarsh0728 in #3702
• fix: only remove offset from tracker after ack/nack succeed by @nirajkumarbarot in #3665
• fix(UI): add copy view for jetstream/isb spec by @adarsh0728 in #3735
• fix(ui): case-insensitive level filter for pipeline pod logs by @suryapratap-01 in #3737
• fix(UI): contextual modal title and chart description in dark mode by @adarsh0728 in #3738
• feat(metrics): add MonoVertex data processing health metric by @suryapratap-01 in #3740
• fix: lower default readTimeout from 1s to 100ms by @vigith in #3741
• feat(ui): beta Pod View for pipeline vertices and MonoVertex by @suryapratap-01 in #3739
• chore: make new pods view default by @adarsh0728 in #3747
• chore(deps): bump the ui-dependencies group across 1 directory with 7 updates by @dependabot[bot] in #3745
• chore(deps): bump rustls from 0.23.45 to 0.23.46 in /rust in the rust-dependencies group across 1 directory by @dependabot[bot] in #3746
• chore(deps): bump github.com/go-jose/go-jose/v4 from 4.1.4 to 4.1.6 in the go-dependencies group across 1 directory by @dependabot[bot] in #3748
• fix(UI): scale badge changes by @adarsh0728 in #3749
• fix: redact secrets in logs (v2) by @madsbv in #3719
• fix(UI): drive Pod View health metrics from discovery signal tags by @suryapratap-01 in #3750