Major Features 🔥

Distributed Tracing: Numaflow can now export platform traces using OpenTelemetry Protocol (OTLP)
Ordered Processing (input-order preservation): A new opt-in ordered processing mode provides partitioned FIFO semantics
OnSuccess Sink: A new OnSuccess sink allows users to write a follow-up message to a secondary sink after the primary sink has successfully accepted the original message.
MonoVertex Bypass Routing: Mvtx now supports bypass / short-circuit routing, letting users skip the UDF and/or primary sink and send messages directly to a fallback sink or onSuccess sink based on user-defined tags.
Dark mode theme

Installation

kubectl create namespace numaflow-system

# Install numaflow controller and UI
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.8.0/install.yaml

# Install validating webhook (Optional)
kubectl apply -n numaflow-system -f https://github.com/numaproj/numaflow/releases/download/v1.8.0/validating-webhook-install.yaml

New Contributors

@juniemariam made their first contribution in #3086
@txmxthy made their first contribution in #3169
@bobo made their first contribution in #3187
@JagjotB made their first contribution in #3376
@suryapratap-01 made their first contribution in #3407
@DattoDarragh made their first contribution in #3430

Full Changelog: v1.7.0...v1.8.0

What's Changed

docs: updated CHANGELOG.md by @github-actions[bot] in #3055
chore: default to use structured logging by @whynowy in #3056
chore: Fix code formatting in example code in user documentation by @BulkBeing in #3058
fix: Kafka source/sink - pass client certs for mTLS when insecureSkipVerify is false by @BulkBeing in #3059
feat: add support for OnSuccess sink by @vaibhavtiwari33 in #3040
doc: add map to mvtx example by @vigith in #3066
fix: cron parser uses 6 field format by @vigith in #3068
fix(test): we need 5 sec ticker in test by @vigith in #3069
chore: revert to use ReadWriteOnce as default accessMode to avoid breaking changes by @whynowy in #3070
fix: build issues in #3071 by @BulkBeing in #3072
fix: add back drop metrics after sink refactor by @adarsh0728 in #3074
docs: updated CHANGELOG.md by @github-actions[bot] in #3073
doc: a bit more content to pipeline document by @vigith in #3075
fix: use std context by @vigith in #3081
chore: bump crypto from 0.36.0 to 0.45.0 by @vigith in #3082
chore: remove unused eval which uses expr that has CVE by @vigith in #3083
doc: update watermark doc with init source delay by @vaibhavtiwari33 in #3084
doc: Update timeout doc for accumulator where idle detection config is required by @vaibhavtiwari33 in #3085
feat: enable OTLP metrics exporter for daemon services by @juniemariam in #3086
chore(deps): bump node-forge from 1.3.1 to 1.3.2 in /ui by @dependabot[bot] in #3088
fix: alignment and text wrapping in Errors tab by @adarsh0728 in #3078
chore: Add a small elaboration for cron schedule parsing error by @vaibhavtiwari33 in #3094
test: Add e2e test for on-success-sink by @vaibhavtiwari33 in #3067
fix: graceful shutdown of sigkills in map udf container by @yhl25 in #3096
doc: add fallback container for mvtx sink configuration by @vigith in #3097
fix: Remove godebug env var initialization by @vaibhavtiwari33 in #3092
chore: replace kafka with redpanda for e2e testing by @whynowy in #3077
chore: e2e tests for metadata changes by @adarsh0728 in #3065
chore: clean up managedFields from k8s objects from UI display by @whynowy in #3098
doc: Documentation for on-success sink by @vaibhavtiwari33 in #3107
chore: downgrade health check error log to warn by @adarsh0728 in #3108
feat: Spec changes for short-circuiting mvtx by @vaibhavtiwari33 in #3105
feat (UI): support onSuccess sink in MonoVertex graph by @adarsh0728 in #3110
chore: update onSuccess sink image by @adarsh0728 in #3120
chore: Upgrade Rust toolchain to 1.92 by @BulkBeing in #3124
fix: avoid panic in sink code by @yhl25 in #3128
chore(refactor): more idiomatic rust for server info by @KeranYang in #3130
chore: refactor map component by @yhl25 in #3132
feat: Mvtx short-circuiting using a bypass router struct by @vaibhavtiwari33 in #3126
docs: updated CHANGELOG.md by @github-actions[bot] in #3137
docs: Documentation for metric exposition format differences b/w numaflow versions by @vaibhavtiwari33 in #3142
chore: add dummy rust daemon server by @KeranYang in #3138
chore: Metrics changes for Mvtx UDF, on-success sink and drop metrics by @vaibhavtiwari33 in #3139
refactor: define errors for ISB Reader by @vigith in #3148
chore(deps): bump lodash from 4.17.21 to 4.17.23 in /ui by @dependabot[bot] in #3149
refactor: error types and handling for Writer by @vigith in #3151
test: E2E test for bypass scenarios for map and source transformer in monovertex by @vaibhavtiwari33 in #3134
feat: SQS System Attributes and Custom Attributes Propagation by @cosmic-chichu in #3095
chore: code cleanup and cargo fixes by @yhl25 in #3153
lint: fail if cargo check has warnings by @vigith in #3156
docs: Documentation for monovertex bypass routing by @vaibhavtiwari33 in #3152
lint(clippy): clippy fix 144/165 by @vigith in #3157
lint(clippy): remaining(105/165) by @vigith in #3158
lint(clippy): remaining(12/165) by @vigith in #3159
chore: support gRPC for rust mvtx daemon server by @KeranYang in #3143
lint(clippy): fix all the remaining clippy issues by @vigith in #3161
ci: block CI on clippy errors by @vigith in #3163
doc: Update FAQs with a scenario for Server Info File not ready by @vaibhavtiwari33 in #3165
docs(map): call out unary, batch, and streaming by @vigith in #3164
chore: add critical error metric for numaplane analysis template by @adarsh0728 in #3154
Add Starboard to USERS.md by @txmxthy in #3169
chore: adding Playerdata to the list of users by @Koalk in #3171
chore: write a macro for emitting critical errors by @vigith in #3168
chore: support for keys in http source by @yhl25 in #3172
chore: Preallocate Bytes/BytesMut when possible by @BulkBeing in #3173
feat: add ISB Reader Trait and add it as Associated Type in NumaflowTypeConfig by @vigith in #3175
feat: ISB Writer Trait integration (propagated via NumaflowTypeConfig) by @vigith in #3178
chore(deps): bump bytes from 1.10.1 to 1.11.1 in /rust by @dependabot[bot] in #3186
feat(ui): add dark mode theme by @bobo in #3187
refactor: write and resolve should return the same type and Orchestrator to orchestrate retries by @vigith in #3190
refactor: introducing tasks for map operations by @yhl25 in #3167
fix(UI): minor improvements for dark mode by @adarsh0728 in #3195
fix(UI): force graph remount when pipeline structure changes by @adarsh0728 in #3197
fix: Restart on non-retryable kafka source ack errors by @BulkBeing in #3194
docs: add on success sink details in user-defined sinks doc by @vaibhavtiwari33 in #3202
chore(perf): Avoid message cloning in Sink by @BulkBeing in #3203
chore(deps): bump time from 0.3.41 to 0.3.47 in /rust by @dependabot[bot] in #3199
feat(ui): show onSuccess and fallback sink container icons in Pipeline graph by @sapkota-aayush in #3200
fix: use vertex name as partition label for source pending metric by @bobo in #3205
chore(deps): bump jsonpath from 1.1.1 to 1.2.0 in /ui by @dependabot[bot] in #3196
chore(deps): bump webpack from 5.94.0 to 5.105.0 in /ui by @dependabot[bot] in #3207
perf: Use Intra-task concurrency in source transformer by @BulkBeing in #3212
feat(test): a simple in-memory buffer for simulating ISB by @vigith in #3213
refactor: introduce ISBFactory for creating ISB reader/writer by @vigith in #3217
chore: avoid watermark regression logs in source by @yhl25 in #3225
feat: use initContainerStatuses in health checks by @dpadhiar in #3201
fix(UI): pending chart for source vertex by @adarsh0728 in #3209
feat: check initContainerStatuses are in Waiting or Terminated state during health check by @dpadhiar in #3226
feat(ui): horizontal source+transformer row and sink arrow layout. by @sapkota-aayush in #3224
fix(UI): side input fix by @adarsh0728 in #3228
chore(ci): increase timeout for unit tests to 20 minutes by @vigith in #3233
test: use SimpleBuffer for testing ISBReaderOrchestrator by @vigith in #3231
chore: use JetstreamWatcher by @yhl25 in #3239
chore: chunk sqs writes to batch size 10 by @yhl25 in #3240
docs: updated CHANGELOG.md by @github-actions[bot] in #3237
test: ISBWriterOrchestrator tests using SimpleBuffer by @vigith in #3238
chore(deps): bump jsonpath from 1.2.0 to 1.2.1 in /ui by @dependabot[bot] in #3230
test: Utilities for unit testing framework by @vaibhavtiwari33 in #3215
refactor: object-safety for ISBWriter and ISBReader by @vigith in #3245
refactor: enable both h2, http/1.1 for rust mvtx daemon server by @KeranYang in #3180
chore(perf): use trait_variant instead of async_trait by @vigith in #3250
fix: minor fixes for reduce by @yhl25 in #3241
chore(doc): metrics doc updation by @adarsh0728 in #3242
refactor: trait for Key/Value Store with impl for Jetstream by @vigith in #3253
chore: upgrade Rust toolchain to 1.93 by @vigith in #3254
refactor: use KVStore trait for watermark operations by @vigith in #3256
doc: data-plane streaming architecture by @vigith in #3258
chore: parse daemon responses properly by @KeranYang in #3260
docs: updated CHANGELOG.md by @github-actions[bot] in #3255
feat: in-memory kv store with impl KVStore for testing by @vigith in #3261
refactor: nit fixes related to typos, doc comments, method names by @vaibhavtiwari33 in #3266
fix: Handle udsource disconnect when ACK is invoked by @BulkBeing in #3265
fix: check child resources health in isbsvc health check API by @adarsh0728 in #3269
feat(controller): enable ordered processing spec by @vigith in #3278
chore(deps): bump rollup from 2.79.2 to 2.80.0 in /ui by @dependabot[bot] in #3279
chore(deps): bump go.opentelemetry.io/otel/sdk from 1.38.0 to 1.40.0 by @dependabot[bot] in #3280
chore(deps): bump github.com/nats-io/nats-server/v2 from 2.10.27 to 2.11.12 by @dependabot[bot] in #3259
feat(controller): fix scaling for ordered processing and validations by @vigith in #3281
chore: ordered processing should read only from a single partition per processor by @vigith in #3284
fix: shuffle by key for partition-affinity if ordered processing is set by @vigith in #3285
fix: calc pending and watermark for ordered by @vigith in #3286
fix: Mapper UDF error propagation to UI by @BulkBeing in #3288
chore: avoid logging isbsvc client information by @yhl25 in #3295
fix: pending for http to fix autoscaling by @vigith in #3292
fix(test): update tokio runtime flavor used in the spawned OS thread for component servers by @vaibhavtiwari33 in #3290
doc: Ordered Processing by @vigith in #3297
chore(deps): bump quinn-proto from 0.11.12 to 0.11.14 in /rust by @dependabot[bot] in #3301
test: E2E test for ordered processing by @vaibhavtiwari33 in #3300
doc: improve ordered doc by @vigith in #3304
chore: major upgrade for kube and k8s-openapi crates by @vigith in #3307
test: tests for source forwarder using simple buffer ISB by @vaibhavtiwari33 in #3283
fix: Utilize to vertex's ordered processing enabled config when building ToVertexConfig by @vaibhavtiwari33 in #3306
fix: remove trailing comma in UniqueStringList.ToString by @syayi in #3309
test: Unit tests for ordered processing by @vaibhavtiwari33 in #3303
perf: precompile DNS1035 regex as package-level variable by @syayi in #3311
docs: updated CHANGELOG.md by @github-actions[bot] in #3313
refactor(style): remove unnecessary else blocks after return in util functions by @syayi in #3314
opex: emit SQS metrics by @ayildirim21 in #3315
chore(deps): bump google.golang.org/grpc from 1.75.0 to 1.79.3 by @dependabot[bot] in #3323
chore(deps): bump flatted from 3.3.1 to 3.4.2 in /ui by @dependabot[bot] in #3325
test: Tests for sink forwarder using simple buffer backed ISBs by @vaibhavtiwari33 in #3282
chore(deps): bump jsonpath from 1.2.1 to 1.3.0 in /ui by @dependabot[bot] in #3326
fix: map stream can ack inflight messages during panic and tests by @vaibhavtiwari33 in #3267
doc: ordered processing end-to-end example by @vigith in #3328
refactor: Merge Heartbeat and OT Buckets for Watermark Progression by @yhl25 in #3287
chore(deps): bump yaml from 2.4.5 to 2.8.3 in /ui by @dependabot[bot] in #3332
feat: Support configuring port for http source by @vaibhavtiwari33 in #3335
chore(deps): bump node-forge from 1.3.2 to 1.4.0 in /ui by @dependabot[bot] in #3339
feat: http port support for http source by @yhl25 in #3341
chore(deps): bump @xmldom/xmldom from 0.8.10 to 0.8.12 in /ui by @dependabot[bot] in #3343
chore(deps): bump picomatch from 2.3.1 to 2.3.2 in /ui by @dependabot[bot] in #3333
chore(deps): bump lodash from 4.17.23 to 4.18.1 in /ui by @dependabot[bot] in #3345
chore: update golang version to 1.25 by @vaibhavtiwari33 in #3346
chore(deps): bump github.com/nats-io/nats-server/v2 from 2.11.12 to 2.11.15 by @dependabot[bot] in #3329
chore: address review comments from previous PR by @vaibhavtiwari33 in #3347
chore(deps): bump github.com/go-jose/go-jose/v4 from 4.1.3 to 4.1.4 by @dependabot[bot] in #3348
chore: add currentRate and maxRate to monovertex rate limit scaling log by @adarsh0728 in #3349
fix: replace single sleep before fetching pending count with a timeout by @vaibhavtiwari33 in #3350
refactor(test): replace loop in #3350 with backoff::Retry by @vigith in #3351
fix(test): cargo test fails if redis is not running by @vigith in #3352
doc: Update in-built kafka source documentation by @vaibhavtiwari33 in #3354
chore(deps): bump go.opentelemetry.io/otel/sdk from 1.40.0 to 1.43.0 by @dependabot[bot] in #3356
fix: start lastScaledAt as creationTime by default by @Koalk in #3358
chore: remove phantom data markers by @vaibhavtiwari33 in #3360
chore(deps): bump follow-redirects from 1.15.6 to 1.16.0 in /ui by @dependabot[bot] in #3362
chore(deps): bump rand from 0.9.2 to 0.9.3 in /rust by @dependabot[bot] in #3363
fix: Pod deletion (scale down) caused transient unhealthy vertex/mvtx by @whynowy in #3366
tests: Watermark tests by @vaibhavtiwari33 in #3344
chore: liveness check for udf in mvtx by @whynowy in #3372
chore: use correct partition id when multiple topics are used in Kafka source by @yhl25 in #3338
chore(deps): bump github.com/moby/spdystream from 0.5.0 to 0.5.1 by @dependabot[bot] in #3374
chore: Upgrade to Rust 1.95 by @BulkBeing in #3375
docs: update http source documentation for configurable ports by @JagjotB in #3376
fix: allow token cancellation from streaming_read errors by @vaibhavtiwari33 in #3373
chore: components should cancel the token on critical errors by @yhl25 in #3381
fix: include udf in MonoVertex kustomize transformations. Closes #3322 by @JagjotB in #3382
chore: upgrade nats-server to 2.12.7 from 2.10.29 by @vigith in #3384
docs: updated CHANGELOG.md by @github-actions[bot] in #3388
fix: include onSuccess in kustomize transformations by @JagjotB in #3389
refactor: Nack Messages by default to avoid acking failed messages by @yhl25 in #3342
feat: add OTLP export into tracing setup with context propagation helpers by @adarsh0728 in #3385
chore(deps): bump @xmldom/xmldom from 0.8.12 to 0.8.13 in /ui by @dependabot[bot] in #3393
fix: remove unnecessary scrollbars in errors tab by @JagjotB in #3392
chore: update USERS.md entry description for Intuit by @yhl25 in #3396
fix: Exit with non-zero status on error and log errors when we receive duplicate message sink UDF by @BulkBeing in #3398
fix: append parent index information in message ID offset by @vaibhavtiwari33 in #3394
feat: add reduce metrics for rust data plane by @adarsh0728 in #3364
fix: use min event_time across in-flight msgs before publishing idle src wm by @vaibhavtiwari33 in #3400
fix: nack messages during token cancellation in ISBWrite task by @vaibhavtiwari33 in #3404
fix: return result status to cause monovertex controller requeue by @juliev0 in #3405
fix: use podman as fallback when docker is not available in numaflow by @suryapratap-01 in #3407
fix(doc): name reduce_pbq_write was missing the auto-appended _total suffix by @vigith in #3409
chore: consider graceful timeout for batch map operations by @yhl25 in #3406
fix: race between udf server processing and msg id insertion in map for tracking by @vaibhavtiwari33 in #3401
fix: implement getMonoVertexStatus to return correct health status by @suryapratap-01 in #3408
chore(deps): bump fast-uri from 3.0.1 to 3.1.2 in /ui by @dependabot[bot] in #3419
chore(deps): bump @babel/plugin-transform-modules-systemjs from 7.24.7 to 7.29.4 in /ui by @dependabot[bot] in #3420
fix: implement stronger status logic for getPipelineStatus by @suryapratap-01 in #3416
fix: mark JS stream as full until proven otherwise by @vaibhavtiwari33 in #3422
fix: prevent int32 overflow in MonoVertex autoscaler desiredReplicas by @suryapratap-01 in #3421
fix: add monovtx_udf_drop_total metric for MonoVertex map UDF drops by @adarsh0728 in #3427
Adding Kaseya to USERS.md by @DattoDarragh in #3430
chore: update compatibility matrix for numaflow version 1.7/1.8 by @vaibhavtiwari33 in #3426
chore: discard duplicates among inflight messages by @yhl25 in #3429
fix: return with rx early when stream mapper is closed by @vaibhavtiwari33 in #3431
fix(doc): not operator instead of nor by @suryapratap-01 in #3432
chore: update to go 1.25.10 by @vigith in #3433
chore: sort USERS.md company list alphabetically by @yhl25 in #3434
feat: add spans for tracing by @adarsh0728 in #3397
feat: isb spans for tracing by @adarsh0728 in #3437
chore: graceful shutdown instead of being stuck on eot error by @yhl25 in #3439
doc: document tracing configuration and span hierarchy by @adarsh0728 in #3440
feat: introduce concurrency env for udf invocation by @yhl25 in #3317
feat(ui): improve MonoVertex bypass visualization using reactflow by @suryapratap-01 in #3438
chore: Update builtin kafka documentation by @BulkBeing in #3442
fix: remove concurrency default set by the kubebuilder by @vaibhavtiwari33 in #3445

Navigation Menu

Uh oh!

v1.8.0

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

Major Features 🔥

Installation

New Contributors

What's Changed

Contributors

Uh oh!