Jetstream Backend Implementation

[k15r]

Description

Implement a new eventing backend based on NATS-Jetstream.
The current NATS based backend provides different delivery guarantees compared to other Backends. With Jetstream we will be able to provide similar at-least-once guarantees.

Acceptance:

• deploy jetstream enabled NATS cluster #12389
• Update Backend CR to allow additional Backend that leverages Jetstream #12390
• Proposal for jetstream stream usage #12391
  • Currently there is no need to configure storage, max size etc individually per subject. This allows us to use one stream for the entire backend.
  • This approach also allows us to setup the stream immediately when the backend is started
• works with the existing Subscription CRD
• push-based consumer
• configurable max-in-flight (mapped to MaxAckPending)
• Use Exponential Message Delivery Backoff in JetStream dispatcher #13697
• performance comparison with existing (non-jetstream) NATS backend #13736
• event delivery policy: use jetstream default for now
• e2e test exists (fast integration)
• Add Jetstream documentation #14013

Implementation

• Implement a new controller/dispatcher to support NATS JetStream #13223
• Implement sending to Nats Jetstream using EPP #13224
• implement selecting the new controller / dispatcher / EPP using the Eventing Backend

---

⚠️ Funnel process - do not remove!

Name	Value
theme	Enterprise-grade
business value	8
technical value	8
user value	8
open-source value	8
effort
requested by

[mfaizanse]

More issues:

• Use an appropriate stream name extracted out of eventPrefix [JetStream] #13544
• Use a better consumer name when creating subscriptions on JetStream #13546

[mfaizanse]

• NATS JetStream rejects new messages if storage is full #14475

[k15r]

Jetstream is now the default backend. Closing as completed

[robinnarsinghranabhat]

@k15r I cannot find any way to use WorkQueuePolicy for my retention.
I have multiple worker servers and I only want the available to process the message.

Also, I couldn't find any documentation on using pull-consumer rather than push-consumer approach.