Skip to content

enhancement(nats sink): add support for JetStream message headers#23510

Merged
pront merged 13 commits intovectordotdev:masterfrom
benjamin-awd:nats-sink-header
Aug 6, 2025
Merged

enhancement(nats sink): add support for JetStream message headers#23510
pront merged 13 commits intovectordotdev:masterfrom
benjamin-awd:nats-sink-header

Conversation

@benjamin-awd
Copy link
Contributor

@benjamin-awd benjamin-awd commented Aug 2, 2025

Summary

This PR adds support in the nats sink for JetStream message headers.

It introduces a configurable, templated Nats-Msg-Id header that ensures a unique ID for each message. This enables broker-level deduplication, resulting in stronger delivery guarantees and exactly-once semantics when combined with idempotent consumers.

This PR also adds proper integration tests for JetStream.

Vector configuration

sinks:
    my_jetstream_sink:
        type: nats
        inputs: [ "my_source" ]
        endpoint: "nats://localhost:4222"
        subject: "my.subject"
        jetstream:  # jetstream: true will still be supported for backwards compatability  
            enabled: true
            headers:
                message_id: "{{ event_id }}"

How did you test this PR?

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Closes: #23509

@benjamin-awd benjamin-awd requested review from a team as code owners August 2, 2025 15:17
@github-actions github-actions bot added domain: sinks Anything related to the Vector's sinks domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation labels Aug 2, 2025
@benjamin-awd benjamin-awd changed the title Nats sink header feat(nats sink): add support for Jetstream message headers Aug 2, 2025
@benjamin-awd benjamin-awd changed the title feat(nats sink): add support for Jetstream message headers enhancement(nats sink): add support for Jetstream message headers Aug 2, 2025
@benjamin-awd benjamin-awd changed the title enhancement(nats sink): add support for Jetstream message headers enhancement(nats sink): add support for JetStream message headers Aug 2, 2025
Copy link
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@pront pront enabled auto-merge August 6, 2025 15:47
auto-merge was automatically disabled August 6, 2025 16:06

Head branch was pushed to by a user without write access

@benjamin-awd
Copy link
Contributor Author

benjamin-awd commented Aug 6, 2025

EDIT: figured it out -- I used two slashes instead of three for the enabled docstring

Running into a weird issue with make generate-component-docs, for some reason the doc-string for enabled isn't going through:

	jetstream: {
	...
		required: false
		type: object: options: {
			enabled: {
				description: "Whether to enable Jetstream."
				required: false
				type: bool: default: false
			}

I guess there must be something I'm missing from my config here:

/// Configuration for sending messages using NATS JetStream.
#[configurable_component]
#[serde_with::serde_as]
#[derive(Clone, Debug, Eq, PartialEq, Default)]
#[serde(deny_unknown_fields)]
pub struct JetStreamConfig {
    // Whether to enable Jetstream.
    #[configurable(derived)]
    #[serde(default)]
    pub enabled: bool,

    /// A map of NATS headers to be included in each message.
    #[configurable(metadata(docs::templateable))]
    #[serde(default)]
    pub(super) headers: Option<NatsHeaderConfig>,
}

impl From<bool> for JetStreamConfig {
    fn from(enabled: bool) -> Self {
        Self {
            enabled,
            ..Default::default()
        }
    }
}

@pront pront enabled auto-merge August 6, 2025 16:42
@pront pront added this pull request to the merge queue Aug 6, 2025
Merged via the queue into vectordotdev:master with commit a9c4c16 Aug 6, 2025
54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow headers in NATS Jetstream sink

3 participants