Support for Kafka message headers #1718
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lalithsuresh
approved these changes
May 6, 2024
added 2 commits
May 6, 2024 12:06
Th `optimized` profile creates artifacts in `target/optimized`, not `target/release`. Signed-off-by: Leonid Ryzhyk <leonid@feldera.com>
Added `headers` field to Kafka output connector config to specify a list
of headers to be added to each message.
Kafka headers are a list of key-value pairs. Keys are strings. The
same key can occur multiple times, therefore the ordering is important.
The value component of the header is an optional byte array. For
convenience, we allow specifying it as an actual array of bytes in JSON
(`[1,2,3,4,5]`) or as a UTF8-encoded string ("foobar"). Here is an
example Kafka connector config with headers that I added to the SecOps
demo:
```
"config": {
"format": {
"name": "avro",
"config": {
"schema": schema,
"registry_urls": [pipeline_to_schema_registry],
}
},
"transport": {
"name": "kafka_output",
"config": {
"bootstrap.servers": pipeline_to_redpanda_server,
"topic": "secops_vulnerability_stats_avro",
"headers": [{"key": "header1", "value": "this is a string"},
{"key": "header2", "value": list(b'byte array')}]
}
}
}
```
Signed-off-by: Leonid Ryzhyk <leonid@feldera.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added
headersfield to Kafka output connector config to specify a listof headers to be added to each message.
Kafka headers are a list of key-value pairs. Keys are strings. The
same key can occur multiple times, therefore the ordering is important.
The value component of the header is an optional byte array. For
convenience, we allow specifying it as an actual array of bytes in JSON
(
[1,2,3,4,5]) or as a UTF8-encoded string ("foobar"). Here is anexample Kafka connector config with headers that I added to the SecOps
demo:
Is this a user-visible change (yes/no): yes