feat: Adding support to propagate unique ID to message for Kafka EventSource messages #1453
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Dillen Padhiar <dpadhiar@ucsd.edu>
sarabala1979
reviewed
Dec 8, 2021
whynowy
reviewed
Dec 9, 2021
eventsources/common/common.go
Outdated
| type Options func(*event.Event) error | ||
|
|
||
| // Option to set different ID for event | ||
| func SetCustomID(id string) Options { |
There was a problem hiding this comment.
A better function name would be WithID(id string).
eventsources/sources/kafka/start.go
Outdated
|
|
||
| // Function can be passed as Option to generate unique id for kafka event | ||
| // eventSourceName:eventName:kafka-url:topic:partition:offset | ||
| func genUniqueID(eventSourceName string, eventName string, kafkaURL string, topic string, partition int32, offset int64) string { |
There was a problem hiding this comment.
Minor - func genUniqueID(eventSourceName, eventName, kafkaURL, topic string, partition int32, offset int64).
eventsources/sources/kafka/start.go
Outdated
| // eventSourceName:eventName:kafka-url:topic:partition:offset | ||
| func genUniqueID(eventSourceName string, eventName string, kafkaURL string, topic string, partition int32, offset int64) string { | ||
|
|
||
| kafkaID := fmt.Sprintf("%s:%s:%s:%s:%d:%d", eventSourceName, eventName, kafkaURL, topic, partition, offset) |
There was a problem hiding this comment.
kafkaURL could be very long, which contains several URLs separated by comma. I would suggest to split the URL to only take the first one.
eventsources/sources/kafka/start.go
Outdated
| @@ -375,9 +380,20 @@ func (consumer *Consumer) processOne(session sarama.ConsumerGroupSession, messag | |||
| return errors.Wrap(err, "failed to marshal the event data, rejecting the event...") | |||
| } | |||
|
|
|||
| if err = consumer.dispatch(eventBody); err != nil { | |||
| kafkaID := genUniqueID(consumer.eventSourceName, consumer.eventName, consumer.kafkaEventSource.URL, message.Topic, message.Partition, message.Offset) | |||
There was a problem hiding this comment.
Name it messageID or kafkaMessageID, kafkaID is very tricky.
Signed-off-by: Dillen Padhiar <dpadhiar@ucsd.edu>
whynowy
approved these changes
Dec 9, 2021
5 tasks
BulkBeing
pushed a commit
to BulkBeing/argo-events
that referenced
this pull request
Mar 7, 2022
juliev0
pushed a commit
to juliev0/argo-events
that referenced
this pull request
Mar 29, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist:
closes: #1427
Logs from kafka-eventsource pod: