Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add kafka topic/partition/offset to the extension of event #896

Merged
merged 7 commits into from
May 25, 2023
Merged

add kafka topic/partition/offset to the extension of event #896

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2a574e8
add kafka partition/offset to the extension of event
yanmxa May 12, 2023
455fd8f
initialize header with kafka information
yanmxa May 14, 2023
e033080
fixed the integration test
yanmxa May 16, 2023
1ab33b9
fixed the sarama binding test issue
yanmxa May 17, 2023
22ddc6c
add check empty in the integration test
yanmxa May 23, 2023
cfac14f
verfiy the kafka topic value
yanmxa May 24, 2023
998a3a0
clarify the test code
yanmxa May 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 9 additions & 3 deletions protocol/kafka_sarama/v2/message.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ package kafka_sarama
import (
"bytes"
"context"
"strconv"
"strings"

"github.com/cloudevents/sdk-go/v2/binding"
Expand Down Expand Up @@ -35,22 +36,27 @@ type Message struct {
}

// Check if http.Message implements binding.Message
var _ binding.Message = (*Message)(nil)
var _ binding.MessageMetadataReader = (*Message)(nil)
var (
_ binding.Message = (*Message)(nil)
_ binding.MessageMetadataReader = (*Message)(nil)
)

// NewMessageFromConsumerMessage returns a binding.Message that holds the provided ConsumerMessage.
// The returned binding.Message *can* be read several times safely
// This function *doesn't* guarantee that the returned binding.Message is always a kafka_sarama.Message instance
func NewMessageFromConsumerMessage(cm *sarama.ConsumerMessage) *Message {
var contentType string
headers := make(map[string][]byte, len(cm.Headers))
headers := make(map[string][]byte, len(cm.Headers)+3)
for _, r := range cm.Headers {
k := strings.ToLower(string(r.Key))
if k == contentTypeHeader {
contentType = string(r.Value)
}
headers[k] = r.Value
}
headers[prefix+"kafkaoffset"] = []byte(strconv.FormatInt(cm.Offset, 10))
headers[prefix+"kafkapartition"] = []byte(strconv.FormatInt(int64(cm.Partition), 10))
headers[prefix+"kafkatopic"] = []byte(cm.Topic)
return NewMessage(cm.Value, contentType, headers)
}

Expand Down
11 changes: 9 additions & 2 deletions test/integration/kafka_sarama/kafka_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,10 @@ import (
)

const (
TEST_GROUP_ID = "test_group_id"
TEST_GROUP_ID = "test_group_id"
KAFKA_OFFSET = "kafkaoffset"
KAFKA_PARTITION = "kafkapartition"
KAFKA_TOPIC = "kafkatopic"
)

func TestSendEvent(t *testing.T) {
Expand All @@ -30,7 +33,11 @@ func TestSendEvent(t *testing.T) {
clienttest.SendReceive(t, func() interface{} {
return protocolFactory(t)
}, eventIn, func(e event.Event) {
test.AssertEventEquals(t, eventIn, test.ConvertEventExtensionsToString(t, e))
got := test.ConvertEventExtensionsToString(t, e)
eventIn.SetExtension(KAFKA_OFFSET, got.Extensions()[KAFKA_OFFSET])
eventIn.SetExtension(KAFKA_PARTITION, got.Extensions()[KAFKA_PARTITION])
eventIn.SetExtension(KAFKA_TOPIC, got.Extensions()[KAFKA_TOPIC])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't looked deeper at the different test.<> functions, but was wondering what this is supposed to do here? You're setting extensions on the input event based on got and then asserting. Is this correctly exercising and asserting the changes here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is another way we can remove the KAFKA_OFFSET , KAFKA_PARTITION, and KAFKA_TOPIC from got. Is this way better?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious, how do we remove extension attributes from a CE? I didn't see that option in here: https://github.com/cloudevents/sdk-go/blob/main/v2/event/event_writer.go

I'm ok with copying things from got into eventIn so that the Assert works (or removing them from got if that's possible), but it seems to me that there should be a check of the values in got to make sure they're not blank - even if we need to do those manually instead via a compare with eventIn.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@duglin Thanks for your advice! I have added the empty check for the event extensions.

test.AssertEventEquals(t, eventIn, got)
})
})
}
Expand Down
11 changes: 9 additions & 2 deletions test/integration/kafka_sarama_binding/kafka_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,10 @@ import (
)

const (
TEST_GROUP_ID = "test_group_id"
TEST_GROUP_ID = "test_group_id"
KAFKA_OFFSET = "kafkaoffset"
KAFKA_PARTITION = "kafkapartition"
KAFKA_TOPIC = "kafkatopic"
)

func TestSendStructuredMessageToStructured(t *testing.T) {
Expand Down Expand Up @@ -55,7 +58,11 @@ func TestSendBinaryMessageToBinary(t *testing.T) {
test.SendReceive(t, binding.WithPreferredEventEncoding(context.TODO(), binding.EncodingBinary), in, s, r, func(out binding.Message) {
eventOut := MustToEvent(t, context.Background(), out)
assert.Equal(t, binding.EncodingBinary, out.ReadEncoding())
AssertEventEquals(t, eventIn, ConvertEventExtensionsToString(t, eventOut))
got := ConvertEventExtensionsToString(t, eventOut)
eventIn.SetExtension(KAFKA_OFFSET, got.Extensions()[KAFKA_OFFSET])
eventIn.SetExtension(KAFKA_PARTITION, got.Extensions()[KAFKA_PARTITION])
eventIn.SetExtension(KAFKA_TOPIC, got.Extensions()[KAFKA_TOPIC])
AssertEventEquals(t, eventIn, got)
})
})
}
Expand Down