Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logs support on Kafka receiver #2944

Merged
merged 4 commits into from
Apr 20, 2021

Conversation

sincejune
Copy link
Contributor

Description:
Add logs support on Kafka exporter

Link to tracking Issue:
Related to #1331
Related to open-telemetry/opentelemetry-collector-contrib#268
Related to #2943

Testing:

  • Added corresponding unit tests according to traces and metrics.
  • Tested locally with Kafka 2.11-2.0.0

Documentation:
Update Kafka receiver README and Receiver README.

@sincejune sincejune requested a review from a team as a code owner April 15, 2021 08:57
@codecov
Copy link

codecov bot commented Apr 15, 2021

Codecov Report

Merging #2944 (37e049c) into main (90b2dec) will increase coverage by 0.04%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2944      +/-   ##
==========================================
+ Coverage   91.58%   91.63%   +0.04%     
==========================================
  Files         312      312              
  Lines       15351    15441      +90     
==========================================
+ Hits        14059    14149      +90     
  Misses        883      883              
  Partials      409      409              
Impacted Files Coverage Δ
receiver/kafkareceiver/factory.go 100.00% <100.00%> (ø)
receiver/kafkareceiver/kafka_receiver.go 100.00% <100.00%> (ø)
receiver/kafkareceiver/otlp_unmarshaller.go 100.00% <100.00%> (ø)
receiver/kafkareceiver/unmarshaller.go 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90b2dec...37e049c. Read the comment docs.

@@ -27,6 +27,15 @@ type Unmarshaller interface {
Encoding() string
}

// logsUnmarshaller deserializes the message body.
type logsUnmarshaller interface {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is private means you don't expect users to implement? Also probably we should rename the previous one to TracesUnmarshaller.

Copy link
Contributor Author

@sincejune sincejune Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @bogdandrutu. Fixed in 0cb875f

And actually, there're many methods to be renamed (adding Traces). I would like to do this in another PR cuz it's already not a small one. Thoughts?

@@ -53,18 +53,30 @@ func WithAddUnmarshallers(encodingMarshaller map[string]Unmarshaller) FactoryOpt
}
}

// WithAddLogsUnmarshallers adds logsUnmarshallers.
func WithAddLogsUnmarshallers(encodingMarshaller map[string]logsUnmarshaller) FactoryOption {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Names are inconsistent, probably worth updating the previous one to include traces.

Also I would change the API to:

func WithLogsUnmarshallers(encoding string, unmarshaller logsUnmarshaller) FactoryOption {}

Or even consider to add a name func on the interface to simplify usage.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually there is an Encoding so we can do:

func WithLogsUnmarshallers(unmarshallers logsUnmarshaller...) FactoryOption {}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method is actually consistent with the traces' method here:

// WithAddUnmarshallers adds marshallers.
func WithAddUnmarshallers(encodingMarshaller map[string]Unmarshaller) FactoryOption {
return func(factory *kafkaReceiverFactory) {
for encoding, unmarshaller := range encodingMarshaller {
factory.unmarshalers[encoding] = unmarshaller
}
}
}

I'd like to keep the parameters consistent with trace's method and there are multiple encodings and return one factory in traces' method. But I'd like to rename the method name to be consistent in a subsequent PR. Thoughts?

@bogdandrutu bogdandrutu merged commit 0a2ea1b into open-telemetry:main Apr 20, 2021
@sincejune sincejune deleted the kafka-logs-receiver branch April 20, 2021 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants