Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: kafka batching #3151

Merged
merged 10 commits into from
Apr 9, 2023
Merged

chore: kafka batching #3151

merged 10 commits into from
Apr 9, 2023

Conversation

fracasula
Copy link
Collaborator

@fracasula fracasula commented Mar 30, 2023

Description

Making sure that batching on Kafka works as expected.

Notion Ticket

< Notion Link >

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov
Copy link

codecov bot commented Mar 30, 2023

Codecov Report

Patch coverage: 74.60% and project coverage change: +0.06 🎉

Comparison is base (d7ac3a8) 52.28% compared to head (f1e3e94) 52.35%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3151      +/-   ##
==========================================
+ Coverage   52.28%   52.35%   +0.06%     
==========================================
  Files         321      321              
  Lines       52789    52809      +20     
==========================================
+ Hits        27602    27646      +44     
+ Misses      23527    23512      -15     
+ Partials     1660     1651       -9     
Impacted Files Coverage Δ
runner/runner.go 69.88% <42.10%> (ø)
services/streammanager/kafka/kafkamanager.go 76.81% <82.14%> (+0.06%) ⬆️
regulation-worker/cmd/main.go 71.95% <100.00%> (ø)
services/streammanager/kafka/client/producer.go 84.03% <100.00%> (+4.21%) ⬆️

... and 7 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@fracasula fracasula changed the title test: kafka batching chore: kafka batching Apr 4, 2023
@fracasula fracasula mentioned this pull request Apr 6, 2023
1 task
@@ -0,0 +1,478 @@
package kafka_batching
Copy link
Contributor

@atzoum atzoum Apr 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: does batching work in combination with AVRO Schema Lists and/or multiple topic support? This is not evident by looking here, where it seems like the process method is performing more steps than its batch counterpart. Maybe batch is called with events that have already been processed in a previous step?

@utsabc @koladilip

Copy link
Contributor

@atzoum atzoum Apr 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I can answer that question myself... events are first transformed at processor, then batched at router!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants