Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: batch event schema messages for faster processing #3406

Merged
merged 1 commit into from May 29, 2023

Conversation

atzoum
Copy link
Contributor

@atzoum atzoum commented May 26, 2023

Description

Related to this, rudder-server will now batch statistics for multiple rudder events in the same message so that schema workers can increase their event processing throughput even more.

Notion Ticket

Link

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@atzoum atzoum requested a review from cisse21 May 26, 2023 12:07
@atzoum atzoum marked this pull request as draft May 26, 2023 12:22
@codecov
Copy link

codecov bot commented May 26, 2023

Codecov Report

Patch coverage: 77.94% and project coverage change: +0.34 🎉

Comparison is base (4a94c15) 68.12% compared to head (5f8469e) 68.47%.

❗ Current head 5f8469e differs from pull request most recent head c7158d6. Consider uploading reports for the commit c7158d6 to get more accurate results

Additional details and impacted files
@@                Coverage Diff                @@
##           release/1.9.x    #3406      +/-   ##
=================================================
+ Coverage          68.12%   68.47%   +0.34%     
=================================================
  Files                329      330       +1     
  Lines              53492    52780     -712     
=================================================
- Hits               36443    36141     -302     
+ Misses             14719    14309     -410     
  Partials            2330     2330              
Impacted Files Coverage Δ
warehouse/admin.go 3.17% <0.00%> (ø)
warehouse/errors.go 89.47% <ø> (+3.75%) ⬆️
...ns/datalake/schema-repository/schema_repository.go 85.00% <ø> (ø)
warehouse/integrations/manager/manager.go 95.83% <ø> (ø)
warehouse/jobs/jobs.go 7.69% <0.00%> (ø)
warehouse/utils/utils.go 82.27% <ø> (ø)
warehouse/client/client.go 48.31% <33.33%> (ø)
warehouse/identities.go 19.40% <43.75%> (+2.55%) ⬆️
warehouse/integrations/datalake/datalake.go 60.00% <47.61%> (ø)
warehouse/warehouse.go 63.58% <55.17%> (+7.57%) ⬆️
... and 32 more

... and 10 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@atzoum atzoum force-pushed the chore.eschemaBatching branch 2 times, most recently from 7414469 to 79811f8 Compare May 29, 2023 05:59
@atzoum atzoum changed the base branch from master to release/1.9.x May 29, 2023 05:59
@atzoum atzoum changed the title [WIP] chore: batch event schema messages for faster processing chore: batch event schema messages for faster processing May 29, 2023
@atzoum atzoum marked this pull request as ready for review May 29, 2023 06:00
@atzoum atzoum merged commit e9f2174 into release/1.9.x May 29, 2023
28 checks passed
@atzoum atzoum deleted the chore.eschemaBatching branch June 16, 2023 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants