Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: drop union query, fair pickup and multitenant handle #3521

Merged
merged 1 commit into from
Jun 28, 2023

Conversation

atzoum
Copy link
Contributor

@atzoum atzoum commented Jun 20, 2023

Description

Since router isolation feature is now in place, there is no longer the need for the fair pickup algorithm, i.e. MultiTenantJobsDB and multitenant.Stats. Therefore we are removing them with the following adaptations:

  • MultiTenantJobsDB#GetAllJobs has been moved to JobsDB#GetToProcess and method's signature has been adapted accordingly. A future improvement shall follow which will adapt the method's current implementation and perform a single query instead of 3 separate ones.
  • processor is now responsible for retrieving job pileup counters and populating the pending events stats during startup, what was previously done by multitenant.Stats

Notion Ticket

Link

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov
Copy link

codecov bot commented Jun 20, 2023

Codecov Report

Patch coverage: 80.83% and project coverage change: +0.02 🎉

Comparison is base (0dbb328) 67.79% compared to head (13b84cc) 67.81%.

❗ Current head 13b84cc differs from pull request most recent head fd0963d. Consider uploading reports for the commit fd0963d to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3521      +/-   ##
==========================================
+ Coverage   67.79%   67.81%   +0.02%     
==========================================
  Files         325      327       +2     
  Lines       52627    53100     +473     
==========================================
+ Hits        35678    36012     +334     
- Misses      14603    14707     +104     
- Partials     2346     2381      +35     
Impacted Files Coverage Δ
app/apphandlers/processorAppHandler.go 9.77% <0.00%> (+0.76%) ⬆️
app/cluster/dynamic.go 55.49% <ø> (+0.43%) ⬆️
router/batchrouter/factory.go 100.00% <ø> (ø)
router/batchrouter/handle.go 63.39% <ø> (+0.18%) ⬆️
router/batchrouter/handle_lifecycle.go 75.09% <ø> (+2.53%) ⬆️
router/factory.go 100.00% <ø> (ø)
router/worker.go 82.42% <ø> (+0.06%) ⬆️
runner/runner.go 69.63% <ø> (-0.13%) ⬇️
utils/misc/misc.go 53.07% <ø> (-0.70%) ⬇️
processor/processor.go 87.60% <72.97%> (-0.31%) ⬇️
... and 8 more

... and 58 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@atzoum atzoum changed the title [WIP] chore: drop union query fair pickup and multitenant handle [DO NOT MERGE] chore: drop union query fair pickup and multitenant handle Jun 21, 2023
@atzoum atzoum marked this pull request as ready for review June 21, 2023 07:27
unprocessedAfterJobID := unprocessed.Jobs[len(unprocessed.Jobs)-1].JobID
mtoken.unprocessedAfterJobID = &unprocessedAfterJobID
}
list = append(list, unprocessed.Jobs...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[question] does it make sense to move this in the previous if block? what purpose does it serve if len(unprocessed.Jobs) == 0?

waitingAfterJobID := waiting.Jobs[len(waiting.Jobs)-1].JobID
mtoken.waitingAfterJobID = &waitingAfterJobID
}
list = append(list, waiting.Jobs...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[question] does it make sense to move this in the previous if block? what purpose does it serve if len(waiting.Jobs) == 0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if waiting.Jobs is empty this is a no-op, so it doesn't hurt either :)

router/router_test.go Outdated Show resolved Hide resolved
router/router_test.go Outdated Show resolved Hide resolved
@atzoum atzoum force-pushed the chore.dropUnionQuery branch 5 times, most recently from a0c9e28 to b30f881 Compare June 27, 2023 09:51
@atzoum atzoum changed the title [DO NOT MERGE] chore: drop union query fair pickup and multitenant handle chore: drop union query fair pickup and multitenant handle Jun 27, 2023
@atzoum atzoum changed the title chore: drop union query fair pickup and multitenant handle chore: drop union query, fair pickup and multitenant handle Jun 27, 2023
@atzoum atzoum merged commit 573d6ff into master Jun 28, 2023
34 checks passed
@atzoum atzoum deleted the chore.dropUnionQuery branch June 28, 2023 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants