Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(jobsdb): unify get processed and unprocessed queries #3742

Merged
merged 1 commit into from
Aug 21, 2023

Conversation

atzoum
Copy link
Contributor

@atzoum atzoum commented Aug 14, 2023

Description

By unifying getUnprocessed and getProcessed under a single getJobs method which is able to handle both processed and unprocessed job states with a single database query, we can execute less queries, which translates to lower overall query times and finally less jobsdb locks:

  • router now executes a single GetToProcess query instead of 3 (GetToRetry, GetWaiting, GetUnprocessed)
  • batchrouter now executes a single GetToProcess query instead of 2 (GetToRetry, GetUnprocessed)
  • stash now executes a single GetToProcess query instead of 2 (GetToRetry, GetUnprocessed)

Other changes:

  • Rename jobsdb.GetQueryParamsT to jobsdb.GetQueryParams
  • Rename jobsdb.HandleT to jobsdb.Handle
  • Using generics for achieving type safe dbRequests

Monitoring dashboard adaptations

After this change has been rolled out, we need to unify the following JobsDB panels in Grafana under a single Get Jobs Stats panel:

  • Processed Fetch Stats
  • Unprocessed Fetch Stats Time

Linear Ticket

Link

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@codecov
Copy link

codecov bot commented Aug 14, 2023

Codecov Report

Patch coverage: 84.63% and project coverage change: -0.27% ⚠️

Comparison is base (188b95c) 68.80% compared to head (d80f4ce) 68.54%.
Report is 1 commits behind head on master.

❗ Current head d80f4ce differs from pull request most recent head db33a50. Consider uploading reports for the commit db33a50 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3742      +/-   ##
==========================================
- Coverage   68.80%   68.54%   -0.27%     
==========================================
  Files         344      344              
  Lines       51936    51259     -677     
==========================================
- Hits        35736    35136     -600     
+ Misses      13896    13854      -42     
+ Partials     2304     2269      -35     
Files Changed Coverage Δ
app/apphandlers/embeddedAppHandler.go 75.95% <0.00%> (-0.73%) ⬇️
app/apphandlers/processorAppHandler.go 9.32% <0.00%> (+0.76%) ⬆️
enterprise/replay/dumpsloader.go 0.00% <0.00%> (ø)
enterprise/replay/replay.go 0.00% <0.00%> (ø)
enterprise/replay/setup.go 0.00% <0.00%> (ø)
processor/manager.go 90.00% <ø> (-0.17%) ⬇️
router/batchrouter/handle_async.go 0.00% <0.00%> (ø)
router/batchrouter/handle.go 62.91% <83.33%> (-0.30%) ⬇️
jobsdb/jobsdb.go 72.50% <89.63%> (+0.03%) ⬆️
jobsdb/backup.go 69.25% <91.66%> (-0.34%) ⬇️
... and 13 more

... and 38 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@atzoum atzoum force-pushed the chore.jobsdbSingleQuery branch 3 times, most recently from 9a6aee5 to 1243a54 Compare August 16, 2023 12:24
@atzoum atzoum changed the title [WIP] chore(jobsdb): unify get processed and unprocessed queries chore(jobsdb): unify get processed and unprocessed queries Aug 17, 2023
@atzoum atzoum force-pushed the chore.jobsdbSingleQuery branch 4 times, most recently from 2d84675 to 6c4f7c1 Compare August 17, 2023 08:44
@atzoum atzoum marked this pull request as ready for review August 17, 2023 12:45
@atzoum atzoum force-pushed the chore.jobsdbSingleQuery branch 3 times, most recently from 48f4306 to 57ae64d Compare August 21, 2023 05:55
@cisse21 cisse21 merged commit 66c7e2e into master Aug 21, 2023
31 checks passed
@cisse21 cisse21 deleted the chore.jobsdbSingleQuery branch August 21, 2023 09:14
@atzoum atzoum restored the chore.jobsdbSingleQuery branch August 21, 2023 09:15
@atzoum atzoum deleted the chore.jobsdbSingleQuery branch August 21, 2023 09:15
@atzoum atzoum restored the chore.jobsdbSingleQuery branch August 21, 2023 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants