Skip to content

Improve performance of task claims by managers#852

Merged
bennybp merged 9 commits intomainfrom
claim_speed
Dec 11, 2024
Merged

Improve performance of task claims by managers#852
bennybp merged 9 commits intomainfrom
claim_speed

Conversation

@bennybp
Copy link
Contributor

@bennybp bennybp commented Oct 21, 2024

Description

We've noticed that with large databases, the time to claim tasks is a bit slow (multiple seconds). This is particularly noticeable in Issue #850 .

This PR aims to improve this significantly via two main ways:

  • Improving the SQL query. Not the task queue has an available column which can be used to filter, rather than the status column of the joined base_record table. This seems to be the fastest way, although it denormalizes stuff a bit
    • I tried a conditional WHERE status = 'waiting' index on base_record but that didn't even help as much
  • Improving the task specification generation (which happens sometimes when claiming) so that data is retrieved in fewer SQL queries. Also, I eliminated creation of pydantic objects here, which also trims a bit of time.

Tests on a copy of a production database show a reduction from ~2-3 seconds to 0.5-0.7 seconds for 200 tasks. Getting it much lower is going to be tough, since some of that is data transfer overhead over the internet.

Note that this does require a database migration with the upgrade-db subcommand.

Status

  • Code base linted
  • Ready to go

@bennybp
Copy link
Contributor Author

bennybp commented Dec 11, 2024

Not seeing any problems with this on my testing instance, so looks good to me

@bennybp bennybp merged commit 9375e1d into main Dec 11, 2024
@bennybp bennybp deleted the claim_speed branch December 11, 2024 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant