feat(similarity): Add delete record by hash task #72767
Conversation
github-actions
bot
added
the
Scope: Backend
The task is registered here. and yes, I've made it an option, good idea - thanks! |
| len_hashes = len(hashes) | ||
| end_index = min(last_deleted_index + batch_size, len_hashes) | ||
| delete_grouping_records_by_hash(project_id, hashes[last_deleted_index:end_index]) | ||
| if end_index < len_hashes: |
There was a problem hiding this comment.
any reason to call the task again instead of just calling the delete_grouping_records_by_hash function repetitively?
There was a problem hiding this comment.
it's based on the logic we used for the backfill - I didn't want this task to timeout, so I'm respawning the task instead
There was a problem hiding this comment.
what is our current timeout set to on the celery task?
There was a problem hiding this comment.
haha whoops just realized I didn't even add a timeout, but I was thinking of using the same timeouts from the backfill soft_time_limit=60 * 15, time_limit=60 * (15 + 5)
There was a problem hiding this comment.
got it. i'd consider thinking about increasing the batch size to 100, and optionally doing some calls in a loop, so that way we have to spawn less celery tasks, as each one should run pretty quick (less than a second)
There was a problem hiding this comment.
nice, I didn't realize they were that quick - thanks!
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #72767 +/- ##
=========================================
Coverage ? 78.08%
=========================================
Files ? 6620
Lines ? 295396
Branches ? 50892
=========================================
Hits ? 230671
Misses ? 58461
Partials ? 6264
|
Add the task that calls the seer delete record by hash endpoint
Given a list of hashes, batch by size 20