-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(similarity): Add delete record by hash task #72767
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need to register the celery task? and could we make batch size an option? i have a feeling we can probably go pretty high with it, into the thousands
The task is registered here. and yes, I've made it an option, good idea - thanks! |
len_hashes = len(hashes) | ||
end_index = min(last_deleted_index + batch_size, len_hashes) | ||
delete_grouping_records_by_hash(project_id, hashes[last_deleted_index:end_index]) | ||
if end_index < len_hashes: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any reason to call the task again instead of just calling the delete_grouping_records_by_hash
function repetitively?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's based on the logic we used for the backfill - I didn't want this task to timeout, so I'm respawning the task instead
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is our current timeout set to on the celery task?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
haha whoops just realized I didn't even add a timeout, but I was thinking of using the same timeouts from the backfill soft_time_limit=60 * 15
, time_limit=60 * (15 + 5)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
got it. i'd consider thinking about increasing the batch size to 100, and optionally doing some calls in a loop, so that way we have to spawn less celery tasks, as each one should run pretty quick (less than a second)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice, I didn't realize they were that quick - thanks!
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #72767 +/- ##
=========================================
Coverage ? 78.08%
=========================================
Files ? 6620
Lines ? 295396
Branches ? 50892
=========================================
Hits ? 230671
Misses ? 58461
Partials ? 6264
|
Add the task that calls the seer delete record by hash endpoint
Given a list of hashes, batch by size 20