-
Notifications
You must be signed in to change notification settings - Fork 105
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace custom task cancellation with RQ 1.7 functionality #1040
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Replaced some custom code for task cancellation with RQ native functionality. |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,6 +4,8 @@ | |
|
||
from django.db import transaction | ||
from django.urls import reverse | ||
from rq.command import send_stop_job_command | ||
from rq.exceptions import InvalidJobOperation, NoSuchJobError | ||
from rq.job import Job, get_current_job | ||
from rq.worker import Worker | ||
|
||
|
@@ -12,7 +14,6 @@ | |
from pulpcore.constants import TASK_FINAL_STATES, TASK_STATES | ||
from pulpcore.exceptions import MissingResource | ||
from pulpcore.tasking import connection | ||
from pulpcore.tasking.constants import TASKING_CONSTANTS | ||
|
||
_logger = logging.getLogger(__name__) | ||
|
||
|
@@ -50,18 +51,19 @@ def cancel(task_id): | |
task_status.state = TASK_STATES.CANCELED | ||
task_status.save() | ||
|
||
if job.is_started: | ||
redis_conn.sadd(TASKING_CONSTANTS.KILL_KEY, job.get_id()) | ||
try: | ||
send_stop_job_command(redis_conn, job.get_id()) | ||
send_stop_job_command(redis_conn, resource_job.get_id()) | ||
except (InvalidJobOperation, NoSuchJobError): | ||
# We don't care if the job isn't currently running when we try to cancel | ||
pass | ||
|
||
if resource_job.is_started: | ||
redis_conn.sadd(TASKING_CONSTANTS.KILL_KEY, resource_job.get_id()) | ||
# A hack to ensure that we aren't deleting resources still being used by the workhorse | ||
time.sleep(0.5) | ||
|
||
resource_job.delete() | ||
job.delete() | ||
|
||
# A hack to ensure that we aren't deleting resources still being used by the workhorse | ||
time.sleep(1.5) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe we should keep this but reduce it to ~0.1 seconds? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah I'm not entirely sure what to do. I'm trying to think through it; so the resources delete while the task is in use, and it receives a database write error in the task at some point? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe that's ok? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There's an inherent delay in between when the cancellation is triggered and when it actually happens. Before that delay was due to polling, now it's message passing so it should be faster but technically still present. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The main thing is that we should avoid cancellations being marked as failures. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @bmbouter I'm thinking I should just err towards caution and add it back for now. |
||
|
||
with transaction.atomic(): | ||
for report in task_status.progress_reports.all(): | ||
if report.state not in TASK_FINAL_STATES: | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍