[batch/qob] Ctrl-c when interactively running a batch cancels the batch - #12801
Conversation
| print(f'Waiting for batch {batch_handle.id}...') | ||
| starting_job_id = min(j._client_job.job_id for j in unsubmitted_jobs) | ||
| status = batch_handle.wait(disable_progress_bar=disable_progress_bar, starting_job=starting_job_id) | ||
| status = await batch_handle._async_batch.wait(disable_progress_bar=disable_progress_bar, starting_job=starting_job_id) |
There was a problem hiding this comment.
This change was made because it is both silly to nest two run_until_complete and doing so also causes the KeyboardInterrupt stacktrace to be printed twice. This fixes so the stacktrace is only printed once and you don't get complaints about a task not handling the exception.
| if batch._batch_handle is not None: | ||
| print("Received a keyboard interrupt, cancelling the batch...") | ||
| batch._batch_handle.cancel() | ||
| raise |
There was a problem hiding this comment.
I tried to make this change a long time ago and got a bunch of push back.
cc: @konradjk ; If you ctrl-c a hailtop.batch script that is waiting on a batch (e.g. b.run(wait=True)), should the batch be cancelled?
There was a problem hiding this comment.
For now, let's just make the QoB change and revisit this after some discussion.
There was a problem hiding this comment.
There was a problem hiding this comment.
Interesting, I remember the argument but feel like these two behaviors are consistent with each other. Happy to remove the batch behavior for now though.
There was a problem hiding this comment.
I completely agree with you but, as the thread suggests, there was divergent opinions. I think people think of Batch as a submission sort of thing. IMHO, if you're trying to submit you should wait=False, but 🤷 . Let's take the win and fight this battle separately.
|
|
||
| def _cancel_on_ctrl_c(self, coro): | ||
| try: | ||
| async_to_blocking(coro) |
2 participants
CHANGELOG: Hitting CTRL-C while interactively using Batch or Query-on-Batch cancels the underlying batch.