[batch/qob] Ctrl-c when interactively running a batch cancels the batch #12801
Conversation
| @@ -794,7 +800,7 @@ async def compile_job(job): | |||
| if verbose: | |||
| print(f'Waiting for batch {batch_handle.id}...') | |||
| starting_job_id = min(j._client_job.job_id for j in unsubmitted_jobs) | |||
| status = batch_handle.wait(disable_progress_bar=disable_progress_bar, starting_job=starting_job_id) | |||
| status = await batch_handle._async_batch.wait(disable_progress_bar=disable_progress_bar, starting_job=starting_job_id) | |||
There was a problem hiding this comment.
This change was made because it is both silly to nest two run_until_complete and doing so also causes the KeyboardInterrupt stacktrace to be printed twice. This fixes so the stacktrace is only printed once and you don't get complaints about a task not handling the exception.
hail/python/hailtop/batch/backend.py
Outdated
| if batch._batch_handle is not None: | ||
| print("Received a keyboard interrupt, cancelling the batch...") | ||
| batch._batch_handle.cancel() | ||
| raise |
There was a problem hiding this comment.
I tried to make this change a long time ago and got a bunch of push back.
cc: @konradjk ; If you ctrl-c a hailtop.batch script that is waiting on a batch (e.g. b.run(wait=True)), should the batch be cancelled?
There was a problem hiding this comment.
For now, let's just make the QoB change and revisit this after some discussion.
There was a problem hiding this comment.
There was a problem hiding this comment.
Interesting, I remember the argument but feel like these two behaviors are consistent with each other. Happy to remove the batch behavior for now though.
There was a problem hiding this comment.
I completely agree with you but, as the thread suggests, there was divergent opinions. I think people think of Batch as a submission sort of thing. IMHO, if you're trying to submit you should wait=False, but 🤷 . Let's take the win and fight this battle separately.
| @@ -443,8 +443,17 @@ async def _rpc(self, | |||
| raise reconstructed_error | |||
| raise reconstructed_error.maybe_user_error(ir) | |||
|
|
|||
| def _cancel_on_ctrl_c(self, coro): | |||
| try: | |||
| async_to_blocking(coro) | |||
CHANGELOG: Hitting CTRL-C while interactively using Batch or Query-on-Batch cancels the underlying batch.