[engine] strategy for interrupting engine execution to support user-driven interrupts

[kwlzn]

in order to support a saner fix for #5220, it seems like we'll want to have some way to gracefully interrupt an in-flight engine execution that can be called from a python signal handler in the daemon.

[kwlzn]

mod any potential concerns wrt partial request computation, on the surface this seems like approximately:

1. switch from Future.wait() to Future.poll() in Scheduler::execute and check for some atomic flag/kill switch to short circuit (and drop the future to cancel the work?)
2. plumb an api to set the kill switch
3. call that from the signal handler in the daemon

[stuhood]

I think this sounds reasonable. One adjustment would be necessary:

Currently the main thread is the one actually doing the work by calling wait. Because rust futures are (mostly) "pull based", if we switched from wait to poll there, no thread would actually be working on the future. So I think that we'd additionally need to move the root futures into a CpuPool (or some other Future executor). Then those threads would drive them to completion, and rather than a future directly held by the Graph (a Shared future), Scheduler::execute would be holding a CpuFuture. Dropping it would probably cancel the work on the pool ... but this isn't precisely clear, because CpuPool might only be able to cancel before the work has started.

[stuhood]

Related: rust-lang/futures-rs#696

[stuhood]

An update: it is no longer the case that the main thread is the one doing the work here. We've switched to launching all work on the tokio runtime, and so it should be the case that the main thread is able to wait in a loop and die for signals.

[stuhood]

A first potential commit in this direction is here: master...twitter:stuhood/dont-block-the-main-thread

Will look into @kwlzn's suggestions about the kill switch on Monday.

[stuhood]

This has been fixed since we began respecting signals in the UI loop a while back.