Cluster Python API

[minrk]

Cluster is a Python API for starting/stopping/signaling controllers and engines. It is basically rewriting IPClusterApp as a simpler object with a reusable API. It's a fully asyncio API natively, with automatic _sync methods that provide synchronous versions of everything that should work both inside and outside asyncio.

TODO:

• create cluster subpackage
• Cluster can start/stop/signal engines, connect clients, be used as a context manager, etc.
• basic tests for Cluster
• simplify ipcluster cli to build on Cluster API
• (more) Cluster tests
• set up mpi on CI (probably means conda)
• ClusterManager for multiple Clusters
• serialize cluster to/from dict for e.g. ipcluster stop/list (Serialize Cluster objects #480)
• work out signature / options, etc.
• work out distribution of connection info (ControllerLaunchers should return connection info #483)
• signaling single engines (Add registration information to launchers #475)

closes #22
closes #216
closes #243
closes #241
closes #460
closes #94

[minrk]

@sahil1105 it's not quite ready, but the basics are here if you want to start reviewing. In particular, you can check out the example notebook walking through:

• starting/stopping clusters
• using them as context managers
• starting engines with MPI
• interrupting and restarting engines

[minrk]

@sahil1105 I think this is reasonably complete for a first draft. I'm still debugging little discrepancies between my environment and CI, but everything's working for me locally

[minrk]

@sahil1105 I think this is about ready to go for a first draft if you'd like to review. It's working and relatively complete. I've opened issues for the follow-up tasks to do after landing this.

I'm still ironing out the kinks of testing MPI on CI (everything passes for me locally)

[minrk]

All tests are passing now

[sahil1105]

@sahil1105 it's not quite ready, but the basics are here if you want to start reviewing. In particular, you can check out the example notebook walking through:

• starting/stopping clusters
• using them as context managers
• starting engines with MPI
• interrupting and restarting engines

Will do. Thanks!

[sahil1105]

@minrk Sorry about the delay, looks great to me, thanks!
Couple of questions:

1. Is there a way/API to send signal tompiexec itself if needed? I know we're handling SIGINT, etc. using a signal handler on each process without involving mpiexec which makes sense, but we might for instance send a SIGUSER to mpiexec which the application could receive and handle.
2. When we restart NB kernel (w/o calling stop_cluster()), are all controller/engine resources cleaned up or left dangling (and if so what's the best way to clean them)?
3. For certain implementation, like MPI, when the engines are started using a Launcher, do we need heartbeats between engine and controller? Also, if the mpiexec process dies while running user code, will the user (client) get notified immediately (since the launcher has a handle on the mpiexec process), or will it still need to wait until enough heartbeats have missed?

[minrk]

Is there a way/API to send signal to mpiexec itself if needed?

Yes, signal_engines does exactly this. I still think we should implement sending signals to engines directly for a host of reasons (mpiexec's own poor signal handling among them). That would become a client method instead of a cluster method, though, so we wouldn't lose the ability to signal mpiexec from the Launcher. That approach is not yet implemented, though (#475).

When we restart NB kernel (w/o calling stop_cluster()), are all controller/engine resources cleaned up or left dangling (and if so what's the best way to clean them)?

They should be, but it cannot be assumed because cleanup code cannot be guaranteed to run (e.g. killed with SIGKILL, no cleanup events can be assumed to have run). I can work on cleaning up as often as possible (e.g. with atexit, which I started working on already, but haven't pushed it yet). Ultimately, we need #480 to interact with clusters not started by the current process. Actions would include:

• listing running clusters
• connecting to a cluster started previously
• shutting down clusters not started by this process

For certain implementation, like MPI, when the engines are started using a Launcher, do we need heartbeats between engine and controller?

That's a tricky one. If a launcher can be trusted to be notified, yes. mpiexec is not such a launcher in general, because a process can begin exiting at which point it becomes unresponsive and should be considered 'dead'. The hearbeat mechanism should catch this, but mpiexec will not because the process still exists and/or has begun 'cleanly' shutting down via MPI_Finalize. mpiexec only notifies about "abnormal" exit, i.e. exit without finalize.

So relying on mpiexec can mean engines are less likely to be notified of shutdown than the current mechanism.

We can, however, at least send the same notification about engine shutdown in all the cases where mpiexec does exit, which should have most of the benefit–improved responsiveness–without the cost of removing it–missed shutdown events. I've opened #491 to track this.

[sahil1105]

Is there a way/API to send signal to mpiexec itself if needed?

Yes, signal_engines does exactly this. I still think we should implement sending signals to engines directly for a host of reasons (mpiexec's own poor signal handling among them). That would become a client method instead of a cluster method, though, so we wouldn't lose the ability to signal mpiexec from the Launcher. That approach is not yet implemented, though (#475).

Oh ok, makes sense. Thanks.

When we restart NB kernel (w/o calling stop_cluster()), are all controller/engine resources cleaned up or left dangling (and if so what's the best way to clean them)?

They should be, but it cannot be assumed because cleanup code cannot be guaranteed to run (e.g. killed with SIGKILL, no cleanup events can be assumed to have run). I can work on cleaning up as often as possible (e.g. with atexit, which I started working on already, but haven't pushed it yet). Ultimately, we need #480 to interact with clusters not started by the current process. Actions would include:

• listing running clusters
• connecting to a cluster started previously
• shutting down clusters not started by this process

Got it, makes sense. Yes, I did notice that sometimes there are lingering processes sometimes, but a simple bash script to terminate all ipcontroller processes could take care of that for now.

For certain implementation, like MPI, when the engines are started using a Launcher, do we need heartbeats between engine and controller?

That's a tricky one. If a launcher can be trusted to be notified, yes. mpiexec is not such a launcher in general, because a process can begin exiting at which point it becomes unresponsive and should be considered 'dead'. The hearbeat mechanism should catch this, but mpiexec will not because the process still exists and/or has begun 'cleanly' shutting down via MPI_Finalize. mpiexec only notifies about "abnormal" exit, i.e. exit without finalize.

Ah ok, understood.
Yes, I was interested in the abnormal exit case where one (or more) processes exit and MPI closes all of them and exits with code 6 or code 9 or something.

So relying on mpiexec can mean engines are less likely to be notified of shutdown than the current mechanism.

We can, however, at least send the same notification about engine shutdown in all the cases where mpiexec does exit, which should have most of the benefit–improved responsiveness–without the cost of removing it–missed shutdown events. I've opened #491 to track this.

Agreed. Sounds good, thanks!

[minrk]

Thanks for the review, @sahil1105!