This copies the basic ssh tunneling from the Client to the Engine. The same semantics apply. In the controller, the ssh server is configured separately from the client ssh server, to prevent unnecessary tunneling from engines to the controller.
This is super useful, but I think it would be really good to have even a short paragraph illustrating this in the docs. Not all users are familiar with the details of ssh tunneling, so I'm sure a short illustrative example will do lots of good here.
Otherwise, the code looks solid (and the current test suite still passes) but I'm a little concerned that there's no test coverage at all, despite a fair amount of new functionality. I know that testing multiprocess things like this is super tricky, but even some light tests that at least do api validation will help us catch silly mistakes in the future. Obviously if you can think of some robust tests for it, that would be even better.
Beyond some docs and testing, it looks otherwise great for merging. Thanks for the excellent work, this will be very useful!
We simply can't test ssh tunneling except by hand. We can't depend on ssh being installed, or passwordless keys, or permissions, etc. Testing ssh in any meaningful way simply requires the use of multiple machines.
I'll toss up an example in the docs, though.
split open_tunnel part of tunnel_connection into separate method
This allows connection forwarding without establishing the final connection
(needed if the final connection is delayed, e.g. heartbeats)
add ssh tunneling to Engine
'enginessh' alias added to ipcontroller to new IPControllerApp.engine_ssh_server
ssh/keyfile added to ipengine/EngineFactory
remove now-obsolete note that engine's don't support ssh
add delay configurable to EngineSetLaunchers
add ssh tunnel notes to parallel process doc
simple engine ssh example added to parallel docs
specify sshkey is *private*