Support for Ad-hoc clusters

[yadudoc]

Ad-hoc clusters are a bunch of machines that can be reached via ssh but otherwise do not have a batch scheduler or container orchestrator. Our approach in general for these situations is to create an executor for each node, with the workers launched via an SSH channel. Creating separate executors means that load-balancing across nodes will not work. This sort of situation is common in clouds and private clusters and the driving use-case is from DESC (@jchiang87).

We could prototype an ad-hoc provider that takes a list channels pointing to each of the available nodes like this :

node_addresses = ['node1', 'node2', 'node3', ...]
config = Config(
    executors=[
        HighThroughputExecutor(
            label="Ad-Hoc",
            worker_debug=True,
            cores_per_worker=1,
            provider=AdHocProvider(
                channels=[SSHChannel(hostname=n) for n in node_addresses],
            ),
        )
    ],
    strategy=None,
)