Move the tune driver into a remote task

[ericl]

Why are these changes needed?

This allows Tune to work efficiently (and work correct log dirs) when running in Ray client against a remote cluster.

TBD: whether this is on by default

[richardliaw]

This looks good to me; cc @krfricke and @amogkam

[krfricke]

Generally looks good. Some tests are failing for initialization reasons, this is probably a one line fix

[krfricke]

Should this be moved to the else block?

[krfricke]

Maybe throw a warning/exception if remote=True and a trial executor have been passed explicitly

[krfricke]

(feel free to ignore though since this is a private parameter for now)

[krfricke]

Actually, instead of using the _impl_ functions, can we do something like this?

def run(trainable,
        config=None,
        trial_executor=None,
        # ...
        _remote: Optional[bool] = None
):
    if _remote is True and trial_executor:
        raise ValueError("cannot use custom trial executor")
    elif _remote is None:
        _remote = ray.util.client.ray.is_connected()

    _ray_auto_init()

    if _remote:
        return ray.get(ray.remote(num_cpus=0)(run).remote(
            trainable,
            config=config,
            _remote=False))

    # Rest of the current tune.run code

This way we keep the functionality in the run() and run_experiments() definition, avoid argument duplication in those methods, and it might even be more readable. What do you think?

[ericl]

Great suggestion; updated.