[tune] TensorFlow Distributed Trainable

[oliverhu]

Why are these changes needed?

Follow on the discussion on https://github.com/ray-project/ray/discussions/11111. Tune currently supports distributed hyperparameter search with distributed training for (pytorch, horovod). We should also support Tensorflow default distributed training strategies including MultiWorkerMirrorStrategy and Parameter Serving Strategy.

Related issue number

https://github.com/ray-project/ray/discussions/11111

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[oliverhu]

Sample run output: https://gist.github.com/oliverhu/dca9f4d722e0f516ac8aba9701c59eee

[richardliaw]

Thanks a bunch for this PR!

Can you be sure to add the following:

• Run the test in the test suite similar to this (https://github.com/ray-project/ray/pull/9739/files#diff-5a9919a777b978fee388f8d0477558ecb309ade3bdb68f0cb400a13be873cddeR290)
• Add documentation (like this https://github.com/ray-project/ray/pull/9739/files#diff-e28682a144e210b167be5d875472c84ee9a69add6036a6ccf966394949115d75R288)
• Add the example to the docs like (https://github.com/ray-project/ray/pull/9739/files#diff-c6b9448a9f2448ee8cb89d555bd600092ee8305b6e4d267dc9290d48201fe3ccR44)

[oliverhu]

@richardliaw thanks for detailed comments.. updated