[Ray SGD] Support num_steps continue training

[amogkam]

Currently, when training or validating with num_steps parameter set, each call will reset the data loaders from the beginning. This PR makes sure that subsequent training or validation calls consumes the entire data loader before resetting it. Also provides functionality to automatically cycle around the data loader if we still need to do more steps of training or validation.

Why are these changes needed?

Related issue number

Closes #8907

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[richardliaw]

looks good, just one comment about test.

Does this all get reset when there are changes in worker pool?

[amogkam]

Yes that's a good point. This does get reset whenever the worker pool size changes. I'll raise an issue for this, but I think it requires some more thought on how to handle this.

[amogkam]

@richardliaw finally...tests are passing