Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[doc][train] Fix ray.train.report docstring to clarify that it is not a barrier #42422

Merged
merged 5 commits into from Jan 26, 2024

Conversation

justinvyu
Copy link
Contributor

Why are these changes needed?

ray.train.report gets called by all distributed training workers, but does not actually act as a synchronous barrier for the workers. Ray Train workers continue their training execution immediately after finishing ray.train.report.

Here's what happens:

  • The training coordinator waits on all futures to resolve, so it will be blocked if ones of the workers is a straggler.
  • The workers themselves will just put their results on a queue -- once it's on the queue, it will resolve one of the coordinator's futures.
  • The workers continue onto the next batch/epoch immediately. (The workers will typically run into another barrier via an all-reduce at some point.)

Users should add a barrier themselves if they need workers to be synchronized at the beginning or end of a report. See our lightning integration utility for one example: #40875

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
python/ray/train/_checkpoint.py Show resolved Hide resolved
python/ray/train/_internal/session.py Outdated Show resolved Hide resolved
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
@justinvyu justinvyu merged commit d9bd752 into ray-project:master Jan 26, 2024
9 checks passed
@justinvyu justinvyu deleted the fix_report_docstring branch January 26, 2024 02:43
khluu pushed a commit that referenced this pull request Jan 27, 2024
…ot a barrier (#42422)

`ray.train.report` gets called by all distributed training workers, but does not actually act as a synchronous barrier *for the workers.* Ray Train workers continue their training execution immediately after finishing `ray.train.report`. This PR fixes the docstring to correctly describe this behavior.

---------

Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: khluu <khluu000@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants