[Client] Add warnings when user schedules many tasks with ray client #16454
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ckw017
changed the title
AmeerHajAli
reviewed
Jun 16, 2021
AmeerHajAli
reviewed
Jun 16, 2021
AmeerHajAli
reviewed
Jun 16, 2021
AmeerHajAli
reviewed
Jun 16, 2021
python/ray/util/client/worker.py
Outdated
Comment on lines
341
to
356
| logger.warning( | ||
| f"More than {TASK_WARNING_THRESHOLD} remote tasks have been " | ||
| "scheduled. This can be slow on Ray Client due to " | ||
| "communication overhead. If you're running many fine-grained " | ||
| "tasks consider batching them (details in the Ray Design " | ||
| "Pattern document).") | ||
| self.warning_raised = True | ||
| if not self.warning_raised and \ | ||
| self.total_outbound_message_size > MESSAGE_SIZE_THRESHOLD: | ||
| logger.warning( | ||
| "More than 10MB of messages have been created to schedule " | ||
| "tasks on the server. If you're running many fine-grained " | ||
| "tasks consider batching them (details in the Ray Design " | ||
| "Pattern document). If you have large arguments that are " | ||
| "frequently reused, consider storing them remotely with " | ||
| "ray.put or wrapping them in an actor object.") |
There was a problem hiding this comment.
@richardliaw , does this warning look right to you?
AmeerHajAli
reviewed
Jun 16, 2021
ijrsvt
reviewed
Jun 16, 2021
|
Robert suggested linking to the right section in the doc or some clear simple action steps for the user. |
AmeerHajAli
reviewed
Jun 16, 2021
richardliaw
reviewed
Jun 16, 2021
richardliaw
reviewed
Jun 16, 2021
richardliaw
reviewed
Jun 16, 2021
ijrsvt
reviewed
Jun 16, 2021
ijrsvt
reviewed
Jun 17, 2021
ijrsvt
reviewed
Jun 17, 2021
AmeerHajAli
reviewed
Jun 17, 2021
python/ray/util/client/worker.py
Outdated
| # Link to the Ray Design Pattern doc to use in the task overhead warning | ||
| # message | ||
| DESIGN_PATTERN_DOC_LINK = \ | ||
| "https://docs.google.com/document/d/167rnnDFIVRhHhK4mznEIemOtj63IOhtIPvSYaPgI4Fg/" # noqa E501 |
There was a problem hiding this comment.
can you please use this link (which goes directly to the subsection):
https://docs.google.com/document/d/167rnnDFIVRhHhK4mznEIemOtj63IOhtIPvSYaPgI4Fg/edit#heading=h.f7ins22n6nyl
AmeerHajAli
reviewed
Jun 17, 2021
AmeerHajAli
reviewed
Jun 17, 2021
python/ray/util/client/worker.py
Outdated
| "objects remotely with ray.put. An example of this is shown " | ||
| "in the \"Closure capture of large / unserializable object\" " | ||
| "section of the Ray Design Patterns document, available here: " | ||
| f"{DESIGN_PATTERN_DOC_LINK}", UserWarning) |
There was a problem hiding this comment.
you need another link here, which points to the exact subsection.
https://docs.google.com/document/d/167rnnDFIVRhHhK4mznEIemOtj63IOhtIPvSYaPgI4Fg/edit#heading=h.1afmymq455wu
AmeerHajAli
reviewed
Jun 17, 2021
AmeerHajAli
reviewed
Jun 17, 2021
AmeerHajAli
reviewed
Jun 17, 2021
AmeerHajAli
added
the
@author-action-required
AmeerHajAli
approved these changes
Jun 17, 2021
ckw017
removed
the
@author-action-required
Co-authored-by: Ameer Haj Ali <ameerh@berkeley.edu>
Co-authored-by: Ameer Haj Ali <ameerh@berkeley.edu>
ckw017
force-pushed
the
client_usage_warnings
branch
from
4b6cdbd
to
80fb2f0
Compare
|
Reviewers can merge at their discretion |
|
Thanks @ckw017 ! |
DmitriGekhtman
pushed a commit
that referenced
this pull request
Jun 21, 2021
…16454) * Add warnings when user schedules many tasks with ray client * add test_client_warnings to BUILD * better variable names * use util.debug.log_once() * batching -> explanation of batching * Switch to warnings.warn * Add links to Ray Design Pattern doc with code snippets * Cleaner linking and refer to sections directly * Better testNoWarning * add sys.exit(pytest.main(...)) * Update python/ray/util/client/worker.py Co-authored-by: Ameer Haj Ali <ameerh@berkeley.edu> * Update python/ray/util/client/worker.py Co-authored-by: Ameer Haj Ali <ameerh@berkeley.edu> * better error messages * Switch links to new readthedocs sections * Revert "Switch links to new readthedocs sections" This reverts commit d3785bf. Co-authored-by: Ameer Haj Ali <ameerh@berkeley.edu>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Using a lot of f.remote() calls with ray client can be extremely slow, so raises a one time warning can be useful to alert the user.
Raises a warning if the total number of tasks scheduled exceeds 1000 or the total size of ClientTask messages passed to the server exceeds 10MB since the beginning of the session. This warning will only be raised at most once per connection.
Related issue number
Closes #15009
Checks
scripts/format.shto lint the changes in this PR.