-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[tune] Fix trial runner/controller whitelist attributes #35769
Conversation
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have a few suggestions!
"has_resources_for_trial", | ||
"pause_trial", | ||
"save", | ||
"resource_updater", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"resource_updater", | |
"_resource_updater", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
see above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this one actually doesn't have a public counterpart.
Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Just need to fix the resource_updater
attribute, the hasattr
check seems to be causing all the tests to fail.
"has_resources_for_trial", | ||
"pause_trial", | ||
"save", | ||
"resource_updater", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this one actually doesn't have a public counterpart.
Signed-off-by: Kai Fricke <kai@anyscale.com>
…35769) With PBT and BOHB, we currently can get these warnings: ``` 2023-05-24 04:03:59,962 WARNING trial_runner.py:1543 -- You are trying to access pause_trial interface of TrialRunner in TrialScheduler, which is being restricted. If you believe it is reasonable for your scheduler to access this TrialRunner API, please reach out to Ray team on GitHub. A more strict API access pattern would be enforced starting 1.12s.0 ``` While technically we should have seen those warnings before (in the old execution path), it looks like they only come up ever since we activated the new execution path. This PR adds the rest of those attributes we access in our tune-provided schedulers to the whitelist to get rid of the warnings. Signed-off-by: Kai Fricke <kai@anyscale.com>
…35769) With PBT and BOHB, we currently can get these warnings: ``` 2023-05-24 04:03:59,962 WARNING trial_runner.py:1543 -- You are trying to access pause_trial interface of TrialRunner in TrialScheduler, which is being restricted. If you believe it is reasonable for your scheduler to access this TrialRunner API, please reach out to Ray team on GitHub. A more strict API access pattern would be enforced starting 1.12s.0 ``` While technically we should have seen those warnings before (in the old execution path), it looks like they only come up ever since we activated the new execution path. This PR adds the rest of those attributes we access in our tune-provided schedulers to the whitelist to get rid of the warnings. Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Why are these changes needed?
With PBT and BOHB, we currently can get these warnings:
While technically we should have seen those warnings before (in the old execution path), it looks like they only come up ever since we activated the new execution path.
This PR adds the rest of those attributes we access in our tune-provided schedulers to the whitelist to get rid of the warnings.
Background
We introduced the warnings as preparation to refactor the scheduler interface. We wanted to discourage users to directly call trial runner/executor APIs. The plan was to redefine the interface and get rid of these calls in our own implementations. However, we have deprioritized this effort - that's why we keep a whitelist. We should revisit fixing the scheduler interface in the future.
Related issue number
Addresses part of #35640
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.