Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tune] Fix trial runner/controller whitelist attributes #35769

Merged
merged 8 commits into from
Jun 1, 2023

Conversation

krfricke
Copy link
Contributor

@krfricke krfricke commented May 25, 2023

Why are these changes needed?

With PBT and BOHB, we currently can get these warnings:

2023-05-24 04:03:59,962 WARNING trial_runner.py:1543 -- You are trying to access pause_trial interface of TrialRunner in TrialScheduler, which is being restricted. If you believe it is reasonable for your scheduler to access this TrialRunner API, please reach out to Ray team on GitHub. A more strict API access pattern would be enforced starting 1.12s.0

While technically we should have seen those warnings before (in the old execution path), it looks like they only come up ever since we activated the new execution path.

This PR adds the rest of those attributes we access in our tune-provided schedulers to the whitelist to get rid of the warnings.

Background

We introduced the warnings as preparation to refactor the scheduler interface. We wanted to discourage users to directly call trial runner/executor APIs. The plan was to redefine the interface and get rid of these calls in our own implementations. However, we have deprioritized this effort - that's why we keep a whitelist. We should revisit fixing the scheduler interface in the future.

Related issue number

Addresses part of #35640

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: Kai Fricke <kai@anyscale.com>
Copy link
Contributor

@justinvyu justinvyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have a few suggestions!

"has_resources_for_trial",
"pause_trial",
"save",
"resource_updater",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"resource_updater",
"_resource_updater",

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this one actually doesn't have a public counterpart.

python/ray/tune/execution/tune_controller.py Show resolved Hide resolved
python/ray/tune/execution/tune_controller.py Show resolved Hide resolved
python/ray/tune/execution/tune_controller.py Show resolved Hide resolved
Kai Fricke added 2 commits May 30, 2023 09:33
@krfricke krfricke requested a review from justinvyu May 30, 2023 08:33
Copy link
Contributor

@justinvyu justinvyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Just need to fix the resource_updater attribute, the hasattr check seems to be causing all the tests to fail.

"has_resources_for_trial",
"pause_trial",
"save",
"resource_updater",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this one actually doesn't have a public counterpart.

Kai Fricke added 2 commits May 31, 2023 21:18
Signed-off-by: Kai Fricke <kai@anyscale.com>
@krfricke krfricke merged commit 9c17d90 into ray-project:master Jun 1, 2023
@krfricke krfricke deleted the tune/whitelist-attr branch June 1, 2023 17:16
scv119 pushed a commit to scv119/ray that referenced this pull request Jun 16, 2023
…35769)

With PBT and BOHB, we currently can get these warnings:

```
2023-05-24 04:03:59,962 WARNING trial_runner.py:1543 -- You are trying to access pause_trial interface of TrialRunner in TrialScheduler, which is being restricted. If you believe it is reasonable for your scheduler to access this TrialRunner API, please reach out to Ray team on GitHub. A more strict API access pattern would be enforced starting 1.12s.0
```

While technically we should have seen those warnings before (in the old execution path), it looks like they only come up ever since we activated the new execution path.

This PR adds the rest of those attributes we access in our tune-provided schedulers to the whitelist to get rid of the warnings. 

Signed-off-by: Kai Fricke <kai@anyscale.com>
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
…35769)

With PBT and BOHB, we currently can get these warnings:

```
2023-05-24 04:03:59,962 WARNING trial_runner.py:1543 -- You are trying to access pause_trial interface of TrialRunner in TrialScheduler, which is being restricted. If you believe it is reasonable for your scheduler to access this TrialRunner API, please reach out to Ray team on GitHub. A more strict API access pattern would be enforced starting 1.12s.0
```

While technically we should have seen those warnings before (in the old execution path), it looks like they only come up ever since we activated the new execution path.

This PR adds the rest of those attributes we access in our tune-provided schedulers to the whitelist to get rid of the warnings.

Signed-off-by: Kai Fricke <kai@anyscale.com>
Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants