Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] Allow SAC to use custom models as Q- or policy nets and deprecate "state-preprocessor" for image spaces. #13522

Merged
merged 26 commits into from
Feb 2, 2021
Merged

[RLlib] Allow SAC to use custom models as Q- or policy nets and deprecate "state-preprocessor" for image spaces. #13522

merged 26 commits into from
Feb 2, 2021

Conversation

sven1977
Copy link
Contributor

@sven1977 sven1977 commented Jan 18, 2021 •
edited

SAC's model is currently hard to customize due to the following reasons:

  • The "use_state_preprocessor" is confusing as it must be set manually for e,g. image observation spaces.
  • Such a pre-processor (e.g. a CNN stack) currently does not learn as a bug in SAC's code prevents the weights of the pre-processor to be updated by any optimizer.
  • Using custom sub-modules for Q-nets or policy-nets right now is hard, unless you override the entire SACModel class.
  • Running SAC in mixed observation spaces (e.g. Tuple[img, discrete(2), Box((1,))]) is not possible.

This PR fixes all these limitations and bugs.

Why are these changes needed?

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

michaelzhiluo
michaelzhiluo requested changes Jan 22, 2021
View reviewed changes
Copy link
Contributor

@michaelzhiluo michaelzhiluo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Placeholder

@sven1977 sven1977 changed the title [RLlib] Fix SAC not learning weights of "state-preprocessor" (e.g. CNN in front of Q- and policy nets). [RLlib] Allow SAC to use custom models as Q- or policy nets and deprecate "state-preprocessor" for image spaces. Jan 25, 2021
@sven1977 sven1977 merged commit 52c94b7 into ray-project:master Feb 2, 2021
@sven1977 sven1977 mentioned this pull request Feb 10, 2021
Closed
fishbone pushed a commit to fishbone/ray that referenced this pull request Feb 16, 2021
@sven1977 @fishbone
[RLlib] Allow SAC to use custom models as Q- or policy nets and depre…
10c31e8 
…cate "state-preprocessor" for image spaces. (ray-project#13522)
fishbone added a commit to fishbone/ray that referenced this pull request Feb 16, 2021
@fishbone
Revert "[RLlib] Allow SAC to use custom models as Q- or policy nets a…
0db8ec3 
…nd deprecate "state-preprocessor" for image spaces. (ray-project#13522)"

This reverts commit 10c31e8.
@MischaPanch MischaPanch mentioned this pull request Aug 14, 2021
Closed
@sven1977 sven1977 deleted the sac_fix_preprocessor_not_learning_2 branch June 2, 2023 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michaelzhiluo michaelzhiluo michaelzhiluo approved these changes

Assignees

@michaelzhiluo michaelzhiluo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@sven1977 @michaelzhiluo