Skip to content

Modification of reward signals and rl_trainer for SAC #2433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 269 commits into from
Aug 15, 2019
Merged

Modification of reward signals and rl_trainer for SAC #2433

merged 269 commits into from
Aug 15, 2019

Conversation

@ervteng
Copy link
Contributor

@ervteng ervteng commented Aug 14, 2019 •
edited
Loading

Had to make additional changes to reward signals and rl_trainer to abstract to off-policy.

  • Adds evaluate_batch to reward signals. Evaluates on minibatch rather than on BrainInfo.
  • Changes the way reward signal results are reported in rl_trainer so that we get the pure, unprocessed environment reward separate from the reward signals.
  • Moves end_episode to rl_trainer

awjuliani and others added 30 commits April 24, 2019 14:24
@awjuliani
All reward signals use strength to scale output
9fa51c1
@awjuliani
produce scaled and unscaled reward
7f24677
@awjuliani
Remove unused dictionary
4a714d0
@awjuliani
Current trainer config
3e2671d
@awjuliani
Add discrete control and pyramid experimentation
77211d8
@awjuliani
Minor changes to GAIL
2334de8
@awjuliani
Add relevant strength parameters
439387e
@awjuliani
Replace string
ba793a3
@awjuliani
Add support for visual observations w/ GAIL
a52ba0b
@awjuliani
Finish implementing visual obs for GAIL
5b2ef22
@awjuliani
Include demo files
13542b4
@awjuliani
Fix for RNN w/ GAIL
ae7a8b0
@awjuliani
Keep track of reward streams separately
bf89082
@awjuliani
Bootstrap value estimates separately
360482b
@awjuliani
Add value head
c78639d
@awjuliani
Use sepaprate value streams for each reward
3b2485d
@awjuliani
Add VAIL
40bc9ba
@awjuliani
Use adaptive B
c6e1504
@vincentpierre
Comments improvements
60d9ff7
@vincentpierre
Added comments and refactored a pievce of the code
49ec682
@vincentpierre
Added Comments
d9847e0
@vincentpierre
Fix on Curriosity
dc7620b
@vincentpierre
Fixed typo
28e0bd5
@vincentpierre
Added a forgotten comment
0257d2b
@vincentpierre
Stabilized Vail learning. Still no learning for Walker
fd55c00
@vincentpierre
Fixing typo on curiosity when using visual input
2343b3f
@vincentpierre
Added some comments
c74ad19
@vincentpierre
modified the hyperparameters
2dd7c61
@vincentpierre
Fixed some of the tests, will need to refactor the reward signals in …
42429a5 
…order to test them all appropriately
@vincentpierre
Putting the has_updated fags inside each reward signal
ec0e106
@ervteng ervteng requested a review from harperj August 14, 2019 00:05
@ervteng ervteng changed the base branch from master to develop August 14, 2019 00:05
harperj
harperj reviewed Aug 14, 2019
View reviewed changes
Copy link
Contributor

@harperj harperj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of questions and minor suggestions, but mostly LGTM.

@harperj
Copy link
Contributor

harperj commented Aug 15, 2019

Latest changes LGTM. Can you briefly summarize how you tested this PR?

@ervteng
Copy link
Contributor Author

ervteng commented Aug 15, 2019

Latest changes LGTM. Can you briefly summarize how you tested this PR?

Tested GAIL with Crawler and Curiosity + GAIL with Pyramids. Tested GAIL with GridWorld for visual obs.

harperj
harperj approved these changes Aug 15, 2019
View reviewed changes
Copy link
Contributor

@harperj harperj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, approved!

@ervteng ervteng merged commit 27cd6dd into develop Aug 15, 2019
@ervteng ervteng deleted the develop-rewardsignalsforsac branch August 15, 2019 22:29
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@harperj harperj harperj approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ervteng @harperj @awjuliani @vincentpierre