Update Chapter 10 #86
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
YanjieGao
reviewed
Apr 21, 2022
YanjieGao
reviewed
Apr 21, 2022
| - 对于采样器：策略（Policy， 即Q-Network）与环境交互，同样涉及探索与利用。但是GORILA里定义了一个Bundled Mode，即采样器的策略与学习器中实时更新的Q-Network是捆绑的。 | ||
|
|
||
| - 对于学习器： 学习器中对于Q-Network的参数梯度会发给参数服务器。 | ||
|
|
||
| - 对于重放缓冲区：在GORILA里分两种形式，在本地（local） 模式下就存在采样器所在的机器上；而多机（Golbal）模式下将所有的数据聚合在分布式数据库中，这样的优点是可伸缩性好，缺点是会有额外的通信开销。 | ||
|
|
||
| - 对于参数服务器：存储Q-Network中参数的梯度（Gradient）的变化，好处是可以让Q-Network进行回滚，并且可以通过多个梯度（Gradient)来使训练过程更加稳定。在分布式环境中，不可避免的就是稳定性问题（比如节点消失、网速变慢或机器变慢）。GORILA中采用了几个策略来解决这个问题，如丢弃过旧的和损失值（Loss）太过偏离均值时的梯度。 | ||
| >>>>>>> main |
YanjieGao
reviewed
Apr 21, 2022
|
图中可能时间作为下横轴，圆圈内为算法名更好些 |
Closed
YanjieGao
approved these changes
Apr 21, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.