Async n-step q-learning and one step sarsa #1084
Conversation
| @@ -137,6 +139,22 @@ template < | |||
| > | |||
| class OneStepQLearningWorker; | |||
|
|
|||
| template < | |||
| typename EnvironmentType, | |||
There was a problem hiding this comment.
Can you comment on the template parameter?
|
Would there be more comments on this PR? |
| * @tparam EnvironmentType The type of the reinforcement learning task. | ||
| * @tparam NetworkType The type of the network model. | ||
| * @tparam UpdaterType The type of the optimizer. | ||
| * @tparam PolicyType The type of the behavior policy. * |
| using ActionType = typename EnvironmentType::Action; | ||
| using TransitionType = std::tuple<StateType, ActionType, double, StateType>; | ||
|
|
||
| /** |
There was a problem hiding this comment.
Do you mind to add a method description here; something like should do:
Construct N step Q-Learning worker with the given parameters and environment.
| network.Backward(actionValue, gradients); | ||
|
|
||
| // Accumulate gradients. | ||
| totalGradients += gradients; |
There was a problem hiding this comment.
We should initialize totalGradients with zero.
| * @tparam EnvironmentType The type of the reinforcement learning task. | ||
| * @tparam NetworkType The type of the network model. | ||
| * @tparam UpdaterType The type of the optimizer. | ||
| * @tparam PolicyType The type of the behavior policy. * |
| ActionType>; | ||
|
|
||
| /** | ||
| * @param updater The optimizer. |
There was a problem hiding this comment.
Can you add a method description here?
| if (terminal || pendingIndex >= config.UpdateInterval()) | ||
| { | ||
| // Initialize the gradient storage. | ||
| arma::mat totalGradients(learningNetwork.Parameters().n_rows, |
There was a problem hiding this comment.
We should initialize totalGradients with zeros here.
| double targetActionValue = actionValue[std::get<4>(transition)]; | ||
| if (terminal && i == pending.size() - 1) | ||
| targetActionValue = 0; | ||
| targetActionValue = std::get<2>(transition) + |
There was a problem hiding this comment.
Should we use an else case here? That way we can slightly simplify the expression if (terminal && i == pending.size() - 1) is true.
| config(config), | ||
| deterministic(deterministic), | ||
| pending(config.UpdateInterval()) | ||
| { reset(); } |
There was a problem hiding this comment.
Can you use Upper camel casing for all method names?
|
Thanks for your feedback. Hope it's ready to merge now. |
|
Thanks for another great contribution. |
No description provided.