-
Notifications
You must be signed in to change notification settings - Fork 509
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
第 10 章 Actor-Critic 算法代码实践 #29
Comments
看来你自己已经明白了,reinforce的更新是在跑完整个序列后,ac的不是,代码如下,具体看step在哪里运行 for i in reversed(range(len(reward_list))): # 从最后一步算起
reward = reward_list[i]
state = torch.tensor([state_list[i]], dtype=torch.float).to(self.device)
action = torch.tensor([action_list[i]]).view(-1, 1).to(self.device)
log_prob = torch.log(self.policy_net(state).gather(1, action))
G = self.gamma * G + reward
loss = -log_prob * G # 每一步的损失函数
loss.backward() # 反向传播计算梯度
self.optimizer.step() # 梯度下降 # 时序差分目标
td_target = rewards + self.gamma * self.critic(next_states) * (1 - dones)
td_delta = td_target - self.critic(states) # 时序差分误差
log_probs = torch.log(self.actor(states).gather(1, actions))
actor_loss = torch.mean(-log_probs * td_delta.detach())
# 均方误差损失函数
critic_loss = torch.mean(F.mse_loss(self.critic(states), td_target.detach()))
self.actor_optimizer.zero_grad()
self.critic_optimizer.zero_grad()
actor_loss.backward() # 计算策略网络的梯度
critic_loss.backward() # 计算价值网络的梯度
self.actor_optimizer.step() # 更新策略网络的参数
self.critic_optimizer.step() # 更新价值网络的参数 |
请问可以再详细的介绍一下吗?我还是不太理解,示例代码中AC算法执行的是 rl_utils.py 中的 train_on_policy_agent 函数,是收集一整个 Episode 之后的信息再执行 AC 的 update 函数,即使 AC 的 update 函数不是跟 REINFORCE 算法中的 update 函数一样用for循环,但是也不符合每个时间步更新策略这个特点。 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
本章Actor-Critic代码是在序列结束后才更新的,并没有体现Actor-Critic相比于REINFORCEMENT的优势:即可以单步更新,能够给出单步更新的Actor-Critic代码呀?
The text was updated successfully, but these errors were encountered: