You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I opened this issue regarding divergence problems. I think you already suggested some solutions (#11 ) to this problem but still there are some problems. I spent a couple of months to handle this problem but still faced with same issue. For a better discussion, I think I need to explain the training environment.
I am using your approach in vision-based collision avoidance problem. If the agent reaches to a goal point or encounter collision, I restart my episode in a random position. It seems like my agent is learning well initially; however, the critic network starts to diverge and the network is totally destroyed at the end. It seems like the critic network thinks that current states are very good.
Here are my training settings:
initial alpha value: 1e-3
actor learning rate: 3e-5
critic learnign rate: 3e-4
CNN layers for actor and critic: DQN structure layers
Actor network uses Squashed Normal
phi value (The value in the 26~44 line in iq.py): 1
SAC with single critic without updating alpha
I have tried with different initial alpha value and found out that using higher initial alpha value gives more stability to network but resulted in poor performance (The behavior of agent is far from the expert behavior). Am I using it in a wrong way or needs more hyperparameter tunings?
I attach loss figures for more clear understanding. Waiting for your response. Many Thanks.
This is my iq.py to update critic.
def q_update(self, current_Q, current_v, next_v, done_masks, is_expert):
with torch.no_grad():
y = (1-done) * self.args.gamma * next_v
reward = (current_Q - y)[is_expert]
# 1st loss function
loss = -(reward).mean()
# 2nd loss function
value_loss = (current_v - y).mean()
loss += value_loss
# Use χ2 divergence (calculate the regularization term for IQ loss using expert and policy states) (works online)
reward = current_Q - y
chi2_loss = 1/(4 * 0.5) * (reward**2).mean()
return loss
The text was updated successfully, but these errors were encountered:
Hi, I opened this issue regarding divergence problems. I think you already suggested some solutions (#11 ) to this problem but still there are some problems. I spent a couple of months to handle this problem but still faced with same issue. For a better discussion, I think I need to explain the training environment.
I am using your approach in vision-based collision avoidance problem. If the agent reaches to a goal point or encounter collision, I restart my episode in a random position. It seems like my agent is learning well initially; however, the critic network starts to diverge and the network is totally destroyed at the end. It seems like the critic network thinks that current states are very good.
Here are my training settings:
iq.py
): 1I have tried with different
initial alpha value
and found out that using higherinitial alpha value
gives more stability to network but resulted in poor performance (The behavior of agent is far from the expert behavior). Am I using it in a wrong way or needs more hyperparameter tunings?I attach loss figures for more clear understanding. Waiting for your response. Many Thanks.
This is my
iq.py
to update critic.The text was updated successfully, but these errors were encountered: