For pi0.6 star, are the following steps correct if we try to implement from the base pi0.5 checkpoint :
a) Fine tune the model normally for a task
b) For every demonstration data set and a baseline inference : human augments the trajectory and labels it as 0 or 1
c) Rewards are synthesized so that every time step is penalized and termination success gives a large value
d) The rewards are binned into 201 bins and the value function is trained as a multi class classifier
e) Now, after the value function is trained, for every state action pair, an estimated advantage function is approximated and binarized to success or failure
f) For the new / old / augmented dataset the potential successful sequences as estimated by the value function is trained again via a cross entropy loss : so when the value function feels the episode will succeed it will back prop weighted by alpha ?
Is the mental model correct; or are we missing something ?
For pi0.6 star, are the following steps correct if we try to implement from the base pi0.5 checkpoint :
a) Fine tune the model normally for a task
b) For every demonstration data set and a baseline inference : human augments the trajectory and labels it as 0 or 1
c) Rewards are synthesized so that every time step is penalized and termination success gives a large value
d) The rewards are binned into 201 bins and the value function is trained as a multi class classifier
e) Now, after the value function is trained, for every state action pair, an estimated advantage function is approximated and binarized to success or failure
f) For the new / old / augmented dataset the potential successful sequences as estimated by the value function is trained again via a cross entropy loss : so when the value function feels the episode will succeed it will back prop weighted by alpha ?
Is the mental model correct; or are we missing something ?