A couple of questions

[slerman12]

First, what are EPS_START, EPS_STOP, and EPS_STEPS? If I want episodes to last until the game naturally terminates an episode, how would I modify these? Could I just set EPS_STEPS to be a really large value?

Second, I'm using a 3D state space, and for some reason the following lines:

s = np.vstack(s)
a = np.vstack(a)
r = np.vstack(r)
s_ = np.vstack(s_)
s_mask = np.vstack(s_mask)

result in s and s_ having 3 dimensions instead of the proper 4 (batch dimension included). I changed these lines to:

s = np.array(s)
a = np.vstack(a)
r = np.vstack(r)
s_ = np.array(s_)
s_mask = np.vstack(s_mask)

Is that an acceptable solution or am I screwing up the logic in this way?

Thank you so much for your clarification. This code is immensely helpful and I appreciate it and the thorough explanation very much.