Question: Why does DoomGame::updateState not update if isEpisodeFinished()?

[mhe500]

Hello,

In one of my use cases it is necessary to have an observation even when the step leads to a done state. To-date I've been creating a dummy game state of zeros, however this is not ideal.

I've removed the condition and everything seems to still work (in my limited test cases), but I'm wondering if there was any specific reason to not provide game state after the final step that I'm not seeing?

Thanks.
mhe

[Miffyli]

I have been coping with the same issue for the past few years, and usually I feed in the latest working observation (not sure if this is the best idea, though. Blank observation sounds more reasonable). I assume this has to do with the ZDoom moving out from some game loop once game ends, and thus game information is not available for reading. This sounds like something @mwydmuch would know if he happens to stop by at some point :)

[mhe500]

Thanks for the info @Miffyli. I would speculate the same reason, @mwydmuch coding defensively.

For the case of a timeout, however, not having the next observation will make your discounted returns incorrect (e.g., in the simple case, r_t + gamma * VorQ(s_{t+1})). Either using zeros (my hack) or the current state s_t (your hack) in place of s_{t+1} is certainty wrong (presuming you do not set your discount to 0 on a timeout). FWIW I think your approach is more reasonable as the two states are closer together.

Anyway, I plan to leave it this way for a while and see if it works. If its not problematic I may open an issue to propose the idea of determining the reason that a tic is not possible, and in the specific case of timeout, providing the observation.

[mwydmuch]

I want to close this issue, sorry I didn't answer back when it was discussed, so I will do it now. You were right. After the end of the episode, ZDoom engine changes its state to dead/summary screen, and some internal objects/variables are destroyed, so we cannot provide a proper state, especially considering the number of additional data and variables ViZDoom may provide. And because we didn't want to introduce additional checks when game logic is processed, for simpler code, ViZDoom updates and checks state only after complete logic tic. So back then, we decided just to provide None/Null state. We could also save the state of the environment every step (even when the skip is larger than 1) and provide the last one available, but this is not perfect and would cost some performance for skips larger than 1, so we decided not to do that.