How to stack frames ?

[simoninithomas]

Hi, I'm a little bit confused: I'm implementing a DQN in tensorflow and I want to stack each 4 frames.

I saw that in one of the issues you explained that we can use frame_repeat that defines for how many frames the given action will be executed. game.make_action(actions[a], frame_repeat)
First, is there the best way to get 4 frames?

Second, how we get the 4 frames?

Thanks for your help !

[Miffyli]

Are you trying to do what e.g. Mnih et al. (2015) did in their DQN paper, i.e. stack successive four frames into one state?

tics (or frame_repeat) in DoomGame.make_action(action, tics) specify for how many game frames the given action will be repeated, and next state returned will be given tics further from previous state.

The purpose of frame stacking is to give agent tiny bit of history and crude type of memory by giving also the previous frames as an input, in addition to current frame. VizDoom does not do this for you, so you have to manually store the states from DoomGame.get_state() and build up your stacked states for your DQN agent.

Roughly it can be implemented like this (pseudo-codeish)

from collections import deque
# Initialize deque with zero-images 
stacked_frames = deque([np.zeros((HEIGHT,WIDTH)) for i in range(stack_size)]) 
# Main training loop
while training:
    state = doomgame.get_state()
    # Append frame to deque, automatically removes the oldest frame
    stacked_frames.append(state.screen_buffer)
    # Build the stacked state (first dimension specifies different frames)
    stacked_state = np.stack(stacked_frames)
    # Feed the state to your agent for training/selecting actions
    action = agent.get_action(stacked_state)
    # Execute action for frame_skip steps
    doomgame.make_action(action, frame_skip)

[simoninithomas]

Perfect ! Thank you very much !

[mwydmuch]

Thanks @Miffyli for a great example. I linked this issue in a new FAQ.md file.