[Proposal] Add the functionality of FireResetEnv to the AtariPreprocessing wrapper

[balazsgyenes]

Proposal

I am porting an example of DQN on Atari Breakout from Gym/Stable-Baselines3 to Gymnasium, and unfortunately there is no equivalent to the Stable-Baselines3's FireResetEnv wrapper in Gymnasium. The very good AtariPreprocessing wrapper seems to handle all the other functionality present in Stable-Baselines3's Atari wrappers, but not this.

I propose building this into the existing AtariPreprocessing as an optional feature, and if there is interest, I am more than happy to do this myself and submit a pull request :)

Motivation

Breakout requires an agent to use FIRE to launch the next "ball" after losing a life or losing the game.
Without this feature, the agent can get stuck if it hasn't learned to press FIRE after losing a life (or losing the game), and the FIRE action otherwise does nothing.
I am not sure if other Atari environments also have this problem, or only Breakout.

I found this long discussion about the functionality in the gym repo. Regardless, I don't understand the reason this wasn't ported, or (apparently) ported and then later removed? If it's default behavior in SB3, there are certainly people who would be interested in using it in Gymnasium.

Pitch

I would propose adding this functionality directly to the AtariPreprocessing wrapper, with a corresponding argument to switch the behaviour on and off. The default behaviour would remain the same. The wrapper would monitor the return value of lives() from the ALE (as it already does for terminal_on_life_loss), and apply an action with self.env.step(1) if lives have decreased. For reference, here is the source code of the FireResetEnv wrapper in Stable-Baselines3.

This implementation would be independent of the episodic life behavior (i.e. you have FireReset without episode life, or vice versa, or both features together).

Alternatives

One alternative is to directly use Stable-Baselines3's FireResetEnv wrapper, but this only works if also using the EpisodicLifeEnv wrapper. The fire reset functionality only triggers on an environment reset, not on a loss of life.
Maybe it's also possible to add the FIRE action to the environment's getStartingActions, but this is probably messy and backwards-incompatible.

Additional context

No response

Checklist

• I have checked that there is no similar issue in the repo

[pseudo-rnd-thoughts]

@balazsgyenes Thanks for pointing out the issue.
I think the AtariPreprocessing was implemented to include just the heuristics proposed Machado et al. (2018) however I think it is reasonable to include the FireReset heuristics but disabled by default.

Would you be able to make a PR to include it?

[shehper]

Is this issue resolved, or may I work on this issue? I am looking for a simple, good first issue to work on. Thank you.

[pseudo-rnd-thoughts]

@balazsgyenes Hasn't submitted a PR for it so @shehper if you want to work on it in the meantime then go for it

[balazsgyenes]

Sorry for the delay. I do have a local branch that isn't quite finished yet, I can push it and create a PR later today.

[pseudo-rnd-thoughts]

No worries, keep us updated on how it is going?