Update ALE for ROM fixes and snapshot/restore of state

[shelhamer]

ALE development has gone on from the point that atari-py forked. While atari-py forked from ALE without keeping it's history, the diff can be inspected by comparing atari-py/atari_py/ale_interface/src and ale/src. Among the various fixes and improvements two stand out for the purposes of gym:

• ROM fixes that alter the env semantics(!)
• proper snapshot/restore of the emulator state including pseudorandomness (cloneSystemState() and restoreSystemState())

This is required to address openai/gym#402. I'm happy to carry out the gym change to expose the snapshot/restore functionality next.

Do not merge until we double-check that nothing breaks and decide that we're happy with the new ROM semantics. Once merged, gym should bump the env versions when the new atari-py is incorporated.

[shelhamer]

Note that I almost certainly blew away improves by Trevor Blackwell—I'll review the history and spare them from this diff. Apologies! Just kidding—the customizations that have been made for atari-py are safe.

[shelhamer]

Right now this includes new ALE output on when making an env. I'll most likely disable it, or better yet switch it to being a verbose option.

A.L.E: Arcade Learning Environment (version 0.5.1)
[Powered by Stella]
Use -help for help screen.
Warning: couldn't load settings file: ./ale.cfg
Game console created:
  ROM file:  /Users/shelhamer/h/desk/atari/atari-py/atari_py/atari_roms/pong.bin
  Cart Name: Video Olympics (1978) (Atari)
  Cart MD5:  60e0ea3cbe0913d39803477945e9e5ec
  Display Format:  AUTO-DETECT ==> NTSC
  ROM Size:        2048
  Bankswitch Type: AUTO-DETECT ==> 2K

Running ROM file...
Random seed is 436853534
[2017-04-12 23:31:47,088] Making new env: PongNoFrameskip-v3

By default only errors are logged.

[shelhamer]

Note to self: double-check stellarc/ale.cfg The configuration file is deprecated and defaults are handled by settings in emucore.

[shelhamer]

Alright, this should be ready. I did a few simple checks of the environments, but more testing and review might be a good idea before merge.

[joschu]

@nottombrown could you try running our RL algorithm baselines with this branch?

[nottombrown]

Bumped the version and am testing now using PPO (because it's fast)

 ॐ  python -c 'import pkg_resources;  print(pkg_resources.require("atari-py")[0].version)'
0.0.22.dev0

[nottombrown]

Results

Is it supposed to preserve seed determinism with previous versions?

I'm a bit confused by the graph above, @shelhamer. It looks like enduro was preserved, but not pong

[joschu]

Awesome -- the rare uniform improvement. Let's merge!

[shelhamer]

@nottombrown I'd expect slight improvement in Qbert and Breakout because of minor alterations of the ROMs in 199fdf3 (which fix a terminal condition and reduce the minimal action set respectively), and otherwise equivalent results.

However, seeds are not preserved across this change because of a switch in the ALE RNG in Farama-Foundation/Arcade-Learning-Environment@a5241ce; see https://github.com/mgbellemare/Arcade-Learning-Environment/commits/master/src/emucore/Random.cxx for more history. There is still determinism for the new seed/RNG scheme, but it will give different random numbers so it's all the more important to bump the Atari envs to v4.

[shelhamer]

Thanks @nottombrown for catching the version bump!

[Kojoley]

Only GNU extension allows declaring variables in the middle of the block.