Solve Minefield

[cswinter]

Get > 0.99 episodic return on the Minefield task, which can be run with poetry run python enn_ppo/enn_ppo/train.py --gym-id=Minefield --env-kwargs='{"max_mines": 10}'.

[cswinter]

The Minefield task requires taking actions based on the features of other entities, which our current architecture is incapable of.
At a minimum, solving this environment requires #9, and possibly also better positional encodings.

Baseline run (67acae6): https://wandb.ai/cswinter/enn-ppo/runs/3tf8uk3l

[cswinter]

#108 makes the Minefield task more configurable, and also adds a translate argument that allows all x/y positions to be translated relative to the vehicle. This makes this task much easier, but it is still somewhat tricky due to the sparsity of rewards. It ought to be solvable now with the right hyperparameters. Additionally adding rotational invariance by rotating all object positions so that the direction of the actor is always 0 should also decrease its difficulty.

[cswinter]

Solution for max_mines=0 (W&B): poetry run python enn_ppo/enn_ppo/train.py --gym-id=Minefield --num-env=64 --num-steps=32 --track --total-timesteps=1000000 --n-layer=1 --env-kwargs='{"max_mines": 0, "translate": true}' --learning-rate=0.003

[cswinter]

We can almost solve max_mines=1, but it still struggles to squeeze out the last bits of performance (W&B): poetry run python enn_ppo/enn_ppo/train.py --gym-id=Minefield --num-env=64 --num-steps=32 --track --total-timesteps=2500000 --n-layer=2 --env-kwargs='{"max_mines": 1, "translate": true}' --learning-rate=0.003

There might still be a bug in the environment that makes it unsolvable in some cases, or the hyperparameters are bad, or something else is missing.

[cswinter]

Solved by poetry run python enn_ppo/enn_ppo/train.py --processes=16 --total-timesteps=25000000 --num-steps=512 --num-envs=512 --num-minibatches=64 --n-layer=2 --d-model=32 --learning-rate=0.01 --gamma=0.999 --ent-coef=0.0001 --gym-id=Minefield --track --env-kwargs="{\"max_mines\": 10, \"translate\": true}"
Commit: 766a379
W&B: 211230-205335-stack-more-layers