How to Learn and Represent Abstractions: An Investigation using Symbolic Alchemy

Paper | Visualization Tool

This repository contains the code to train and evaluate the A2C EPN agent on the Symbolic Alchemy environment as detailed in our paper. We also release a visualization tool to debug the agent's behavior here.

A2C EPN

Setup

You will first need to download and install the DeepMind Alchemy Environment as follows:

$ git clone https://github.com/deepmind/dm_alchemy.git
$ pip install wheel
$ pip install --upgrade setuptools
$ pip install ./dm_alchemy

Then install the remaining requirements

pip install -r requirements.txt

Pretrained Models

• Best A2C EPN agent achieves an average episode score of 272.85 ± 1.78 with the modified representations.

• A2C EPN agent trained without extra reward shaping (i.e. penalties) achieves a score of 236.06 ± 2.18.

Abstract

Alchemy is a new meta-learning environment rich enough to contain interesting abstractions, yet simple enough to make fine-grained analysis tractable. Further, Alchemy provides an optional symbolic interface that enables meta-RL research without a large compute budget. In this work, we take the first steps toward using Symbolic Alchemy to identify design choices that enable deep-RL agents to learn various types of abstraction. Then, using a variety of behavioral and introspective analyses we investigate how our trained agents use and represent abstract task variables, and find intriguing connections to the neuroscience of abstraction. We conclude by discussing the next steps for using meta-RL and Alchemy to better understand the representation of abstract variables in the brain.

Citation

@misc{alkhamissi2021_symbolic_alchemy,
      title={How to Learn and Represent Abstractions: An Investigation using Symbolic Alchemy}, 
      author={Badr AlKhamissi and Akshay Srinivasan and Zeb-Kurth Nelson and Sam Ritter},
      year={2021},
      eprint={2112.08360},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}