This project introduces the Decision Adapter, a neural network architecture designed to improve generalisation in reinforcement learning. In general, we use a hypernetwork to generate weights for a part of our main policy's neural network. This hypernetwork is conditioned on the context, a vector that represents certain aspects of the current task.
The file structure of this project is as follows:
src -> Where all of the source is
├── common -> Some general utils
└── runs ->
└── v0200/v0200.py -> The main run file
├── genrlise ->
│ ├── envs -> The environment definitions we use
│ ├── analyse -> General code relevant to analysis
│ ├── rlanalyse -> Additional analysis code
│ ├── utils -> More utilities
│ ├── common -> Common utilities
│ │ └── networks ->
│ │ │ └── segmented_adapter.py -> The Hypernetwork Adapter architecture
│ ├── contexts -> General code for working with contexts
│ │ ├── context_encoder.py ->
│ │ ├── context_sampler.py ->
│ │ └── problem.py ->
│ └── methods -> The code for each of the methods
│ ├── base ->
│ ├── clean -> This contains most of the code
│ │ └── sac ->
│ │ │ ├── clean_sac_adapter_v1.py -> The Decision Adapter method
│ │ │ └── base_clean_sac.py -> The training code for all methods
│ └── hyper -> Some training utilities
You can create the environment as follows:
conda env create -f env.yml
conda activate DA
We've found that this conda approach works, but we've also included the pip requirements file as sometimes that works better.
E.g.,
conda create -n DA python=3.9 pip=21.2.4 setuptools=58.0.4
conda activate DA
pip install -r requirements.txt
We note that the environment worked on Ubuntu 22.04 and 20.04 with CUDA 11.3. You may need to change the packages in env.yml to match your CUDA version, or alternatively change the CUDA version to match the one in env.yml.
Also, install Mujoco (e.g. using the instructions here).
In particular, we had to run the following
sudo apt install libosmesa6-dev libgl1-mesa-glx libglfw3And we added the required license key to the correct location (e.g. ~/.mujoco/mjkey.txt).
In addition to this, we've found that we had to delete ~/anadonda3/envs/DA/lib/libstdc++.so* for mujoco to work properly.
Then, to run any Python file in this repository, please use ./run.sh /path/to/file.py instead of using Python directly. Furthermore, run everything from the root of the repository.
This section specifies how to reproduce our results. In particular, the training must first be run, followed by the analysis code. Each training experiment is referred to by an experiment number, for instance v0601a-aa3. Here, this will be experiment v0601a, and setting/method/algorithm aa3. This experiment will then correspond to a yaml configuration file in artifacts/config/v020x_benchmarks/v0801/v0801-a1.yaml, which defines the type of experiment to run.
To run this experiment, one can run
./exp.sh v0601a-aa3 <slurm partition>If you want to run the code locally, you can use
./exp.sh v0601a-aa3 dummy False False TrueNote, to ensure out of memory errors do not occur, the code is set to run only one training seed at a time. If you want to run multiple seeds in parallel (which will be faster), you can remove the --override-cores=1 inside exp.sh.
This codebase is based on a Slurm cluster, but it can also be run locally. Inside src/genrlise/common/vars.py, the ROOT_DIR function must be changed to accurately reflect the root path of this repository, either locally or on the Slurm-based cluster.
def ROOT_DIR(is_local: bool):
if is_local:
return '/path/to/this/repo'
return '/path/to/this/repo'The following experiments must be run to obtain all of the results (click the arrow to see all of them).
Here, when we write v0601a, it means all of the experiments must be run within this directory (except the v0601a-_base.yaml one).
As an example, v0601a maps to the following commands:
./exp.sh v0601a-aa3
./exp.sh v0601a-bb3
./exp.sh v0601a-fb4
./exp.sh v0601a-u3
./exp.sh v0601a-x3Next, we write the experiments that must be performed as follows:
- Experiment Name 1
- Prerequisite 1
- Prerequisite 2
- ...
Therefore, we must run Prerequisite 1, then Prerequisite 2 (in any order). Once both of these are completed, we can run Experiment Name 1.
These are the main ODE experiments
v0801a(i.e. runv0601aandv0701afirst before running `v0801a):v0601av0701a
v0874b:v0674bv0774b
v1607_3bv1607_2b
v1707_3bv1707_2b
v0611bv0711bv1620_3b:v1619_1b
v1720_3b:v1719_1b
-
v0527n -
v0727n -
v1788_11a:v1578_1a
-
v1788_11d:v1578_1d
-
v1788_11e:v1578_1e
-
v1789_11a:v1579_1a
-
v1789_11d:v1579_1d
-
v1789_11e:v1579_1e
First run the above experiments before starting any of these. These experiments are the ones shown in the appendix.
v0601fv0701f
- First these
v0612bv0612cv0712bv0712c
- These
v0851av0851bv0851cv0851dv0851ev0851f
- Then these
v0856av0856bv0856cv0856dv0856ev0856f
- And these
v0857av0857bv0857cv0857dv0857ev0857f
v0801b:v0601bv0701b
v0801c:v0601cv0701c
v0801d:v0601dv0701d
v0801e:v0601ev0701e
v0981cv0681cv0781c
v0983a:v0683av0783a
v0983b:v0683bv0783b
v0805av0605av0705a
v0805bv0605bv0705b
v0805cv0605cv0705c
v0805dv0605dv0705d
v0805ev0605ev0705e
v0805fv0605fv0705f
v0805gv0605gv0705g
v0805hv0605hv0705h
v1608_3bv1608_2bv1608_9b
v1609_3bv1609_2bv1609_9b
v1708_3bv1708_2bv1708_9b
v1709_3bv1709_2bv1709_9b
v0861av0661av0761a
v0861bv0661bv0761b
v0861dv0661dv0761d
v0861fv0661fv0761f
After running the experiments, the analysis can be run using:
./run.sh src/analysis/progress/paper_plots/exp_results_a.py
./run.sh src/analysis/progress/paper_plots/exp_results_b.pyThen there should be a folder called artifacts/results/progress/paper_plots_a/ and artifacts/results/progress/paper_plots_b/ which contains all of the plots.