Code for paper Lifelong Context-Aware Retrieval and Composition for Scalable Knowledge Reuse. Implementation of modulatory mask combined with PPO. The repository contains MASK RI/LC/BLC implementations. Please see EWC branch for implemenation of PPO and Online EWC. Implementation of Mask Selective Combination (Mask-SC) with PPO which introduces task similarity-based retrieval of prior policies for composition to improve the scalability of forward transfer over long task sequences in lifelong reinforcement learning.
The code was developed on top of the Mask-LRL repository, extending the Mask-LC algorithm.
Figure 1: High-level illustration of Mask Selective Combination.
- CT-graph
- Minigrid
- MiniHack
- Continual World (see note below)
- See requirements.txt file
- See CT-graph requirements.
- See Minigrid requirements.
- See MiniHack requirements.
- See Continual World requirements and how to install. Note, MuJoCo (now freely available) is required to run Continual World
Example commands below using CT-graph environment. To run agents in the CT-graph CT28 curriculum defined in the paper, use the command below:
# baseline PPO agent.
python train_ctgraph.py baseline --seed 86
# selective retrieval + composition (MASK SC) agent.
python train_ctgraph.py ll_supermask --new_task_mask linear_comb --seed 86
Full experiments were run using the commands:
python launcher.py --env ctgraph_sc --exp ct28_mask_sc/
which will produce the path ./log/ct28_mask_sc/ containing all seed runs.
The launcher.py script can also be used to run the single-task experts experiments which can be used to
Note:
- the command to run a balanced linear combination (MASK BLC) agent is the same as the MASK LC command above, but should be run in the
exp_maskblcgit branch. - the full list of commands to run experiments in the paper can be found in the
paper_experiments.txtfile. - sample commands and the full list of commands for
ewcexperiments in the paper can be found in theexp_ewcgit branch. - sample commands and the full list of commands for setting up the single task expert (STE) experiments can be found in the
exp_stegit branch. - In the continualworld curriculum (CW10), the random initialization mask agent implemented in this branch is the MASK RI_C (continuous values mask). The sample command to run MASK RI_\D in CW10 can be found in the
exp_maskri_discrete_mask_cw10git branch.
The analysis pipeline for Mask-SC can be executed using the eval_XXX.py and
ft_auc_analysis.py scripts.
AMSC standardizes each current-task similarity vector before applying temperature-scaled sparsemax. Two selector ablations are available in the CTGraph, MiniGrid, and Continual World trainers:
# NoNorm: sparsemax over raw cosine similarities
--selection_no_normalization
# Shuffled: preserve AMSC's support size but randomly assign it to priors
--selection_shuffle_supportThe shuffled selector uses a dedicated RNG derived from the RL seed. It does not change the random stream used by PPO, SAC, or the environments. Both flags can be combined, although the primary ablations use one flag at a time.
Examples:
python train_ctgraph.py ll_supermask --new_task_mask linear_comb \
--selection_no_normalization
python train_minigrid.py ll_supermask --new_task_mask linear_comb \
--selection_shuffle_support --disable_task_label_input
python train_continualworld.py ll_supermask --new_task_mask linear_comb \
--selection_shuffle_support --disable_task_label_inputTo cite this work, please use the information below. Thanks.
@article{esbn2022masklrl,
title={Lifelong Reinforcement Learning with Modulating Masks},
author={Ben-Iwhiwhu, Eseoghene and Nath, Saptarshi and Pilly, Praveen K and Kolouri, Soheil and Soltoggio, Andrea},
journal={arXiv preprint arXiv:2212.11110},
year={2022}
}
This repository is currently maintained by researchers from Loughborough University.
If you encounter any bugs using the code, please raise an issue in this repository on Github.
The Continual World benchmark was built on top of the Meta-World benchmark, which comprise of a number of simulated robotics tasks. The originally released Continual World employed the use of version 1 (v1) Meta-World environments. However, the Meta-World v1 environments contained some issues in the reward function (discussed here and here) which was fixed in the updated v2 environments. Therefore, the experiments in the paper employed the use of the v2 environment for each task in the Continual World. The modification can be downloaded from the forked repository here.
TBD