Detect, Module, Integrate and Transfer unit (DMIU) is a Shared experience lifelong learning system. A distributed lifelong learning framework where each agent is a lifelong learner capable of sharing and receiving knowledge about tasks from other agents.
Each lifelong learner agent is a PPO controller combined with Supermask Superposition algorithm for knowledge preservation across tasks. Basline agents is a regular PPO controller. No knowledge preservation algorithm is present, hence it catastrophically forgets.
Agent-based (reinforcement) learning. The DMIU code is built on top of existing DeepRL repository. The repository is extended on two front
- Extension of RL agents to lifelong learning framework, methods and experiments.
- Extension of 1. to shared experience lifelong learning framework.
Sample environments: Minigrid, CT-graph
- Same as the requirements of the DeepRL repository.
- Minigrid if running minigrid experiments.
- CT-graph if running CT-graph experiments.
Note: Example commands below using Minigrid environment. To run a baseline agent, use the command below:
python run_ll.py baseline --env_name minigrid --env_config_path ./env_configs/minigrid_sc_3.json --max_steps 102400
To run a single lifelong learner agent, use the command below:
python run_ll.py supermask --env_name minigrid --env_config_path ./env_configs/minigrid_sc_3.json --max_steps 102400
To run a ShELL (multi lifelong learner agents) in a single process (sequential executions per shell train step), use the command below:
python run_shell.py --shell_config_path ./shell.json
To run a ShELL (multi lifelong learner agents) across multiple processes (concurrent executions per shell train step) in a single machine, use the command below:
python run_shell_dist.py --shell_config_path ./shell.json
Note: Minigrid (with 3 simple crossing tasks) is specified as the default environment/tasks in shell.json. To run CT-graph experiments, update shell.json as shown below.
...
"env": {
"env_name": "ctgraph",
"env_config_path": "./env_configs/meta_ctgraph.json"
}
...
The repository is currently developed and maintained by researchers from Loughborough Univeristy, Vanderbilt University, UC Riverside, and UT Dallas
If you encounter any bugs using the code, please raise an issue in the repository on Github.
This material is based upon work supported by the United States Air Force Research Laboratory (AFRL) and Defense Advanced Research Projects Agency (DARPA) under Contract No. HR00112190132.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force Research Laboratory (AFRL) and Defense Advanced Research Projects Agency (DARPA).