Open-Ended Learning Leads to Generally Capable Agents

[francelico]

Link: https://arxiv.org/abs/2107.12808

Authors: Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, Wojciech Marian Czarnecki

Abstract: In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards. We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours. The resulting agent is able to score reward in every one of our humanly solvable evaluation levels, with behaviour generalising to many held-out points in the universe of tasks. Examples of this zero-shot generalisation include good performance on Hide and Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks we characterise the behaviour of our agent, and find interesting emergent heuristic behaviours such as trial-and-error experimentation, simple tool use, option switching, and cooperation. Finally, we demonstrate that the general capabilities of this agent could unlock larger scale transfer of behaviour through cheap finetuning.

Notes: Lots of subfields coming together in this publication.

Extra material: https://www.deepmind.com/publications/open-ended-learning-leads-to-generally-capable-agents

[francelico]

The XLand environment

[francelico]

The Curriculum Generation Process

[francelico]

The Agent Architecture

[francelico]

The Goal Attention (GOAT) module architecture

[francelico]

The objective
Different from the standard RL objective as it uses Pareto optimality, auxiliary losses

[francelico]

Representations Analysis
More generally, approach taken to extract insights in order to inform the design process of such a complex system.

[francelico]

Evaluating Generalisation Capabilities
What constitutes a good generalisation metric, are there different types of generalisation?
For example: generalising over reward functions vs. dynamics vs. state spaces.
And: generalisation as highest average performance over task space vs. generalisation as a lower expected performance bound over task space vs. generalisation as a lower absolute performance bound over task space.

[francelico]

Discussion on exciting future directions of Open-ended learning
topics: emergent behaviors, practical uses beyond research, most exciting future directions for XLand

[francelico]

Discussion on the learning dynamics
Effect and Interplay between environment-agent interactions, agent-agent interactions, evolutionary methods, neural net modules, distillation

[francelico]

Multi-Agent Interactions
A subset of the learning dynamics, more focused around MARL.

[francelico]

Presented on 18/08/22.