Home

Marlben

Marlben is a multi-agent reinforcement learning benchmark that satisifes the following desiderata:

Variety. First, the set of environments must contain both basic and specialized target environments. The former allow algorithms to be compared by standard metrics, such as learning speed and total reward, and the latter, by their ability to adapt to spefic complexities of multi-agent learning, such as coordination, long-term planning, cooperation, specialization, and information exchange. Due to this property, the proposed set of environments can be useful for a wide range of studies. It's worth noting that this property is an advantage over some of the existing framework sets, including NeuralMMO, on which we've built our environments.

Unified structure. Secondly, despite the diversity described above, the environments should have the same low-level structure, including information available to agents, actions, and their rewards. This property is important not only for ease of use and modification of environments to the needs of the researcher, but also for the applicability of the same algorithms to all environments. Compared to existing counterparts, one of the most popular sets of Melting Pot environments is essentially a collection of many unrelated environments. As a consequence, often only a subset of the environments from this set are suitable for a single study.

The variability of each environment.

Random generation. Finally, environments contain an aspect of random (procedural) generation. Random generation makes environments more complex, as it requires agents to have more general and adaptive behavior in order to consistently maximize rewards. Specifically, we have implemented this aspect in all environments through the random generation of cell types on the maps inhabited by agents. It should be noted that in addition to randomly generated maps, we offer deterministic map options, as well as the ability to modify them by the user.

Application examples

Although the library is primarily intended for academic purposes, it can also be useful for applied applications. For example, in an obvious way, our library can be used to debug multi-agent learning algorithms before applying them to real problems. Compared to existing counterparts, our set of environments may be the most useful for this purpose due to their diversity. In addition, the very structure of our code can serve as a template for the implementation of applied projects on multi-agent reinforcement learning.

According to [1], examples of applications of multi-agent reinforcement learning are robot systems, bots for trading on electronic platforms, and decentralized resource management (for example, for network routing). Of course, these applications require training agents either directly in a real environment or in a simulator with subsequent transfer to a real environment. However, our environments can be useful for preliminary debugging of algorithms on similar problems. For example, Gathering can be useful for debugging basic agent navigation before training robot systems, Corridor (and possibly its custom modifications) for debugging negotiations with subsequent resource exchange before training trading bots, and Raid and Boss Fight before training resource managers ( since each agent's health pool serves as a resource). After debugging the algorithms, the user can reuse the code by replacing our environments with the ones necessary for the user. Such a process may be preferable, as the user will spend less time than if he writes code from scratch.

[1] Buşoniu L., Babuška R., De Schutter B. Multi-agent reinforcement learning: An overview //Innovations in multi-agent systems and applications-1. – 209. – С. 183-221.