[GSoC-Week1] First multi-agent environment

[Sid-Bhatia-0]

@findmyway @jonathan-laurent

From tomorrow, I'll start working on a multi-agent environment.
I am thinking of a cooperative multi-agent version of the CollectGemsUndirected environment with full-observability. A few reasons/advantages for choosing this:

1. This problem seems easy to transition from single-agent to multi-agent. Both single agent and multi-agent versions have the same purpose, to collect as many gems as fast as possible.
2. Rewards are distributed somewhat evenly during the episode. I think makes it easier to learn than sparse reward environments (as in the case of goal-reaching environments).
3. The difficulty of this problem is easily tunable with the density of scattered gems and number of agents.
4. Visualization of behavior in this environment can aid in easily detecting if the agents are learning to collaborate with each other (for example, by collecting gems from different regions of the map) or not (competing for the same gems).
5. Fully-observable because we want the agents to have a broader context in order to be able to collaborate effectively. And it is simpler to implement than partially-observable too.

Let me know what you think.

Also, please suggest if there is a better platform to document our discussions. RL.jl has a discussions section. Maybe we can enable it for GridWorlds.jl @findmyway .

[findmyway]

By the way, just to confirm. This environment may contain more than just two agents, right?

[Sid-Bhatia-0]

Yes. It can contain any number of agents. I'll start with an example with 4 agents.

[findmyway]

Hi @Sid-Bhatia-0 ,
Could you update your progress at least twice a week to keep things on track? Once at the beginning of a week to go over what we have achieved in the last week and plan ahead of the next week. Another ad hoc one to discuss what you are doing right now and submit your PRs so that I can help to review them before the end of a week. 😃

[Sid-Bhatia-0]

Right. Thanks for pointing out. Right now, I am working on an environment called CollectGemsUndirectedMultiAgent. The name is a little long. But I would prefer it to be descriptive :)
PR: #143

Last week I did some reading on GPUs and CUDA.jl. We also formulated a plan that entails the following:

1. Implement at least one multi-agent environment and check it by running a random policy on it (shall be done this week).
2. Think about how to use multithreading for batch environments. Needs some brainstorming. This also includes thinking about the role of TileMaps.jl.
3. Later, try to reuse code from multithreaded batch environments to be able to run on GPU.

Apart from this, I am strongly inclined towards adding the REPL-based playing (see example at #143) and recording of environments, and deprecating Makie rendering. Makie rendering has a bunch of issues (brittle, hard to customize, heavyweight, slow) that can be solved by using a simple REPL-based workflow. I am experimenting with it at https://github.com/Sid-Bhatia-0/GridWorldsGame.jl.