Best way to organize self-play

[alex-petrenko]

I am planning to experiment with population-based training and self-play, similar to the recent DeepMind's Q3 CTF paper. The obvious requirement would be the ability to train the agents to play against other agent copies on the same map at the same time.

I could probably wrap a multiplayer session into a single multi-agent interface and use ASYNC_PLAYER mode, maybe with increased tickrate (#209)
However the optimal way to implement this would be to render multiple observations for different agents within the same tick in the same process in synchronous mode, similar to how it's done in single-player.

Any thoughts on what is the right course of action here? Does multi-agent SYNC mode seem feasible or would it require changing half the codebase?

[Miffyli]

We experimented a bit with self-play in 2018 edition of the competition and didn't get anywhere with it (but that could be just our shabby agent codes which didn't work). At the moment I do not have access to the codes/notes to comment further, but I recall stuff like training being much slower and running into deadlocks and whatnot (sometimes player or two stopped responding to ViZDoom's commands, sometimes they segfaulted, etc). But, we did run longer experiments with smaller games, so this could be feasible as long you are fine with the ASYNC_PLAYER way. I can give better comments once I get hand on the notes/code again.

[alex-petrenko]

Hi, @Miffyli! Thanks a lot for the feedback!
So, as far as I understand, you used the central server (game host) with a bunch of ASYNC_PLAYER clients connected to it? Did you run the host in synchronous or asynchronous mode?

My current plan is to host a game in sync (PLAYER) mode, run the separate game instances for clients in the same process, or in different threads/processes. Then, after the host does the step() it will send some signal (e.g. cond var) to all other clients so that they can do their step. If this works, the game can proceed in the normal RL step-by-step fashion, provided that clients and the server are able to communicate their state between steps. At this point I am not sure whether I should run clients in sync or async mode.
I expect this to be much slower than normal singleplayer gameplay, but at least in this setup all participants are equal, collect an equal amount of frames and no one gets the host advantage. Not sure if that will work though.

[Miffyli]

Actually now I remember we also used SYNC mode, because nowadays SYNC mode is supported for multiplayer as well. In that sense you should be good to go.

And with the SYNC mode you should be fine with just step-ing every environment once to get "equal" observations from each agent. I will get hand of the code later/tomorrow, so I can give better comments and what worked and what didn't :)

[alex-petrenko]

That'd be awesome! I would love to chat more about that, so I'll be waiting for your feedback.
There's also a relevant post on DoomWorld forums about this project: https://www.doomworld.com/forum/topic/106770-ai-research-and-doom-multiplayer/?tab=comments#comment-2000068

[Miffyli]

I went through our code, and indeed it does not seem to be anything else than just setting Mode to PLAYER and then host/join games like in the cig_multiplayer examples. There are few quirks, though:

• We limited total number of players per game to 8 (including bots). IIRC the game could end up stuck with more players.
• All agents were in separate processes (multiprocessing.Process), each of which had their own instance of ViZDoom environment.
• We recreated all agents/vizdoom envs after finishing a game (killed the processes and created new ones). There are also a bunch of checking if processes (agents, workers, envs) are still alive. IIRC these were for fighting against occasional random deadlocks/crashes we weren't able to fix properly.

This project should be doable with ViZDoom, but you have to be careful with the networking as it was tad fragile. I would also manually check the states/observations from ViZDoom envs to see if they make sense (all agents progressed by X timesteps, agents executed correct actions, etc).

[alex-petrenko]

Ok, I guess I started to realize how this all works. So when you do game.init() in Python it actually spawns a separate Doom process. Therefore, regardless of whether the mode is sync or async, the host is always listening on the network port, regardless of the "agent loop". This is what allows it to function in the sync mode.

If you implement this "multi-agent" loop naively, I think there's nothing to guarantee that game state updates are broadcasted to all the clients before the next game step is made. I want all my clients to have the latest information about the game without any lag, but if the sync mode loop is too fast, there might be just no time for the changes to propagate to clients between steps.

A proper way to fix it would be to introduce some part of the game state that is synchronized between server and client and can be queried through Python interface, something like "state ID". Then the process is the following:

1. On the host: advance the game 1 step and change the state ID to something unique, e.g. increment the counter
2. Broadcast the changed State ID to the Python client processes, through some IPC means, not through Doom networking
3. On the clients: query the State ID variable until it is matching the latest State ID from the server. After this, advance the game one step and send the signal to the host that you're done.

Edit:
FYI there's a simple example on how to do it in repository actually!
https://github.com/mwydmuch/ViZDoom/blob/1e00b8d05ed58f439f317a24fc7efdf77d6eedea/examples/python/multiple_instances_advance.py
This one relies on sleeping for a few ms on the clients before advancing the state

[Miffyli]

If you use PLAYER mode (sync) with multiplayer, every player will wait till other players have made their actions before proceeding (the ViZDoom code syncs this). After asking @mihahauke , the sleeps in the code you linked are used to emulate the latency from running computations (that can take random amount of time), and to demonstrate that the code works even with random delays. Apparently if clients go out of sync, there should be an error.

[alex-petrenko]

@Miffyli thank you so much, your input was very helpful!
I was able to write an implementation of Vizdoom multi-agent environment following the RLLib interface: https://ray.readthedocs.io/en/latest/rllib-env.html#multi-agent-and-hierarchical

The only problem I have is that I am not able to use make_action(..., skip) where skip is >1
I usually train with skip == 4 which gives a great performance, but in the multi-agent scenario the game always gets stuck sooner or later. With skip == 2 it happens very rarely, with skip == 3 quite often, with skip == 4 - almost instantaneously.
I believe one of the clients gets stuck somewhere in DoomController::waitForDoomWork() and then the entire thing gets stuck forever, not able to advance. I don't know enough about how Doom multiplayer works, but indeed it seems to be very brittle. Either server gets ahead of the client by a couple of frames and can't handle it, or maybe the network packet is lost and never resent. Any thoughts on that? Do you think there's any chance this can work with skip_frames=4?

The other approach to multi-agent Doom, which is a multiple viewpoint rendering within the same game instance, seems even more tempting now. I know that Doom has a split-screen feature, so it should already be supported by the engine on some level: https://www.youtube.com/watch?v=k-fjc8hZaJA
Still I think this is a big effort - basically you'd need to make an 8-way split screen in a multiplayer match.

[alex-petrenko]

Ok, I was able to partially work around that by manipulating the update_state flag. This seems to be saving a lot of time by just not transmitting around the screen buffers we won't use due to frame skipping.

With the update_state trick I get around 3200 environment steps per second for 8 agents, so ~400 steps/sec per agent, or ~100 observations/sec/agent. Without update_state it's about 1900FPS max.
I believe the old Doom networking is probably the bottleneck now. Another idea would be to dig into Doom source code and replace network with something fast and local, like shared memory, pipes, etc.

Here's my current implementation: https://gist.github.com/alex-petrenko/5cf4686e6494ad3260c87f00d27b7e49
This example is not self-contained but should be enough to get the idea)

[alex-petrenko]

https://youtu.be/dHGSZRFTnf0

[Miffyli]

I also recall networking being a bottleneck when we run our things (something-something processes waited for messages something). Unless I am terribly mistaken, ZDoom uses the original P2P networking which does not cope well when you try to squeeze in many players or a lot of frames.

The implementation looks quite nifty and compact, though! With bit of tidying up it could be a nice example. Perhaps also a small write-up on challenges/requirements/limitations of multi-agent ViZDoom training, @mwydmuch ?

[alex-petrenko]

Definitely happy to collaborate! I am planning to tidy it up and get rid of "vizdoomgym" dependency at some point (or maybe we should actually make something like this in VizDoom repo, because for every project I end up using some kind of Gym wrapper). After this, it can probably be added to examples.

Performance-wise, I was able to push pure experience collection to around ~11500 environment frames on a 10-core 20-thread CPU. So this is ~2850 observations/sec with 16 parallel workers, each running an 8-agent environment, so 128 Doom processes in total. Doom renders at the 256x144 resolution, later downsampled to 128x72. Standard 160x120 -> 84x84 will give additional ~5% improvement, but I am sticking to widescreen now.
For comparison, I am getting ~39000 FPS on the same machine with ~200 parallel Doom envs in singleplayer mode.

So it is evident that old ZDoom p2p networking is a major bottleneck. I might look into ways to speed this up, but not right now.

[Miffyli]

"Official" gym environment in this repo would be nice indeed, but integrating it in the code is not too straight forward, since all ViZDoom code is done in C/C++. There has been separate repos for Gym envs, but they have died/quieted down. Easy way could be to add just an Python example that implements Gym API.

The numbers sound promising! Naturally Google had a lot of hardware to throw at their training, but I am willing to bet in local machines (like your 10-core) using ViZDoom is much faster :). I would like to hear when/if you get any results!

[alex-petrenko]

Thanks for the encouragement, definitely will share results!

Side note: it looks like the reward mechanism is behaving weirdly in multiplayer. If I give my agent reward of +1 every tick, e.g. in cig.acs script:

global int 0:reward;

script 2 ENTER
{
	while(1)
	{
		reward += 1.0;
		delay(1);
	}
}

then all of my agents also get the reward for everyone else. E.g. if I have 8 agents in the environment, then every single agent gets +8 reward every tick.
I assume this is just how it works because ACS variables are synchronized between all clients, but maybe this should be documented somewhere.
Bottom line: standard reward mechanism through ACS script does not work with multiplayer. I will have to look at the game variables in Python code to get my reward for kills, medkit pickups, etc.

Please correct me if I am wrong :)

[Miffyli]

Ah yes, forgot to mention this. Indeed the ACS scripts behave wonky, quite possibly due to reason like you said (it dies the += 1 for each agent etc) :). However for deathmatching you can quite easily implement the reward systems by tracking the variables, and with the label/map buffer you should be able to get most of the information for auxiliary rewards if you want to use them (e.g. is crosshair above enemy).