Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiprocess support #67

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

zebarnabe
Copy link

When testing my own agent I noticed that it can take quite some time to run the tournament if you have a few hundred agents, so here I added support to multiprocessing facilities in the main execution code.

I think this is something much needed when running the actual tournament, specially if you have a thousand or more agents to test, otherwise it will take a quite considerable amount of time. This obviously depends a lot on the machine where the code runs.

I've tested the code included on a 12 core machine with Ubuntu 20.04 and python 3.8.5, it is substantially faster than using the original iterative process.

I've tried to modify the original code as little as possible, on my local changes I also have additional stuff to measure how much time each agent is taking (my own takes about 20 times as long as the rest - I'll be optimizing it, but don't expect too much) as well as the generation of a grid comparing all agents with each other. If desired I can create another pull request with some of these extras.

The caveat is the possibility of issues arising in Windows machines, although I left the global variable that allows one to choose the number of child processes spawned, with zero it should work like the iterative approach. In hindsight I should have left the value set to zero, but then again the purpose of this code is to make it run faster than it was before.

An alternative to this approach would be to use MPI instead and use several remote machines in a compute cluster... but it would require a more complex setup and create more limitations for the running environment.

@zebarnabe
Copy link
Author

Changes proposed include the ones in #57 (wrap on the line that executes the code) as it might cause issues without it.

Also include #60 proposed changes as it could cause issues with dynamically load modules in a child process, this is not the case (at least on the environment tested), and when comes to the issue presented in #60 it actually alleviates it somehow, since the child processes only share data with the parent via Queues, meaning that it could only know about the matches in a single child process.

Regarding #60, a method to solve the leakage of information between runs would be to spawn a process for each round, if the process is terminated then the data goes with it, there is however a performance penalty with doing it that way (although, assuming multiple rounds would run in parallel it would be faster than running iteratively).

This reverts commit 0365b20.

Reverting commit of strategy in multiprocess-support branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant