Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A Multi-Processing and Multi-Threading Manager #10

Open
DragonflyRobotics opened this issue Jun 20, 2022 · 0 comments
Open

A Multi-Processing and Multi-Threading Manager #10

DragonflyRobotics opened this issue Jun 20, 2022 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@DragonflyRobotics
Copy link
Member

Problem and Solution Description
Right now, MAGIST occupies only 2 threads, namely the main thread and the daemon thread. This is problematic when the AI is simultaneously training and processing data. This makes it nearly impossible to run without major improvements. If we had a manager that can assign tasks to all the threads across all cores in the system, it could run faster and more efficiently.

Detailed Description
Multi-threading and Multi-processing managers can be used to resolve this. In this, there would be a multiprocessing pool that would take tasks and assign them to a core manager. Each core manager takes the tasks and assigns them to a thread. This way, we get complete resource utilization. However, Python's GIL(Global Interpreter Lock) makes it arduous to implement.

Alternatives
A much simpler idea would be to implement a second worker node in the PriorityQueue class. If we add a second __worker function to the PriorityQueue class as well as a manager to automatically distribute tasks once they are published, we can have more processes running with faster and more efficient execution. This also resolves the issue described in #9 .

@DragonflyRobotics DragonflyRobotics added the enhancement New feature or request label Jun 20, 2022
@DragonflyRobotics DragonflyRobotics moved this from In progress to To do in MAGIST v0.1.0 AlphaBird Jun 20, 2022
@DragonflyRobotics DragonflyRobotics moved this from To do to Reviewer approved in MAGIST v0.1.0 AlphaBird Jun 20, 2022
@DragonflyRobotics DragonflyRobotics moved this from Reviewer approved to Review in progress in MAGIST v0.1.0 AlphaBird Jun 20, 2022
@DragonflyRobotics DragonflyRobotics removed this from Review in progress in MAGIST v0.1.0 AlphaBird Jun 20, 2022
@DragonflyRobotics DragonflyRobotics self-assigned this Jun 20, 2022
@DragonflyRobotics DragonflyRobotics moved this from To do to In progress in MAGIST Long-Term Jul 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
No open projects
MAGIST Long-Term
  
In progress
Development

No branches or pull requests

1 participant