Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have the worker use a process pool #56

Open
kureuil opened this issue Sep 30, 2018 · 0 comments
Open

Have the worker use a process pool #56

kureuil opened this issue Sep 30, 2018 · 0 comments
Labels
enhancement New feature or request

Comments

@kureuil
Copy link
Owner

kureuil commented Sep 30, 2018

When the worker receives a job from the broker, it spawns a brand new process per job. While this is easy to implement and monitor, it is sub-optimal on the performance side.

One possible solution to this performance problem would be to have a pool of processes, which are kept alive as long as possible. The main worker process would send over to these "worker worker" processes the jobs to execute, and they would send a message back to acknowledge or reject the job execution. If one the processes dies, it should be restarted and the job should be sent back to the broker as "rejected".

@kureuil kureuil added the enhancement New feature or request label Sep 30, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant