-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rrrspec-master cpu 100% #63
Comments
Since you say that you're running Cucumber workers, I suppose that you're running RRRSpec with your patch. It makes it difficult for me to guess the internal state. Also I cannot comment on what's going on in your system as I do not know the actual setup precisely. All I can do is giving you some advice.
|
Problem cause: As the workers are on google cloud and the master is in the office the master can assign tasksets much faster than worker can consume it. As the worker always notifies the master before consuming a taskset this creates a loop resulting in the master assigning the same taskset N times. After the execution of several tasksets the system "deadlocks". Having more workers causes the "deadlock" to appear faster. This is because they all notify the master to assign them more tasks. This causes the cpu to rise to 100% in the master. |
I've made a fix at #64 |
I still cannot get the situation. Can you write an interleaved scenario? |
The setup:
The system starts to run ok, but after some 5 mixed invocations of the pull request jobs and the master to staging jobs the rrrspec-master ruby process locks 100% making the system unusable.
After canceling all the running tasksets and waiting something like an hour, the cpu goes idle.
I suspect it's rspec-master saving worker data from redis. You can see the attached screenshot, there is one worker that get repeated in the ui, this is wrong it should show one line for each worker.
The text was updated successfully, but these errors were encountered: