Crashing with hundreds of processes #233
Comments
rgaufman
changed the title
|
i not tested so much processes, but ~100 was ok. Ruby uses 1 core for it concurrency, so 100% cpu usage possible. But still should not. Error repeated if restart eye? As workaround, you can try to run multiple eyes with group of processes in different folders (local eye - |
|
It's just constantly hovering with 100% and not responding to bundle exec eye i -- I can't really try multiple processes without re-architecturing how the app works. Is there something, somewhere I can adjust to reduce how many times each process is checked or something? - anything else you can think of to help with reducing load with many many processes? I temporarily reduced the number of processes to 300 and now it's taking 17 to 30% CPU - it seems it hits some kind of threshold and then everything stops working. |
|
To minimize load, you can remove cpu, memory checks. Also disable identity check Also, you can try increase expire in cache of getting cpu, memory info from OS: Eye::SystemResources.cache.setup_expire(30) |
|
I tried all of those things and still getting this: This is what's in top I've added Any other ideas/suggestions? |
|
hard to say, looks like may be celluloid bug. |
|
I have been trying to play with this without success :( I think I am going to have to come up with another strategy. Maybe to create a simple layer to systemd, where adding an application will create .service files and reload the systemd daemon. Kind of like the whenever gem does with cron. From there, the process will just be a single loop, that queries systemd and checks CPU/Ram and performs the required systemd action. This way, each iteration will just take a bit longer depending on the number of processes, without adding additional load. |
|
Yea ruby just bad, when high concurrency, try to split processes to multiple eyes or snt else, i dont know what to fix here. |
|
Have you made progress on this by chance @kostya? |
|
No, try all advices in this thread. |
|
I ended up switching to systemd with a simple ruby script to manage the processes. It actually was relatively easy, easier than I expected. Just an erb template for the .service file and then adding/removing them as needed, doing a systemctl enable and systemctl daemon-reload and it's doing most of the monitoring eye does but taking up virtually 0% cpu vs 100%+ eye was taking. It is also easy to add/remove dependencies and systemd just handles it all for you. The code is specific to my application, but some examples of how it works are: similarly for add except you need to generate it from an erb template. For checking status, you parse the output of systemctl status process-name. It is working like a dream and starting/stopping/restarting is lightening fast. |
|
@rgaufman With systemd, why use eye at all? |
|
I removed eye from our application, it's fantastic for managing a smaller number of processes but seems celluloid didn't work well when it came to 100+ processes. But even with smaller number of processes, we now just script systemd as it's much more resource efficient and has some other advantages like better dependency handling. |
|
@rgaufman do you have anything to share on this on github? |
I'm trying to monitor around 400 processes, eye jumps to 100% CPU on 1 core and then threads start crashing:
I'm also seeing errors like this:
and:
This is an AMD EPYC 7451 24-Core CPU, so it can handle it a lot more, with all processes running it's only 10% loaded. Are there any parameters I can tune in eye to handle this many processes?
The text was updated successfully, but these errors were encountered: