New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cpu% high #1424
Comments
The RXComputationalScheduler has nothing to do with Hystrix, it's a RxJava thread, and you will has 1 for each CPU you has in your machine, so you probably has an 8-core machine that's why you get 8 threads. About your cpu% going high you will need to provide more information about what you are doing because on the thread dump there are only threads waiting, no threads doing particularly nothing and shouldn't consume your cpu as you say. |
@ssakshi0302 i think you are misinterpreting the percentage value, this % is not the total amount of CPU used overall, it's the total usage from the overall, i think i'm not explaining it clear, i'll try to exemplify: First, check in the |
Hi ,the average cpu usage is 89.8%. As shown in the earlier sceenshot , |
@ssakshi0302 maybe i'm missing something, but i can't see your cpu average in the screenshot, you should be looking at: |
@ssakshi0302 now i see haha, thanks, this looks trick, what are you hystrix command doing? Executing some i/o task or something else? |
@kennedyoliveira : Any clue on this? |
@ssakshi0302 apologies for the delay, it's look trick to me, i'm not the best person to help with metrics, since i don't know the internals, @mattrjacobs can you help him with his doubt on it? which hystrix version are you using? |
Hey I am using version 1.5.3 On Sun, 20 Nov 2016 at 2:01 AM, Kennedy Oliveira notifications@github.com
|
@ssakshi0302 could you try to update to 1.5.8? I remember some weird problems in an old version, but i'm not sure if was about cpu performance. |
@kennedyoliveira : Thanks Oliveira , I will change the version but I wanted to understand why this could have occured and what does this stack indicate. Was it due to any flaw in metrics ? |
@mattrjacobs : Hi , What does the thread RxComputaionalScheduler do?I ould not find much in documentation. Can you please help us out here as we are facing production issues |
@ssakshi0302 i don't know if there are flaw in the metrics, because i don't know how the metrics works inside, i mean the implementation details, matt will help you better with that. About the RxComputationalScheduler, is a fixed size scheduler with total threads equal as total core cpus on your machine, intended to be used for doing computational tasks, it's from RxJava. In your screenshot from the allocation tab, could you print the stacktrace all the way up? Because it shows the internal operators calls but not who called it, may give a hint. |
@kennedyoliveira : Can you mention some email id , so that I can send flight recording itself. It will help you analyse better. |
Rx Computation threads are set up on any JVM which uses RxJava. Since Hystrix uses RxJava, the threads get set up. Within Hystrix they are used for metrics collection/aggregation, which happens in the background. In our internal systems, we've never seen any performance issues from their usage. In the attached thread dump, none of the RxComputation threads were busy, so that does not indicate over-usage of those threads. Please attach a FlightRecorder trace to this issue and I can take a look. |
That is a stack trace for what is allocation memory to TLABs. From the rest of the discussion, I thought your concern was around CPU usage? Can you please attach the entire jfr file? |
Hey, I am not able to attach any jfr file in github, I would be able to On Mon, Nov 21, 2016 at 11:46 PM, Matt Jacobs notifications@github.com
|
flight_recording_3002371128084_6 (copy).txt @mattrjacobs : Hi , |
@mattrjacobs : Any update ? |
I took a look through the JFR you attached, and it looks as you described. A couple questions:
From my initial look, all of these code paths look valid, but it's not clear to me why they're taking as much CPU as they are. That hasn't been the case for the production system I work on, nor in the jmh tests that ship with Hystrix. Could you run the jmh tests and take a flight-recorder trace during these? |
Closing due to inactivity |
Hi, too many RXComputationalScheduler threads are getting created and my application's cpu% is going high after using hystrix.
I am using this configuration
"groupName": "recsagservice", "threadPoolCoreSize": 5,
"cirBreakErrorThresPerc" : 50, "cirBreakReqVolThres":80, "cirBreakSleepTimeInMilliSec":72000 .
Please guide me what could be wrong here. I am attaching thread dump.
thread_hystrix.txt
The text was updated successfully, but these errors were encountered: