ekanite cause a high cpu usage, any fix or how to debug? #35
Comments
|
Thanks for the report @zjqzero -- I need much more information that this. What was the program doing at the time? What resources does the program have? CPU, RAM, etc? Without this information, it's impossible to know if there is actually a problem. You can also enable standard Go CPU and memory profiling, via the command line options. Pass |
|
Thanks for your answer. @otoolep I run ekanite in docker on ubuntu14.04. The server has 1G memory and 1core cpu. every week there will produce about 2.5G data or above ekanite can run normal about 24 hours, here is the 7 days cpu usage here is some log: |
|
@zjqzero -- I need more information. Can you pass
|
|
Also, what rate of log messages are you indexing? |
|
When I use -cpuprof option. The cpu utilization has been normal for a few days. |
|
@otoolep On 20160811 it produce about 463M index and cause high cpu usage |
|
Does the CPU load disappear if you stop indexing? |
|
What OS are you running? |
|
HOST OS is ubuntu 14.04 |
|
Closed in error. |
|
Attached .svg file, showing profiling. Nothing jumps out at me. What rate are you indexing at? How many messages a second? https://github.com/ekanite/ekanite/blob/master/README.md#diagnostics will tell you the endpoint to hit to get stats. Perhaps log messages are just piling up within? I should point out that I have never done any performance optimization on Ekanite. There is probably a ton of stuff we could do to reduce CPU load. |
|
#37 is the start of better diagnostics support. Hopefully I can provide better debugging built into Ekanite soon, to help with issues like this. |
|
ok ， it will help much Sent from my iPhone
|
|
@zjqzero -- upgrade to v1.1.0 (just released) and see if that addresses your issue. v1.1.0 has a fix for an issue that might have resulted in excessive CPU usage. |
|
@otoolep please check. thanks
|
|
OK, thanks @zjqzero -- I ran a quick test of the code before releasing and it seemed fine to me. I'll take a look. Can I see your syslog-ng config? |
|
@otoolep here's my syslog-ng config file: @Version: 3.5 options { source file_app_log { template app { template("<${PRI}>1 ${ISODATE} ${HOST} VWMS ${PROGRAM} ${PID} - $MSG"); template_escape(no); }; destination d_net_app { log { |
|
Please show me the output of:
when the program is supposedly receiving log messages. |
|
zhou@L700:~/Test/ekanited-v1.1.0-linux-amd64/data$ curl localhost:9951/debug/vars |
|
Can you provide me an example log line? |
|
An easy way to see the log messages as they would be received by Ekanite is to start netcat instead, like so:
and watch the received log messages get dumped to stdout. |
|
netcat output: |
|
I'm not sure how it worked before, but can you try changing your syslog-ng config to: (Adding newline to all log messges, as per http://stackoverflow.com/questions/13701627/newlines-get-stripped-in-syslog-ng. Ekanite requires that each log message is newline-delimited.) |
|
Yeah, I am beginning to think there is an error in my syslog-ng template in the README. I meant to follow this example: https://www.loggly.com/docs/syslog-ng-manual-configuration/ and it has the |
|
@otoolep zhou@L700:~/Test$ netcat -l 0.0.0.0 5514 |
|
I just use /var/log/syslog of ubuntu for testing. You can try |
|
Thos log lines you supplied me are not compliant with the format required by Ekanite. You need to supply:
|
|
In other words, your syslog-ng template is not in the correct form for Ekanite. Remove |
|
Since haven't been using the right template, the system's use of regex may cause excessive CPU. I'm closing this until you retest with the correct syslog-ng template. If you still see high CPU usage, please let me know. |
|
ok, i see. thank you. |
|
Ekanited made one of my servers freeze up with just 30 log lines/s... is it supposed to be more performant than that? (Storage, RAM and CPU was definitely not a bottleneck here.) ... I was just using the defaults with the logs from iptables -j LOG. Everything's there, and looks fine, but I had to actually manually kill and restart it. Does it sound like a configuration problem? |
thx.
The text was updated successfully, but these errors were encountered: