Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
telegraf logparser too slow #3539
I use telegraf to collect nginx logs to kafka. But it's too slow, the kafka hosts are in the same data center with the telegraf host, but it has one to two hours delay according to the log timestamp!
telegraf 1.4.5, centos 6.5
About 2000 lines per sec.
FYI, I try filebeat, and no delay there, although as known, filebeat is only a log shipper without grok scan.
Does anyone do some benchmark on telegraf with similar config to my case?
Can confirm, only able to parse at max 1000 entries per second. Easy to reproduce just create/copy apache access log with 100,000 entries and watch how slow it is parsed. With even a single threaded from scratch regex parsing I can achieve at least a million entries per second.
No need for custom log format, use standard format. Based on CPU usage one core is utilized and does not seem to be running at 100% almost as if telelgraf is throttling itself.