Thruk 3.10 livestatus per_source_limit #1386
Description
Describe the bug
We have two Thruk servers both running 3.10
They both connect to the same set of Nagios servers but recently we have been reciving timeouts while connecting to certain large Nagios backends
I can see the issue is livestatus running on the nagios servers hitting their connection limits:
Aug 07 08:34:54 nagserver.domain.com xinetd[2040]: FAIL: livestatus per_source_limit from=thrukserver
Aug 07 08:34:54 nagserver.domain.com xinetd[2040]: FAIL: livestatus per_source_limit from=thrukserver
Aug 07 08:34:54 nagserver.domain.com xinetd[2040]: FAIL: livestatus per_source_limit from=thrukserver
Aug 07 08:34:54 nagserver.domain.com xinetd[2040]: FAIL: livestatus per_source_limit from=thrukserver
However when we increase these limits it simply increases the number of connections until it hits them again
The livestatus socket is currently set to:
Define access restriction defaults
cps = 2000 3
instances = 500
per_source = 500
Thruk Version
Thruk 3.10
running on RHEL 7.6 / 7.7
Running as a VM at:
16 cores
31gb of ram
This has started happening at the same time to both boxes which are in two different datacentres connecting to multiple backends in different regions - no other changes have been made to the hardware/software/network in this time
Is there a timeout limit on livestatus somewhere that we are hitting?
Is there a maximum number of connections that Thruk can handle?
The servers affected are the largest on the platform