Handle Large LiveStatus Datasets Better?

[bjhiggins]

I have a large number of Hosts and Services in my iCinga instance. 3394 Hosts and 4957 Services. When setting up NagiosTV and connectors/livestatus.php - I had to increase my PHP-FPM memory_limit from 256M to 1024M.

I believe that LiveStatus allows for a "Limit" keyword to return a smaller number of records (https://docs.checkmk.com/latest/en/livestatus.html#_limiting_an_output_limit)

Is there any way to update the connector so that it only requests Hosts/Services which are not in an OK Status rather than all records? This would shrink the memory requirements of the PHP script and make response time quicker.

[chriscareycode]

I like this idea of having the API only return items not in OK status to help the product scale and decrease the CPU on the server.

For Nagios CGIs it looks like we can do this adding this hostservice and servicestatus parameter to the API calls:

Hosts
http://10.10.0.42/nagios/cgi-bin/statusjson.cgi?query=hostlist&details=true&hoststatus=down+unreachable+pending

Services
http://10.10.0.42/nagios/cgi-bin/statusjson.cgi?query=servicelist&details=true&servicestatus=warning+critical+unknown+pending

For Livestatus we can do this with:

GET hosts
Filter: state = 1
Filter: state = 2
Or: 2

GET services
Filter: state = 1
Filter: state = 2
Filter: state = 3
Or: 3

After making this change, payloads are way smaller and this will make a huge improvement on the server CPU and the web browser. But now that we are only fetching things that are not UP and not OK, I need to fetch a new lightweight API "hostcount" and "servicecount" to get the "total # of hosts" and "total # of services" since we no longer have that information from the hoststatus and servicestatus apis.

Got through most of the work for this. I can share a dev build with you to try out.

[chriscareycode]

This has been added in https://github.com/chriscareycode/nagiostv-react/releases/tag/v0.8.7

[chriscareycode]

@bjhiggins Please test out 0.8.7, see how it works and report back