core: log a message when reaper max time is reached #3159

this is still logged to icinga.debug, but in favor of letting the user
know in the first place, we'll spit a warning to syslog as well.

reaching the max reaper time means - core is not capable of processing
the checkresult list in memory, it gets longer and longer, and ever
since stashing checkresults onto this list does not happen by just
adding it (O(1)) at the end, but sorting the list by date, it will cause
worst case (O(n), decreasing overall performance just by the reaper not
being able to process more checkresults.
while this won't solve the issue, it will at least warn the user
directly that the reaper runs into performance issues. the current
reaped checkresult counter will be logged as well, in order to get an
idea what's going on. further performance analysis may then be required
by the user.

refs #3159

Changelog

@@ -21,6 +21,7 @@ ENHANCEMENTS
 * core: add hint on icinga.cfg package location, and tip to read Changelog CHANGES on upgrades #2879 - MF
 * core: add new callback NEBTYPE_CONTACTNOTIFICATIONMETHOD_EXECUTE to allow override of sending notifications #2041 - MF
 * core: only reap results when checkresult_list is not overloaded (max_check_result_list_items) (thx Thomas Gelf) #3099 - MF
+* core: log a message when reaper max time is reached #3159 - MF
 
 * classic ui: Dropped statuswml.cgi #2654 - RB
 * classic ui: Dropped statuswrl.cgi #2359 - RB

base/checks.c

@@ -367,6 +367,7 @@ int reap_check_results(void) {
 		time(&current_time);
 		if ((int)(current_time - reaper_start_time) > max_check_reaper_time) {
 			log_debug_info(DEBUGL_CHECKS, 0, "Breaking out of check result reaper: max reaper time exceeded\n");
+			logit(NSLOG_RUNTIME_WARNING, TRUE, "Warning: Breaking out of check result reaper: max reaper time (%d) exceeded. Reaped %d results, but more checkresults to process. Perhaps check core performance tuning tips?\n", max_check_reaper_time, reaped_checks);
 			break;
 		}