Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Checkable#ProcessCheckResult(): don't overwrite check source #8158

Merged
merged 1 commit into from Jan 26, 2021

Conversation

Al2Klimov
Copy link
Member

@Al2Klimov Al2Klimov commented Aug 7, 2020

... set by passive check results.

fixes #7948

@Al2Klimov Al2Klimov added bug Something isn't working area/checks Check execution and results labels Aug 7, 2020
@Al2Klimov Al2Klimov added this to the 2.13.0 milestone Aug 7, 2020
@Al2Klimov Al2Klimov added this to To review in v2.13.0 merge window via automation Aug 7, 2020
@Al2Klimov
Copy link
Member Author

Before

➜  icinga2 git:(master) cat prefix/etc/icinga2/conf.d/api-users.conf
/**
 * The ApiUser objects are used for authentication against the API.
 */
object ApiUser "root" {
  password = "fcd80364c10e2de3"
  // client_cn = ""

  permissions = [ "*" ]
}
➜  icinga2 git:(master) prefix/sbin/icinga2 daemon -d
[2020-08-07 14:20:31 +0200] information/cli: Icinga application loader (version: v2.12.0)
[2020-08-07 14:20:31 +0200] information/cli: Closing console log.
➜  icinga2 git:(master) curl -k -s -u root:fcd80364c10e2de3 -H 'Accept: application/json' 'https://localhost:5665/v1/actions/process-check-result?pretty=1' -X POST -d '{"type":"Host","filter":"true","exit_status":0,"plugin_output":"OK","check_source":"lolcat"}' && curl -k -s -u root:fcd80364c10e2de3 -H 'Accept: application/json' 'https://localhost:5665/v1/objects/hosts?pretty=1'
{
    "results": [
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'alexanders-mbp.int.netways.de'."
        }
    ]
}{
    "results": [
        {
            "attrs": {
                "__name": "alexanders-mbp.int.netways.de",
                "acknowledgement": 0.0,
                "acknowledgement_expiry": 0.0,
                "acknowledgement_last_change": 0.0,
                "action_url": "",
                "active": true,
                "address": "127.0.0.1",
                "address6": "::1",
                "check_attempt": 1.0,
                "check_command": "hostalive",
                "check_interval": 60.0,
                "check_period": "",
                "check_timeout": null,
                "command_endpoint": "",
                "display_name": "alexanders-mbp.int.netways.de",
                "downtime_depth": 0.0,
                "enable_active_checks": true,
                "enable_event_handler": true,
                "enable_flapping": false,
                "enable_notifications": true,
                "enable_passive_checks": true,
                "enable_perfdata": true,
                "event_command": "",
                "flapping": false,
                "flapping_current": 5.9,
                "flapping_last_change": 0.0,
                "flapping_threshold": 0.0,
                "flapping_threshold_high": 30.0,
                "flapping_threshold_low": 25.0,
                "force_next_check": false,
                "force_next_notification": false,
                "groups": [
                    "linux-servers"
                ],
                "ha_mode": 0.0,
                "handled": false,
                "icon_image": "",
                "icon_image_alt": "",
                "last_check": 1596803032.767611,
                "last_check_result": {
                    "active": false,
                    "check_source": "alexanders-mbp.int.netways.de",
                    "command": null,
                    "execution_end": 1596803032.767611,
                    "execution_start": 1596803032.767611,
                    "exit_status": 0.0,
                    "output": "OK",
...

After

➜  icinga2 git:(bugfix/check-source-passive-7948) cat prefix/etc/icinga2/conf.d/api-users.conf
/**
 * The ApiUser objects are used for authentication against the API.
 */
object ApiUser "root" {
  password = "f25bdbfeef667b42"
  // client_cn = ""

  permissions = [ "*" ]
}
➜  icinga2 git:(bugfix/check-source-passive-7948) prefix/sbin/icinga2 daemon -d
[2020-08-07 14:30:05 +0200] information/cli: Icinga application loader (version: v2.12.0-1-gaeb0a0021)
[2020-08-07 14:30:05 +0200] information/cli: Closing console log.
➜  icinga2 git:(bugfix/check-source-passive-7948) curl -k -s -u root:f25bdbfeef667b42 -H 'Accept: application/json' 'https://localhost:5665/v1/actions/process-check-result?pretty=1' -X POST -d '{"type":"Host","filter":"true","exit_status":0,"plugin_output":"OK","check_source":"lolcat"}' && curl -k -s -u root:f25bdbfeef667b42 -H 'Accept: application/json' 'https://localhost:5665/v1/objects/hosts?pretty=1'
{
    "results": [
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'alexanders-mbp.int.netways.de'."
        }
    ]
}{
    "results": [
        {
            "attrs": {
                "__name": "alexanders-mbp.int.netways.de",
                "acknowledgement": 0.0,
                "acknowledgement_expiry": 0.0,
                "acknowledgement_last_change": 0.0,
                "action_url": "",
                "active": true,
                "address": "127.0.0.1",
                "address6": "::1",
                "check_attempt": 1.0,
                "check_command": "hostalive",
                "check_interval": 60.0,
                "check_period": "",
                "check_timeout": null,
                "command_endpoint": "",
                "display_name": "alexanders-mbp.int.netways.de",
                "downtime_depth": 0.0,
                "enable_active_checks": true,
                "enable_event_handler": true,
                "enable_flapping": false,
                "enable_notifications": true,
                "enable_passive_checks": true,
                "enable_perfdata": true,
                "event_command": "",
                "flapping": false,
                "flapping_current": 5.9,
                "flapping_last_change": 0.0,
                "flapping_threshold": 0.0,
                "flapping_threshold_high": 30.0,
                "flapping_threshold_low": 25.0,
                "force_next_check": false,
                "force_next_notification": false,
                "groups": [
                    "linux-servers"
                ],
                "ha_mode": 0.0,
                "handled": false,
                "icon_image": "",
                "icon_image_alt": "",
                "last_check": 1596803453.403651,
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1596803453.403651,
                    "execution_start": 1596803453.403651,
                    "exit_status": 0.0,
                    "output": "OK",
...

Copy link
Contributor

@julianbrost julianbrost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please also test this with a service? Seems to only work for hosts:

curl -skSu root:icinga -H 'Accept: application/json' 'https://localhost:5665/v1/actions/process-check-result' -d '{ "type": "Service", "filter": "host.name==\"master-1\" && service.name==\"icinga-health\"", "exit_status": 0, "plugin_output": "Hello World", "check_source": "foo", "pretty": true }' && curl -skSu root:icinga -H 'Accept: application/json' 'https://localhost:5665/v1/objects/services/master-1!icinga-health?pretty=1'
{
    "results": [
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'master-1!icinga-health'."
        }
    ]
}
{
    "results": [
        {
            "attrs": {
                "__name": "master-1!icinga-health",
[...]
                "last_check": 1611578640.206652,
                "last_check_result": {
                    "active": false,
                    "check_source": "master-1",
                    "command": null,
                    "execution_end": 1611578640.206652,
                    "execution_start": 1611578640.206652,
                    "exit_status": 0.0,
                    "output": "Hello World",
                    "performance_data": [],
                    "schedule_end": 1611578640.206652,
                    "schedule_start": 1611578640.206652,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": true,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": true,
                        "state": 0.0,
                        "state_type": 1.0
                    }
                },
[...]
            "name": "master-1!icinga-health",
            "type": "Service"
        }
    ]
}

v2.13.0 merge window automation moved this from To review to Changes requested Jan 25, 2021
@Al2Klimov
Copy link
Member Author

➜  icinga2 git:(bugfix/check-source-passive-7948) prefix/sbin/icinga2 api setup -x critical
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
Done.

Now restart your Icinga 2 daemon to finish the installation!

➜  icinga2 git:(bugfix/check-source-passive-7948) perl -pi -e 's/\bpassword = ".*?"/password = "123456"/' prefix/etc/icinga2/conf.d/api-users.conf
➜  icinga2 git:(bugfix/check-source-passive-7948) prefix/sbin/icinga2 daemon -d
[2021-01-25 14:02:20 +0100] information/cli: Icinga application loader (version: v2.12.0-478-g4e1778040)
[2021-01-25 14:02:20 +0100] information/cli: Closing console log.
➜  icinga2 git:(bugfix/check-source-passive-7948) curl -k -s -u root:123456 -H 'Accept: application/json' 'https://localhost:5665/v1/actions/process-check-result?pretty=1' -X POST -d '{"type":"Service","filter":"true","exit_status":0,"plugin_output":"OK","check_source":"lolcat"}' && curl -k -s -u root:123456 -H 'Accept: application/json' 'https://localhost:5665/v1/objects/services?pretty=1&attrs=last_check_result'
{
    "results": [
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!users'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!procs'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!ping6'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!swap'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!disk'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!load'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!ping4'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!icinga'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!ssh'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!http'."
        },
        {
            "code": 200.0,
            "status": "Successfully processed check result for object 'Alexanders-MacBook-Pro.local!disk /'."
        }
    ]
}
{
    "results": [
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.013045,
                    "execution_start": 1611579922.013045,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.013045,
                    "schedule_start": 1611579922.013045,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!users",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.013337,
                    "execution_start": 1611579922.013337,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.013337,
                    "schedule_start": 1611579922.013337,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!procs",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.013512,
                    "execution_start": 1611579922.013512,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.013512,
                    "schedule_start": 1611579922.013512,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!ping6",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.013671,
                    "execution_start": 1611579922.013671,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.013671,
                    "schedule_start": 1611579922.013671,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!swap",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.013812,
                    "execution_start": 1611579922.013812,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.013812,
                    "schedule_start": 1611579922.013812,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!disk",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.014027,
                    "execution_start": 1611579922.014027,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.014027,
                    "schedule_start": 1611579922.014027,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!load",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.014174,
                    "execution_start": 1611579922.014174,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.014174,
                    "schedule_start": 1611579922.014174,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!ping4",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.014307,
                    "execution_start": 1611579922.014307,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.014307,
                    "schedule_start": 1611579922.014307,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!icinga",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.014459,
                    "execution_start": 1611579922.014459,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.014459,
                    "schedule_start": 1611579922.014459,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!ssh",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.014698,
                    "execution_start": 1611579922.014698,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.014698,
                    "schedule_start": 1611579922.014698,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!http",
            "type": "Service"
        },
        {
            "attrs": {
                "last_check_result": {
                    "active": false,
                    "check_source": "lolcat",
                    "command": null,
                    "execution_end": 1611579922.014888,
                    "execution_start": 1611579922.014888,
                    "exit_status": 0.0,
                    "output": "OK",
                    "performance_data": null,
                    "schedule_end": 1611579922.014888,
                    "schedule_start": 1611579922.014888,
                    "state": 0.0,
                    "ttl": 0.0,
                    "type": "CheckResult",
                    "vars_after": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 0.0,
                        "state_type": 1.0
                    },
                    "vars_before": {
                        "attempt": 1.0,
                        "reachable": false,
                        "state": 3.0,
                        "state_type": 1.0
                    }
                }
            },
            "joins": {},
            "meta": {},
            "name": "Alexanders-MacBook-Pro.local!disk /",
            "type": "Service"
        }
    ]
}
➜  icinga2 git:(bugfix/check-source-passive-7948)

@julianbrost
Copy link
Contributor

Ok, looks like the issue is not necessarily host vs. service. But something strange is going on here. 1843 service results are processed and all of them show the correct output in the API (freshly chosen for that command), however only 1730 show the proper check source:

$ curl -iskSu root:icinga -H 'Accept: application/json' 'https://localhost:5665/v1/actions/process-check-result' -d '{ "type": "Service", "filter": "true", "exit_status": 0, "plugin_output": "324093284", "check_source": "foo", "pretty": true }' | grep code | sort | uniq -c && curl -iskSu root:icinga -H 'Accept: application/json' 'https://localhost:5665/v1/objects/services?pretty=1&attrs=last_check_result' | grep -e Server -e check_source -e output | sort | uniq -c
   1843             "code": 200.0,
      6                     "check_source": "agent-1",
      6                     "check_source": "agent-2",
      6                     "check_source": "agent-a-1",
      6                     "check_source": "agent-a-2",
      6                     "check_source": "agent-b-1",
      6                     "check_source": "agent-b-2",
      7                     "check_source": "agent-b-a-1",
      6                     "check_source": "agent-b-a-2",
      6                     "check_source": "agent-win",
   1730                     "check_source": "foo",
     16                     "check_source": "master-1",
      6                     "check_source": "master-2",
      6                     "check_source": "satellite-a-1",
      6                     "check_source": "satellite-a-2",
      6                     "check_source": "satellite-b-1",
      6                     "check_source": "satellite-b-2",
      6                     "check_source": "satellite-b-a-1",
      6                     "check_source": "satellite-b-a-2",
   1843                     "output": "324093284",
      1 Server: Icinga/v2.12.0-392-ga8d2983bb

@julianbrost
Copy link
Contributor

Looks like the pattern could be that it doesn't work for services created by apply rules.

@Al2Klimov
Copy link
Member Author

All of my services are created via apply rules.

Have you tried to rebase locally?

@julianbrost
Copy link
Contributor

No, I've been testing on the PR branch as is. But I found out what happened: it's not apply rules but the command endpoint (there's just significant overlap between the two in my test setup). For these, the check source will still be overwritten just a few lines later:

/* override check source if command_endpoint was defined */
if (command_endpoint && !GetExtension("agent_check"))
cr->SetCheckSource(command_endpoint->GetName());

So maybe just put both of these ifs into a new if (cr->GetCheckSource().IsEmpty())?

@Al2Klimov Al2Klimov force-pushed the bugfix/check-source-passive-7948 branch from a8d2983 to c3eba7e Compare January 25, 2021 15:06
v2.13.0 merge window automation moved this from Changes requested to Approved Jan 26, 2021
@julianbrost julianbrost merged commit 9219f68 into master Jan 26, 2021
v2.13.0 merge window automation moved this from Approved to Merged Jan 26, 2021
@icinga-probot icinga-probot bot deleted the bugfix/check-source-passive-7948 branch January 26, 2021 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/checks Check execution and results bug Something isn't working
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

Unable to set check source for passive service via API command check-process-result
2 participants