Add an alert for excessive migrations on a VMI

[orenc1]

If a VMI has been successfully migrated more than 12 times over a period of 24 hours, which is considerably higher than normal operation including an upgrade, an alert with a severity of warning is being fired.

Also, fixing some typos.

Signed-off-by: orenc1 ocohen@redhat.com

What this PR does / why we need it:
Excessive number of migrations on a single VMI over a period of time can suggest issues related to cluster configuration or available resources. Firing an alert if excessive migrations have been found on a VMI.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Release note:

new alert for excessive number of VMI migrations in a period of time.

[orenc1]

/cc @sradco

[orenc1]

Runbook PR:
kubevirt/monitoring#85

[orenc1]

/test pull-kubevirt-unit-test

[Barakmor1]

Hey Oren,
Thanks for the PR.
I tried to use the new alert but i think that it doesn't work as expected:
i created a vm and migrated it two times , these are the results that i got from the new query:

Note that the result of sum by(vmi) (increase(kubevirt_migrate_vmi_succeeded_total[1d])) is 0
maybe you get the correct number after 24h .
Please let me know if i missed something.

[orenc1]

Hi @Barakmor1 ,
thanks for noticing.
the problem was that the query treated to migrations of the same VMI, but with different source/target as separate results, thus the migrations on a specific VMI were not counted correctly.
This is the fixed expression I'm using now:

floor(increase(sum by (vmi) (kubevirt_migrate_vmi_succeeded_total)[1d:1m])) >= 12

it is summing up migrations per VMI, regardless of other labels, then calculates their increase over a period of time (24 hours) with a resolution of 1 minute.

[Barakmor1]

Thanks!
/lgtm

[sradco]

@orenc1 Its 8 increases in total no?

[sradco]

Also, why do you consider the first value as 0 time? I think its the value of the first hour.

[sradco]

That is an important detail and this is why the unit testing can become biased.

[orenc1]

no, because the first one doesn't count (change from _ to 1).
but it does count on the second series, since there is already a datapoint for kubevirt_migrate_vmi_succeeded_total{vmi="vmi-example-1"}

[enp0s3]

@orenc1 @sradco @assafad Hi. IIUC our policy regarding alerts addition is that a runbook should be first created for it in the monitoring repo, otherwise the PR will fail the monitoring lane. Either I don't access the KubeVirtVMIExcessiveMigrations correctly or it doesn't exist. For the latter case lets first add the runbook and then continue with this PR.

[enp0s3]

Waiting for the runbook PR to be merged

[orenc1]

hi @enp0s3 , the runbook PR has been merged.

[enp0s3]

/approve

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enp0s3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [enp0s3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[orenc1]

/retest-required

[kubevirt-commenter-bot]

/retest-required
This bot automatically retries required jobs that failed/flaked on approved PRs.
Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[kubevirt-commenter-bot]

/retest-required
This bot automatically retries required jobs that failed/flaked on approved PRs.
Silence the bot with an /lgtm cancel or /hold comment for consistent failures.