Add rule to collect issues with the pods

[avlitman]

What this PR does

Before this PR:
We currently have rules that calculate the available memory for all pods per container.
After this PR:
This pr adds rule that calculate issues with the pods like the memory of the pod with the highest memory exception value by container based on working set/rss memory (in bytes).

For example, if the virt-api container has 2 pods, one of which has an exception rate of 1000 and the other has -2000: the rule will show 1000.
The rules added is kubevirt_memory_delta_from_requested_bytes which includes both the working_set and rss memory minus the requested memory, see:

**cnv_abnormal added to hco repo, see pr: kubevirt/hyperconverged-cluster-operator#2855

Fixes #
Jira-ticket: https://issues.redhat.com/browse/CNV-39598

Why we need it and why it was done in this way

this pr is the next step after deprecating the KubeVirtComponentExceedsRequestedCPU and KubeVirtComponentExceedsRequestedMemory alerts: kubevirt/monitoring#222.
we need it in order to still track memory exceeded issues and other issues in the future without making noise (like the alerts did)

Links to places where the discussion took place:

The rules calculation discussed widely with cnv maintainers.

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note

New memory statistics added named kubevirt_memory_delta_from_requested_bytes

[avlitman]

@enp0s3 @iholder101 Will appreciate your review (:

[iholder101]

@enp0s3 @iholder101 Will appreciate your review (:

Please tag someone that's more familiar with monitoring for an initial review.
Since @sradco and @machadovilaca are going to be SIG-obersvability's approver/reviewer, they might be a good fit.

[simonpasquier]

It's not clear to me what you're going to do with the collected metrics. Will you use them to fine-tune the default memory requests? What's the impact for users if requests << usage?

[sradco]

/lgtm

[avlitman]

/unhold

[machadovilaca]

/lgtm

[machadovilaca]

/lgtm

[enp0s3]

/approve

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enp0s3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [enp0s3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubevirt-commenter-bot]

Required labels detected, running phase 2 presubmits:
/test pull-kubevirt-e2e-windows2016
/test pull-kubevirt-e2e-kind-1.27-vgpu
/test pull-kubevirt-e2e-kind-sriov
/test pull-kubevirt-e2e-k8s-1.29-ipv6-sig-network
/test pull-kubevirt-e2e-k8s-1.27-sig-network
/test pull-kubevirt-e2e-k8s-1.27-sig-storage
/test pull-kubevirt-e2e-k8s-1.27-sig-compute
/test pull-kubevirt-e2e-k8s-1.27-sig-operator
/test pull-kubevirt-e2e-k8s-1.28-sig-network
/test pull-kubevirt-e2e-k8s-1.28-sig-storage
/test pull-kubevirt-e2e-k8s-1.28-sig-compute
/test pull-kubevirt-e2e-k8s-1.28-sig-operator

[kubevirt-bot]

@avlitman: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubevirt-e2e-k8s-1.27-sig-performance	007b9e2	link	true	/test pull-kubevirt-e2e-k8s-1.27-sig-performance
pull-kubevirt-conformance-arm64	8ed18b4	link	false	/test pull-kubevirt-conformance-arm64

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[avlitman]

/cherry-pick release-1.2

[kubevirt-bot]

@avlitman: once the present PR merges, I will cherry-pick it on top of release-1.2 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kubevirt-bot]

@avlitman: new pull request created: #11703

In response to this:

/cherry-pick release-1.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.