Add guest-to-request memory headroom ratio #9322

iholder101 · 2023-02-27T09:08:33Z

What this PR does / why we need it:
Kubevirt computes [1] the estimation of the total memory overhead needed for infra components inside virt-launcher which are responsible to run the guest (e.g. libvirtd, qemu, etc).

Not only that this overhead currently suffers from known issues and non-accurate calculation which needs to be fixed - this calculation is in essence an educated guess / estimation, and not an accurate calculation. The reason is that even if a careful profiling will take place (which is a very difficult task to do, since the environments on which we would profile makes the results biased), there are still many components we cannot control, e.g. kernel drivers, kernel configuration, inner QEMU buffer allocations, etc.

To solve this problem, we need to both keep improving the overhead estimations, but also provide a solution for the cluster admin to explicitly add some overhead. This is useful in cases where a special configuration is used, or if the cluster-admin chooses to reduce the risk of workloads OOM killed during node pressures in exchange of workloads consuming more memory (therefore less VMIs could be scheduled on a node).

[1] https://github.com/kubevirt/kubevirt/blob/v0.59.0-alpha.2/pkg/virt-controller/services/renderresources.go#L272

Which issue(s) this PR fixes:
Fixes:
https://bugzilla.redhat.com/show_bug.cgi?id=2165618
https://bugzilla.redhat.com/show_bug.cgi?id=2164593

Special notes for your reviewer:
For a reference, you can look at the temporary solution implemented in HCO: kubevirt/hyperconverged-cluster-operator#2206

Release note:

Add guest-to-request memory headroom ratio.
This can be enabled by setting `kubevirt.spec.configuration.additionalGuestMemoryOverheadRatio = "1.234"`

iholder101 · 2023-02-27T09:09:03Z

/cc @acardace

iholder101 · 2023-02-27T09:13:29Z

/cc @vladikr @fabiand

fabiand · 2023-02-27T09:23:07Z

api/openapi-spec/swagger.json

@@ -16853,6 +16853,10 @@
      "description": "EvictionStrategy defines at the cluster level if the VirtualMachineInstance should be migrated instead of shut-off in case of a node drain. If the VirtualMachineInstance specific field is set it overrides the cluster level one.",
      "type": "string"
     },
+     "guestToRequestMemoryHeadroomRatio": {
+      "description": "GuestToRequestMemoryHeadroomRatio can be used to set a ratio between VM's memory guest and the memory allocated to the compute container. A higher ratio means that the VMs would be less compromised by node pressures, but would mean that fewer VMs could be scheduled to a node. If not set, the default is 1.",


Please add a note if the internal overhead calculations are still taken into account or not (ignored)

Will do.
FYI: they are.

vladikr

Thanks for this PR @iholder101

I think we do need such an additional tunable for the admin to increase the VM calculated overhead in the cluster.

That said, I would expect the additional calculated "headroom" simply be part of the GetMemoryOverhead function. That way the overhead calculation will stay consistent, without the need to separately update the requests and limits.
When the requests/limits are calculated, virt-controller adds the overhead to it, it could also add the additional admin-provided overhead at the same step.
This would also work for the case when the guest overhead should be overcommitted: using the OvercommitGuestOverhead.

I'm also not sure about the guestToRequestMemoryHeadroomRatio name.
When I think about the main purpose of this tunable, I think that it is to allow the admin to provide an additional overhead - knowing that our overhead calculation is "best effort".
Would it make sense to rename it to something like additinalGuestOverhead or increaseFuestOverheadRatio?

kubevirt/pkg/virt-controller/services/renderresources.go

Line 272 in ed8b899

    
           func GetMemoryOverhead(vmi *v1.VirtualMachineInstance, cpuArch string) *resource.Quantity {

iholder101 · 2023-03-01T13:06:07Z

Hey @vladikr, thanks for your review!

Thanks for this PR @iholder101

I think we do need such an additional tunable for the admin to increase the VM calculated overhead in the cluster.

That said, I would expect the additional calculated "headroom" simply be part of the GetMemoryOverhead function. That way the overhead calculation will stay consistent, without the need to separately update the requests and limits. When the requests/limits are calculated, virt-controller adds the overhead to it, it could also add the additional admin-provided overhead at the same step. This would also work for the case when the guest overhead should be overcommitted: using the OvercommitGuestOverhead.

That's a very good point! Thank you!
I've made the required changed :)

I just want to note that now this field controls the overhead itself only, not the ratio between the guest memory and requests. IOW, the previous implementation did requests = (<guest-memory> + <overhead>)*<ratio> while the current implementation does requests = <guest-memory> + <overhead>*<ratio>. I actually think it's much better this way.

I'm also not sure about the guestToRequestMemoryHeadroomRatio name. When I think about the main purpose of this tunable, I think that it is to allow the admin to provide an additional overhead - knowing that our overhead calculation is "best effort". Would it make sense to rename it to something like additinalGuestOverhead or increaseFuestOverheadRatio?

kubevirt/pkg/virt-controller/services/renderresources.go

Line 272 in ed8b899

func GetMemoryOverhead(vmi *v1.VirtualMachineInstance, cpuArch string) *resource.Quantity {

We can discuss the name.
The reason I don't like names like you suggested is that the headroom never actually reaches the guest itself. IMO something like additinalGuestOverhead implies that the overhead affects the guest somehow, which is somewhat misleading.

WDYT about something like additinalVirtInfraOverhead?

iholder101 · 2023-03-01T13:42:49Z

Unit tests pass locally
/test pull-kubevirt-unit-test

vladikr · 2023-03-01T17:53:34Z

pkg/virt-controller/services/template.go

@@ -1304,7 +1304,7 @@ func checkForKeepLauncherAfterFailure(vmi *v1.VirtualMachineInstance) bool {
 }

 func (t *templateService) VMIResourcePredicates(vmi *v1.VirtualMachineInstance, networkToResourceMap map[string]string) VMIResourcePredicates {
-	memoryOverhead := GetMemoryOverhead(vmi, t.clusterConfig.GetClusterCPUArch())
+	memoryOverhead := GetMemoryOverhead(vmi, t.clusterConfig.GetClusterCPUArch(), t.clusterConfig.GetConfig().GuestToRequestMemoryHeadroomRatio)


I wish there was an easy way to simply get the config from within the GetMemoryOverhead function...

Yeah, it was a bit painful to write..
I'm not sure how we can improve this in a straight forward way, as this function is being called from different containers. We can always fetch Kubevirt with an API call, but that would be costly in terms of performance.

vladikr · 2023-03-01T21:49:19Z

Hey @vladikr, thanks for your review!

Thanks for this PR @iholder101
I think we do need such an additional tunable for the admin to increase the VM calculated overhead in the cluster.
That said, I would expect the additional calculated "headroom" simply be part of the GetMemoryOverhead function. That way the overhead calculation will stay consistent, without the need to separately update the requests and limits. When the requests/limits are calculated, virt-controller adds the overhead to it, it could also add the additional admin-provided overhead at the same step. This would also work for the case when the guest overhead should be overcommitted: using the OvercommitGuestOverhead.

That's a very good point! Thank you! I've made the required changed :)

👍

I just want to note that now this field controls the overhead itself only, not the ratio between the guest memory and requests. IOW, the previous implementation did requests = (<guest-memory> + <overhead>)*<ratio> while the current implementation does requests = <guest-memory> + <overhead>*<ratio>. I actually think it's much better this way.

I'm also not sure about the guestToRequestMemoryHeadroomRatio name. When I think about the main purpose of this tunable, I think that it is to allow the admin to provide an additional overhead - knowing that our overhead calculation is "best effort". Would it make sense to rename it to something like additinalGuestOverhead or increaseFuestOverheadRatio?

kubevirt/pkg/virt-controller/services/renderresources.go

Line 272 in ed8b899

func GetMemoryOverhead(vmi *v1.VirtualMachineInstance, cpuArch string) *resource.Quantity {

We can discuss the name. The reason I don't like names like you suggested is that the headroom never actually reaches the guest itself. IMO something like additinalGuestOverhead implies that the overhead affects the guest somehow, which is somewhat misleading.

WDYT about something like additinalVirtInfraOverhead?

I understand your point. I didn't actually read it this way so far. To me "Guest Overhead" on the Virtual Machine Instance reads as an overhead caused by the guest...
My main concern with names is consistency. We already use the term "Guest Overhead" in many places including other tunables OvercommitGuestOverhead [1].
If you prefer not to use this term, then we should find something similar that is not confusing.

Does it make sense?

[1] https://kubevirt.io/user-guide/operations/node_overcommit/#overcommit-the-guest-overhead

iholder101 · 2023-03-02T11:51:32Z

@vladikr changed the name to AdditinalGuestOverheadRatio
PTAL

fabiand · 2023-03-02T12:48:53Z

api/openapi-spec/swagger.json

@@ -16825,6 +16825,10 @@
    "description": "KubeVirtConfiguration holds all kubevirt configurations",
    "type": "object",
    "properties": {
+     "additinalGuestOverheadRatio": {


please be more specific with the name
I think this is only calculating the memory, thus
additinalGuestMemoryOverheadRatio could be an option

@vladikr and I discussed the name in comments above.
At least to me, it feels a bit misleading that we mention "guest memory", as the guest memory itself is not affected in any way. @vladikr claimed that this is the naming we chose for many other fields, so I guess it's a topic for further refactoring.

I suggested the name additinalVirtInfraOverhead above, as it specifically refers to the virt infra overhead - not to the guest memory.

Please share your thoughts :)

I'd definetly include memory - if it is memory specific.

Changed to additinalGuestMemoryOverheadRatio.

AdditionalGuestMemoryOverheadRatio can be used to set a ratio between VM's memory guest and the memory allocated to the compute container. A higher ratio means that the VMs would be less compromised by node pressures, but would mean that fewer VMs could be scheduled to a node. If not set, the default is 1. Signed-off-by: Itamar Holder <iholder@redhat.com>

Signed-off-by: Itamar Holder <iholder@redhat.com>

This is better both in terms of performance and in terms of safety. Signed-off-by: Itamar Holder <iholder@redhat.com>

Signed-off-by: Itamar Holder <iholder@redhat.com>

vladikr · 2023-03-02T13:56:01Z

Thanks!

/approve

kubevirt-bot · 2023-03-02T13:56:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vladikr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vladikr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubevirt-bot · 2023-03-02T15:06:22Z

@iholder101: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubevirt-fossa	`38743f1`	link	false	`/test pull-kubevirt-fossa`
pull-kubevirt-check-tests-for-flakes	`38743f1`	link	false	`/test pull-kubevirt-check-tests-for-flakes`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

iholder101 · 2023-03-02T15:38:51Z

/test pull-kubevirt-manifests
/test pull-kubevirt-e2e-k8s-1.26-sig-storage-cgroupsv2

acardace · 2023-03-02T17:02:34Z

/lgtm

acardace · 2023-03-02T17:11:51Z

/cherrypick release-0.59

kubevirt-bot · 2023-03-02T17:11:53Z

@acardace: once the present PR merges, I will cherry-pick it on top of release-0.59 in a new PR and assign it to you.

In response to this:

/cherrypick release-0.59

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubevirt-commenter-bot · 2023-03-02T20:14:56Z

/retest-required
This bot automatically retries required jobs that failed/flaked on approved PRs.
Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

kubevirt-bot · 2023-03-02T22:59:25Z

@acardace: #9322 failed to apply on top of branch "release-0.59":

Applying: Add AdditionalGuestMemoryOverheadRatio to Kubevirt CR
Applying: Add guest-to-request memory headroom to compute containers
Using index info to reconstruct a base tree...
M	pkg/virt-controller/services/renderresources.go
M	pkg/virt-handler/vm.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/virt-handler/vm.go
Auto-merging pkg/virt-controller/services/renderresources.go
CONFLICT (content): Merge conflict in pkg/virt-controller/services/renderresources.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 Add guest-to-request memory headroom to compute containers
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherrypick release-0.59

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tghfly · 2023-06-05T14:32:11Z

How to validate the parameter? I modified CR and set “additionalGuestMemoryOverheadRatio: 1.3", but it still encounters OOM. Can you provide an example YAML file?

iholder101 · 2023-06-05T14:40:59Z

How to validate the parameter? I modified CR and set “additionalGuestMemoryOverheadRatio: 1.3", but it still encounters OOM. Can you provide an example YAML file?

Modifying kubevirt.spec.configuration.additionalGuestMemoryOverheadRatio is exactly what needs to be done.

Perhaps 1.3 is not enough for your case?

Some guiding questions:

How much memory is allocated for the VM (i.e. from the guest perspective)?
How much memory is allocated to the compute container in virt-launcher Pod?
When the container was OOMed - how much memory did it consume?

You can always try setting a higher value and see if it solves it for your case, but I'd recommend to answer the questions above to ensure your get the right values.

In addition, keep in mind that the multiplier (e.g. 1.3) does not multiply the memory that's configured for the VM, but multiplies the virt-infra memory that's allocated automatically by Kubevirt. For example, let's say that you define a VM with 2GB of memory, and let's say that Kubevirt automatically adds 200M for virt infra. In this case the total memory of the compute container is 2.2G. If you set additionalGuestMemoryOverheadRatio to be 1.3, that applies only to the virt infra, so now the container's memory would be: 2GB+1.3*200M = 2GB+260M=2.26G.

tghfly · 2023-06-06T04:55:47Z

vm.yaml

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm01
  namespace: default
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/size: small
        kubevirt.io/domain: vm01
      annotations:
        ovn.kubernetes.io/allow_live_migration: "true"
        ovn.kubernetes.io/logical_switch: provider
        ovn.kubernetes.io/ip_address: 192.168.1.2
    spec:
      nodeSelector:
        kubernetes.io/hostname: k8s-01
      dnsConfig:
        nameservers:
        - 192.168.1.250
      dnsPolicy: "None"
      architecture: amd64
      domain:     
        cpu:
          cores: 4
          model: host-passthrough
        #memory:
        #  guest: 8000Mi
        devices:
          disks:
            - name: root-disk
              disk:
                bus: virtio
              cache: writeback
          interfaces:
            - name: default
              bridge: {}
              model: virtio
          rng: {}
        machine:
          type: q35
        resources:
          limits:
            cpu: 4 
            memory: 8Gi
          requests:
            cpu: 4
            memory: 8Gi
      networks:
        - name: default
          pod: {}
      volumes:
        - name: root-disk
          persistentVolumeClaim:
            claimName: pvc-win10-bootdisk

The vm has been allocated 8 GiB of memory(see on windows10 system ).
The virt-launcher pod has been allocated 8.36 GiB of memory.
virt-launcher pods

 Limits:
      cpu:                            4
      devices.kubevirt.io/kvm:        1
      devices.kubevirt.io/tun:        1
      devices.kubevirt.io/vhost-net:  1
      memory:                         8974342024
    Requests:
      cpu:                            4
      devices.kubevirt.io/kvm:        1
      devices.kubevirt.io/tun:        1
      devices.kubevirt.io/vhost-net:  1
      ephemeral-storage:              50M
      memory:                         8974342024

When OOM occurs, the memory is around 8.36 GiB（watch -n 1 kubectl top pod）.
In addition，Where can I check how much memory is allocated for virt infra, such as 200M? how to calculate?

iholder101 · 2023-06-07T06:29:52Z

@tghfly I see you set memory limits == requests. Can you explain why do you set memory limits? Memory limits are dangerous, and I wouldn't define them unless there's a good reason to do so, as if you exceed the limit the Pod is going to be OOM killed.

BTW, another tip: it's recommended to use spec.domain.memory.guest, and remove spec.domain.resources entirely. In other words: determine the guest's memory and let Kubevirt determine the exact resources of the pod for you.

After you remove the limits, I'd try to see if the virt-launcher pod consumes more memory than it requests. On most cases the answer would be no, but if the answer is yes, try setting additionalGuestMemoryOverheadRatio to raise the requested amount.

Hope this helps. Good luck!

tghfly · 2023-06-07T13:13:17Z

@iholder101 Thanks，We use "limits == requests" to ensure Guaranteed QoS. And in the NUMA binding and CPU Pin
scenarios, it is also necessary to set “limits == requests”.

iholder101 · 2023-06-07T13:18:27Z

@iholder101 Thanks，We use "limits == requests" to ensure Guaranteed QoS. And in the NUMA binding and CPU Pin scenarios, it is also necessary to set “limits == requests”.

Fair enough. You're right, a Guaranteed QoS is necessary sometimes.
But as I've said, having limits is dangerous and you have to stand by the values you define.

You can either set a different spec.domain.memory.guest and spec.domain.resources.memory.request values to manually define the virt overhead, or play with additionalGuestMemoryOverheadRatio until you have enough memory to not be killed.

tghfly · 2023-06-07T13:39:06Z

Yes, we are currently planning to do so. Thank you again for your patient explanation.

iholder101 · 2023-06-07T14:06:47Z

Yes, we are currently planning to do so. Thank you again for your patient explanation.

Sure thing! Good luck!

xpivarc · 2023-09-13T14:34:02Z

@iholder101 Thanks，We use "limits == requests" to ensure Guaranteed QoS. And in the NUMA binding and CPU Pin scenarios, it is also necessary to set “limits == requests”.

@tghfly I think it would be better if you use DedicatedCPUPlacement as this will ensure other containers are in Guaranteed QoS as well and the limits/requests will be set for you

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Feb 27, 2023

kubevirt-bot requested review from kbidarkar and rmohr February 27, 2023 09:08

kubevirt-bot added size/L kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Feb 27, 2023

kubevirt-bot requested a review from acardace February 27, 2023 09:09

iholder101 force-pushed the feature/guest-to-request-memory-headroom-ratio branch from bd64a18 to 4b28a47 Compare February 27, 2023 09:12

kubevirt-bot requested review from fabiand and vladikr February 27, 2023 09:13

fabiand reviewed Feb 27, 2023

View reviewed changes

iholder101 force-pushed the feature/guest-to-request-memory-headroom-ratio branch 2 times, most recently from 3bd2539 to ed8b899 Compare February 27, 2023 11:24

vladikr reviewed Mar 1, 2023

View reviewed changes

iholder101 force-pushed the feature/guest-to-request-memory-headroom-ratio branch from ed8b899 to eda7381 Compare March 1, 2023 13:00

vladikr reviewed Mar 1, 2023

View reviewed changes

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2023

iholder101 mentioned this pull request Mar 2, 2023

Kubevirt: Add guest-to-request memory headroom to virt-launchers kubevirt/hyperconverged-cluster-operator#2245

Closed

11 tasks

iholder101 force-pushed the feature/guest-to-request-memory-headroom-ratio branch 2 times, most recently from df95da1 to 950213e Compare March 2, 2023 11:40

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 2, 2023

fabiand reviewed Mar 2, 2023

View reviewed changes

iholder101 added 2 commits March 2, 2023 15:14

Add guest-to-request memory headroom to compute containers

33f5ddf

Signed-off-by: Itamar Holder <iholder@redhat.com>

iholder101 added 4 commits March 2, 2023 15:15

Refactor: return memory overhead by-value

65a49f0

This is better both in terms of performance and in terms of safety. Signed-off-by: Itamar Holder <iholder@redhat.com>

Validating Webhooks: validate guest-to-headroom ratio is larger than 1

344e1b5

Signed-off-by: Itamar Holder <iholder@redhat.com>

Unit tests: guest-to-request memory headroom

92550a7

Signed-off-by: Itamar Holder <iholder@redhat.com>

Functional tests: guest-to-request memory headroom

38743f1

Signed-off-by: Itamar Holder <iholder@redhat.com>

iholder101 force-pushed the feature/guest-to-request-memory-headroom-ratio branch from 950213e to 38743f1 Compare March 2, 2023 13:21

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 2, 2023

kubevirt-bot assigned acardace Mar 2, 2023

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 2, 2023

kubevirt-bot merged commit 40049bc into kubevirt:main Mar 2, 2023

iholder101 mentioned this pull request Mar 5, 2023

[release-0.59] Add guest-to-request memory headroom ratio #9362

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add guest-to-request memory headroom ratio #9322

Add guest-to-request memory headroom ratio #9322

iholder101 commented Feb 27, 2023 •

edited

iholder101 commented Feb 27, 2023

iholder101 commented Feb 27, 2023

fabiand Feb 27, 2023

iholder101 Feb 27, 2023

iholder101 Feb 27, 2023

vladikr left a comment

iholder101 commented Mar 1, 2023 •

edited

iholder101 commented Mar 1, 2023

vladikr Mar 1, 2023

iholder101 Mar 2, 2023

vladikr commented Mar 1, 2023

iholder101 commented Mar 2, 2023

fabiand Mar 2, 2023

iholder101 Mar 2, 2023

fabiand Mar 2, 2023

iholder101 Mar 2, 2023

vladikr commented Mar 2, 2023

kubevirt-bot commented Mar 2, 2023

kubevirt-bot commented Mar 2, 2023 •

edited

iholder101 commented Mar 2, 2023

acardace commented Mar 2, 2023

acardace commented Mar 2, 2023

kubevirt-bot commented Mar 2, 2023

kubevirt-commenter-bot commented Mar 2, 2023

kubevirt-bot commented Mar 2, 2023

tghfly commented Jun 5, 2023

iholder101 commented Jun 5, 2023

tghfly commented Jun 6, 2023 •

edited

iholder101 commented Jun 7, 2023 •

edited

tghfly commented Jun 7, 2023

iholder101 commented Jun 7, 2023

tghfly commented Jun 7, 2023

iholder101 commented Jun 7, 2023

xpivarc commented Sep 13, 2023

Add guest-to-request memory headroom ratio #9322

Add guest-to-request memory headroom ratio #9322

Conversation

iholder101 commented Feb 27, 2023 • edited

iholder101 commented Feb 27, 2023

iholder101 commented Feb 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladikr left a comment

Choose a reason for hiding this comment

iholder101 commented Mar 1, 2023 • edited

iholder101 commented Mar 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladikr commented Mar 1, 2023

iholder101 commented Mar 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladikr commented Mar 2, 2023

kubevirt-bot commented Mar 2, 2023

kubevirt-bot commented Mar 2, 2023 • edited

iholder101 commented Mar 2, 2023

acardace commented Mar 2, 2023

acardace commented Mar 2, 2023

kubevirt-bot commented Mar 2, 2023

kubevirt-commenter-bot commented Mar 2, 2023

kubevirt-bot commented Mar 2, 2023

tghfly commented Jun 5, 2023

iholder101 commented Jun 5, 2023

tghfly commented Jun 6, 2023 • edited

iholder101 commented Jun 7, 2023 • edited

tghfly commented Jun 7, 2023

iholder101 commented Jun 7, 2023

tghfly commented Jun 7, 2023

iholder101 commented Jun 7, 2023

xpivarc commented Sep 13, 2023

iholder101 commented Feb 27, 2023 •

edited

iholder101 commented Mar 1, 2023 •

edited

kubevirt-bot commented Mar 2, 2023 •

edited

tghfly commented Jun 6, 2023 •

edited

iholder101 commented Jun 7, 2023 •

edited