Update ControllerModifyVolume Status API #4298

sunnylovestiramisu · 2023-10-17T01:12:06Z

One-line PR description:

Other comments:

sunnylovestiramisu · 2023-10-17T21:18:08Z

msau42 · 2023-10-17T21:34:09Z

keps/sig-storage/3751-volume-attributes-class/README.md

@@ -177,22 +177,27 @@ type PersistentVolumeClaimStatus struct {
  ...
  // The VolumeAttributesClassName string cannot be empty
  VolumeAttributesClassName string


remove this field since it is now moved to actualclassname

msau42 · 2023-10-17T21:34:42Z

keps/sig-storage/3751-volume-attributes-class/README.md

  ...
 }

+type ModifyVolumeStatus struct {
+  ActualClassName string


replace Class with VolumeAttributesClass

msau42 · 2023-10-17T21:35:58Z

keps/sig-storage/3751-volume-attributes-class/README.md

 // +enum
 type PersistentVolumeClaimModifyVolumeStatus string

 const (
  // When modify volume is complete, the empty string is set by modify volume controller.
-  PersistentVolumeClaimNoModifyVolumeInProgress PersistentVolumeClaimModifyVolumeStatus = ""
+  PersistentVolumeClaimNoModifyVolumeInSucceeded PersistentVolumeClaimModifyVolumeStatus = ""


It's either succeeded or not started. I don't know if we need a const for this.

What's the difference between a nil value and an empty value? the Status field of the ModifyVolumeStatus is pointer type.

msau42 · 2023-10-17T21:39:23Z

keps/sig-storage/3751-volume-attributes-class/README.md

  ...
 }

+type ModifyVolumeStatus struct {
+  ActualClassName string


I am wondering whether we should actually make this field an object reference instead of just a name.

I am thinking of the scenario where someone may delete and recreate the VAC with the same name but different content. But actually, if we really want to keep track of the attributes are, we may need to keep track of it in the PV object. @jsafrane @gnufied wdyt?

Is uuid enough to distinguish instead of doing comparison of the whole VAC object?

Usually the protection controller will prevent the VAC to be deleted but an admin can force delete.

keps/sig-storage/3751-volume-attributes-class/README.md

msau42 · 2023-10-17T21:43:43Z

/assign @gnufied

keps/sig-storage/3751-volume-attributes-class/README.md

carlory · 2023-10-18T01:40:34Z

keps/sig-storage/3751-volume-attributes-class/README.md

 // +enum
 type PersistentVolumeClaimModifyVolumeStatus string

 const (
  // When modify volume is complete, the empty string is set by modify volume controller.
-  PersistentVolumeClaimNoModifyVolumeInProgress PersistentVolumeClaimModifyVolumeStatus = ""
+  PersistentVolumeClaimNoModifyVolumeInSucceeded PersistentVolumeClaimModifyVolumeStatus = ""


What's the difference between a nil value and an empty value? the Status field of the ModifyVolumeStatus is pointer type.

keps/sig-storage/3751-volume-attributes-class/README.md

carlory · 2023-10-18T02:37:40Z

keps/sig-storage/3751-volume-attributes-class/VolumeAttributesClass-ModifyVolume-Flow-v2.png

IMO, If succeed, pvc.ModifyVolumeStatus.ActualClassName should be updated to pvc.ModifyVolumeStatus.TargetClassName and pvc.ModifyVolumeStatus.Status should be changed to nil.

Yes both the status and actualClassName will be set as you said.

carlory · 2023-10-18T02:54:04Z

A user creates a pvc with a VolumeAttributesClassName set. if the parameters of the PVC are not accepted by the lower version csi-driver but are accepted by the higher version csi-driver, the status of the PVC may be stuck in infeasible state. If the admin upgrades the csi-driver to the higher version, the PVC how to recover the PVC to user expected state? To modify the PVC to a different VolumeAttributesClass which parameters is same as the previous one is not a good UX for the user.

msau42 · 2023-10-18T04:08:51Z

A user creates a pvc with a VolumeAttributesClassName set. if the parameters of the PVC are not accepted by the lower version csi-driver but are accepted by the higher version csi-driver, the status of the PVC may be stuck in infeasible state. If the admin upgrades the csi-driver to the higher version, the PVC how to recover the PVC to user expected state? To modify the PVC to a different VolumeAttributesClass which parameters is same as the previous one is not a good UX for the user.

I am thinking that even in the Infeasible case, we will still have to periodically retry, but at a lower frequency than if it was InProgress

gnufied · 2023-10-18T17:24:32Z

The ModifyStatus field on the image diagram is not in right place.

carlory · 2023-10-23T03:53:11Z

keps/sig-storage/3751-volume-attributes-class/README.md

+const (
+  ...
+  // PersistentVolumeClaimModifyVolume - a user trigger modify volume of pvc has been started
+  PersistentVolumeClaimModifyVolume PersistentVolumeClaimConditionType = "ModifyVolume"


I'm having trouble understanding this name of the new condition type. Is this saying that all the moditication of the volume which includes the reszie operation or just for changing volume attributes? If it's the latter, I think we should rename it to something like ModifyVolumeAttributes or ModifyVolumeAttributesClass.

I do not have a strong opinion of the naming, I was following the previous naming by using actions instead of a noun. This condition is used to document the last error the controller sees.

Discussed in the sig-storage meeting, we will name it "VolumeAttriburesModifyError"

If we are not going to change status when we retry ModifyVolume after failing (and status is set to Infeasible), then we need to somehow communicate to the user that, we are again retrying ModifyVolume (albeit slowly), so we will I think need a condition for that too. cc @msau42

When do you propose we set a ModifyVolume condition to true and false? From api-conventions, it says:

For state transitions which take a long period of time (e.g. more than 1 minute), it is reasonable to treat the transition itself as an observed state. In these cases, the Condition (such as "Resizing") itself should not be transient, and should instead be signalled using the True/False/Unknown pattern. This allows other observers to determine the last update from the controller, whether successful or failed. In cases where the state transition is unable to complete and continued reconciliation is not feasible, the Reason and Message should be used to indicate that the transition failed.

I interpret the last sentence to mean that if we still intend to retry, the status should still be True.

So the main benefits I see of having the condition is:

that the timestamp can be updated every time we retry an operation.

we could potentially return false for some preconditions like VAC doesn't exist or is invalid, PVC not bound, etc.

keps/sig-storage/3751-volume-attributes-class/README.md

carlory · 2023-10-23T04:02:40Z

keps/sig-storage/3751-volume-attributes-class/README.md

 // +enum
+// When a controller receives persistentvolume claim update with PersistentVolumeClaimModifyVolumeStatus for a resource
+// that it does not recognizes, then it should ignore that update and let other controllers
+// handle it.


In which case the controller cannot recognize the status value?

This should not happen but like if someone setting their own status in a fork etc.

I think this comment is for telling consumers that how to handle the scenario where we may add new statuses in the future. I think it's simpler to reword it like "New statuses can be added in the future. Consumers should check for unknown statuses and fail appropriately".

carlory · 2023-10-23T04:05:07Z

keps/sig-storage/3751-volume-attributes-class/README.md

 type PersistentVolumeClaimModifyVolumeStatus string

 const (
-  // When modify volume is complete, the empty string is set by modify volume controller.
-  PersistentVolumeClaimNoModifyVolumeInProgress PersistentVolumeClaimModifyVolumeStatus = ""
+  // State set when modify volume controller wants to modify the volume in control-plane


What is the modify volume controller? pv controller in kube-controller-manager or CSI driver controller?

it is the resize controller actually, we are putting this modify volume controller logic in the external-resizer.

I think we don't need to specify exactly which controller is handling what. That's more of an implementation detail. I think we can reword it to say something like "Pending indicates that the PersistentVolumeClaim cannot be modified due to requirements not being met, such as the PersistentVolumeClaim being in an invalid state or the specified VolumeAttributesClass is existing"

ditto below. I think we can remove references to specific controllers in the comments.

carlory · 2023-10-23T04:05:20Z

keps/sig-storage/3751-volume-attributes-class/README.md

  // State set when modify volume controller starts modifying the volume in control-plane
+  // When the request has been rejected as invalid by the CSI driver. To resolve this error,


This comment should apply to the "Infeasible" one, not "InProgress"

carlory · 2023-10-23T04:12:04Z

keps/sig-storage/3751-volume-attributes-class/README.md

+  // parameters it intends to modify. SPs MUST NOT modify volumes based
+  // on the absence of keys, only keys that are specified should result
+  // in modifications to the volume.
+  map<string, string> mutable_parameters = 3;


Is the parameters field of the VAC optional? If the mutable_parameters is required, the field in the VAC definition may be required as well and at least contains one key.

from chatgpt:

I’m curious why the required and optional keyword is not used in fields in the csi proto file.

We can add it, but that is the CSI spec update. The CSI Spec release has been cut already. We can revisit at next CSI meeting.

Actually in the CSI spec, there are some extra paragraph I just did not copy pasted here because it is stand alone:

// Plugin specific creation-time parameters passed in as opaque // key-value pairs. These mutable_parameteres MAY also be // changed during the lifetime of the volume via a subsequent // `ControllerModifyVolume` RPC. This field is OPTIONAL. // The Plugin is responsible for parsing and validating these // parameters. COs will treat these as opaque.

Please refer to the CSI spec for accuracy.

And also in CSI spec:

message ControllerModifyVolumeRequest { option (alpha_message) = true; // Contains identity information for the existing volume. // This field is REQUIRED. string volume_id = 1; // Secrets required by plugin to complete modify volume request. // This field is OPTIONAL. Refer to the `Secrets Requirements` // section on how to use this field. map<string, string> secrets = 2 [(csi_secret) = true]; // Plugin specific volume attributes to mutate, passed in as // opaque key-value pairs. // This field is REQUIRED. The Plugin is responsible for // parsing and validating these parameters. COs will treat these // as opaque. The CO SHOULD specify the intended values of all mutable // parameters it intends to modify. SPs MUST NOT modify volumes based // on the absence of keys, only keys that are specified should result // in modifications to the volume. map<string, string> mutable_parameters = 3; }

Can we defer this to CSI meeting in general? This PR is focusing on unblocking the k8s API change.

So in CreateVolume it is optional, but in ModifyVolume it is required. I think that makes sense.

From a k8s POV, if a user doesn't specify any vacName in the PVC, then that results in empty mutable_parameters.

A VAC having empty parameters doesn't make sense. If there is nothing to change then why would someone specify a vacName in their PVC?

I am going to remove this session in the doc and please refer to the CSI spec.md

msau42 · 2023-10-24T04:07:02Z

keps/sig-storage/3751-volume-attributes-class/README.md

+// Condition is used to document the last error controller sees
+const (
+  ...
+  // PersistentVolumeClaimVolumeAttriburesModifyError - a user trigger modify volume of pvc has been started


The comment needs to be updated

msau42 · 2023-10-24T04:11:51Z

keps/sig-storage/3751-volume-attributes-class/README.md

+// ModifyVolumeStatus represents the status object of ControllerModifyVolume operation
+type ModifyVolumeStatus struct {
+  // ActualVolumeAttributesClassName is the actual name of the VolumeAttributesClass the PVC is using
+  ActualVolumeAttributesClassName string


Instead of "Actual", I would use "Current" to be consistent with how other resources specify current status (like StatefulSet.status.currentReplicas)

msau42 · 2023-10-24T04:13:37Z

keps/sig-storage/3751-volume-attributes-class/README.md

+type ModifyVolumeStatus struct {
+  // ActualVolumeAttributesClassName is the actual name of the VolumeAttributesClass the PVC is using
+  ActualVolumeAttributesClassName string
+  // TargetVolumeAttributesClassName is the target name of the VolumeAttributesClass the PVC wants to use


It's not quite what was requested in the PVC. It is what is currently being reconciled, which may not necessarily match. For example, if user sets PVC.spec.vacName = A, then B, then C, we may still being trying to reconcile B even though the user has set C.

msau42 · 2023-10-24T04:18:25Z

keps/sig-storage/3751-volume-attributes-class/README.md

 // +enum
+// When a controller receives persistentvolume claim update with PersistentVolumeClaimModifyVolumeStatus for a resource
+// that it does not recognizes, then it should ignore that update and let other controllers
+// handle it.


I think this comment is for telling consumers that how to handle the scenario where we may add new statuses in the future. I think it's simpler to reword it like "New statuses can be added in the future. Consumers should check for unknown statuses and fail appropriately".

msau42 · 2023-10-24T04:28:52Z

keps/sig-storage/3751-volume-attributes-class/README.md

 type PersistentVolumeClaimModifyVolumeStatus string

 const (
-  // When modify volume is complete, the empty string is set by modify volume controller.
-  PersistentVolumeClaimNoModifyVolumeInProgress PersistentVolumeClaimModifyVolumeStatus = ""
+  // State set when modify volume controller wants to modify the volume in control-plane


I think we don't need to specify exactly which controller is handling what. That's more of an implementation detail. I think we can reword it to say something like "Pending indicates that the PersistentVolumeClaim cannot be modified due to requirements not being met, such as the PersistentVolumeClaim being in an invalid state or the specified VolumeAttributesClass is existing"

msau42 · 2023-10-24T04:30:02Z

keps/sig-storage/3751-volume-attributes-class/README.md

  // State set when modify volume controller starts modifying the volume in control-plane
+  // When the request has been rejected as invalid by the CSI driver. To resolve this error,


This comment should apply to the "Infeasible" one, not "InProgress"

msau42 · 2023-10-24T04:30:42Z

keps/sig-storage/3751-volume-attributes-class/README.md

 type PersistentVolumeClaimModifyVolumeStatus string

 const (
-  // When modify volume is complete, the empty string is set by modify volume controller.
-  PersistentVolumeClaimNoModifyVolumeInProgress PersistentVolumeClaimModifyVolumeStatus = ""
+  // State set when modify volume controller wants to modify the volume in control-plane


ditto below. I think we can remove references to specific controllers in the comments.

msau42 · 2023-10-24T04:33:10Z

keps/sig-storage/3751-volume-attributes-class/README.md

@@ -409,12 +449,25 @@ The resource quota controller is the only component capable of monitoring and re
 ### 3. Add new statuses in PVC API to indicate changes of VolumeAttributesClass and the status of the ModifyVolume operation.

 ```
-type VolumeAttributesClassStatus string
+type ModifyVolumeStatus struct {


I would save this for a future cleanup, but I'm not sure why we want to define the api definitions in multiple places in the doc. It means if we want to make changes like this, we need to remember to modify it everywhere.

msau42 · 2023-10-24T04:35:35Z

keps/sig-storage/3751-volume-attributes-class/README.md

-  PersistentVolumeClaimNoModifyVolumeInProgress PersistentVolumeClaimModifyVolumeStatus = ""
+  // State set when modify volume controller wants to modify the volume in control-plane
+  // but the VolumeAttributesClass does not exist.
+  PersistentVolumeClaimControllerModifyVolumePending PersistentVolumeClaimModifyVolumeStatus = "ControllerModifyVolumePending"


I don't know if we need the "Controller" in the name since we're not going to support Node operations.

We are not going to but will we ever consider supporting node operations?

carlory · 2023-10-25T03:14:51Z

@sunnylovestiramisu

Accroding to the discussion on the API PR, we also need to change the VolumeAttributesClassName description in the PVC and PV definition.

A VAC having empty parameters doesn't make sense. If there is nothing to change then why would someone specify a vacName in their PVC?

On the other hand, we need change the parameters field of the VolumeAttributesClass to required from optional. Users must provide at least one key-value pair for the parameters field.

carlory · 2023-10-25T07:41:41Z

keps/sig-storage/3751-volume-attributes-class/README.md

+  // TargetVolumeAttributesClassName is the name of the VolumeAttributesClass the PVC currently being reconciled
+  TargetVolumeAttributesClassName string
+  // Status is the status of the ControllerModifyVolume operation
+  Status *PersistentVolumeClaimModifyVolumeStatus


@sunnylovestiramisu @msau42

Why use a pointer type? Do we want to distinguish a nil value and a zero value? If so, why? If not, why not use a non-pointer type?

History of the VAC name being a pointer: https://github.com/kubernetes/enhancements/pull/3780/files#r1230300561

Hmm actually this should not be a pointer. It should be consistent of the AllocatedResourceStatus

keps/sig-storage/3751-volume-attributes-class/README.md

msau42 · 2023-10-26T04:02:16Z

keps/sig-storage/3751-volume-attributes-class/README.md

-  VolumeAttributesClassName string
-  ModifyVolumeStatus *PersistentVolumeClaimModifyVolumeStatus
+  // ModifyVolumeStatus represents the status object of ControllerModifyVolume operation
+  ModifyVolumeStatus ModifyVolumeStatus


New fields in existing stucts:
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#optional-vs-required

msau42 · 2023-10-26T04:06:14Z

keps/sig-storage/3751-volume-attributes-class/README.md

+const (
+  ...
+  // PersistentVolumeClaimVolumeAttriburesModifyError - a user trigger modify volume of pvc has error
+  PersistentVolumeClaimVolumeAttriburesModifyError PersistentVolumeClaimConditionType = "VolumeAttriburesModifyError"


We also want to add the "VolumeAttributesModifying" condition based on this discussion with @gnufied #4298 (comment)

Why do we need a modifying? We cannot just say "Err but we are retrying" in the messaging?

How does an operator that is possibly monitoring the pvc knows volume is being modified or not? By regexp. for message on the condition? Seems fragile.

carlory · 2023-10-27T02:10:26Z

keps/sig-storage/3751-volume-attributes-class/README.md

  ...
 }

+// ModifyVolumeStatus represents the status object of ControllerModifyVolume operation


If we don't want to distinct the ControllerModifyVolume and XXXModifyVolume, Is it possible to drop the Controller prefix in the ModifyVolumeStatus description?

It is already dropped in the newest revision?

the ControllerModifyVolume word still exists in L197 and L201

msau42 · 2023-10-31T06:36:42Z

/lgtm
/approve

k8s-ci-robot · 2023-10-31T06:36:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msau42, sunnylovestiramisu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/sig-storage/OWNERS~~ [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot requested review from saad-ali and xing-yang October 17, 2023 01:12

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Oct 17, 2023

sunnylovestiramisu mentioned this pull request Oct 17, 2023

[KEP-3751] introduce the VolumeAttributesClass API kubernetes/kubernetes#121104

Merged

sunnylovestiramisu force-pushed the kep-3751 branch 2 times, most recently from 406e8eb to dc6c59c Compare October 17, 2023 21:11

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 17, 2023

sunnylovestiramisu changed the title ~~[WIP] Update ControllerModifyVolume Status API~~ Update ControllerModifyVolume Status API Oct 17, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 17, 2023

sunnylovestiramisu force-pushed the kep-3751 branch from dc6c59c to eb58d9f Compare October 17, 2023 21:14

k8s-ci-robot assigned msau42 Oct 17, 2023

msau42 reviewed Oct 17, 2023

View reviewed changes

keps/sig-storage/3751-volume-attributes-class/README.md Outdated Show resolved Hide resolved

k8s-ci-robot assigned gnufied Oct 17, 2023

sunnylovestiramisu force-pushed the kep-3751 branch from eb58d9f to e75773c Compare October 17, 2023 21:57

carlory reviewed Oct 18, 2023

View reviewed changes

sunnylovestiramisu force-pushed the kep-3751 branch from e75773c to 6376b2f Compare October 20, 2023 23:00

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 20, 2023

sunnylovestiramisu force-pushed the kep-3751 branch from 6376b2f to c25c07c Compare October 20, 2023 23:06

carlory reviewed Oct 23, 2023

View reviewed changes

keps/sig-storage/3751-volume-attributes-class/README.md Outdated Show resolved Hide resolved

carlory reviewed Oct 23, 2023

View reviewed changes

sunnylovestiramisu force-pushed the kep-3751 branch from 5b996bd to e244cc4 Compare October 23, 2023 21:46

msau42 reviewed Oct 24, 2023

View reviewed changes

Update ControllerModifyVolume Status API

d41665c

sunnylovestiramisu force-pushed the kep-3751 branch from e244cc4 to 77912d8 Compare October 24, 2023 16:36

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 24, 2023

Update based on comments

256bee5

sunnylovestiramisu force-pushed the kep-3751 branch from 77912d8 to 256bee5 Compare October 24, 2023 16:38

carlory reviewed Oct 25, 2023

View reviewed changes

keps/sig-storage/3751-volume-attributes-class/README.md Outdated Show resolved Hide resolved

sunnylovestiramisu force-pushed the kep-3751 branch from fce58d1 to 5f0a6c5 Compare October 25, 2023 16:05

Update v3 flow diagram

23fa960

sunnylovestiramisu force-pushed the kep-3751 branch from 5f0a6c5 to 23fa960 Compare October 25, 2023 16:12

msau42 reviewed Oct 26, 2023

View reviewed changes

Update on Oct 26

3b80df9

carlory reviewed Oct 27, 2023

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 31, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 31, 2023

k8s-ci-robot merged commit e6ac4bd into kubernetes:master Oct 31, 2023
4 checks passed

k8s-ci-robot added this to the v1.29 milestone Oct 31, 2023

pacoxu mentioned this pull request Nov 1, 2023

Kubernetes VolumeAttributesClass ModifyVolume #3751

Open

11 tasks

		// State set when modify volume controller starts modifying the volume in control-plane
		// When the request has been rejected as invalid by the CSI driver. To resolve this error,

Update ControllerModifyVolume Status API #4298

Update ControllerModifyVolume Status API #4298

Conversation

sunnylovestiramisu commented Oct 17, 2023

sunnylovestiramisu commented Oct 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunnylovestiramisu Oct 17, 2023 • edited

Choose a reason for hiding this comment

msau42 commented Oct 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlory commented Oct 18, 2023

msau42 commented Oct 18, 2023

gnufied commented Oct 18, 2023

carlory Oct 23, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlory commented Oct 25, 2023 • edited

Choose a reason for hiding this comment

sunnylovestiramisu Oct 25, 2023 • edited

Choose a reason for hiding this comment

sunnylovestiramisu Oct 25, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Oct 31, 2023

k8s-ci-robot commented Oct 31, 2023

sunnylovestiramisu Oct 17, 2023 •

edited

carlory Oct 23, 2023 •

edited

carlory commented Oct 25, 2023 •

edited

sunnylovestiramisu Oct 25, 2023 •

edited

sunnylovestiramisu Oct 25, 2023 •

edited