Update volume group snapshot KEP

xing-yang · xing-yang · commit 98fc279ba459 · 2025-06-08T11:52:19.000-04:00
diff --git a/keps/sig-storage/3476-volume-group-snapshot/README.md b/keps/sig-storage/3476-volume-group-snapshot/README.md
@@ -41,6 +41,7 @@
     - [CreateVolumeGroupSnapshot](#createvolumegroupsnapshot)
     - [DeleteVolumeGroupSnapshot](#deletevolumegroupsnapshot)
     - [GetVolumeGroupSnapshot](#getvolumegroupsnapshot)
+    - [GetSnapshot](#getsnapshot)
 - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
   - [Feature enablement and rollback](#feature-enablement-and-rollback)
   - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
@@ -82,7 +83,7 @@ Items marked with (R) are required *prior to targeting to a milestone / release*
 - [x] (R) Production readiness review completed
 - [x] Production readiness review approved
 - [x] "Implementation History" section is up-to-date for milestone
-- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [x] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
 - [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
 
 <!--
@@ -134,13 +135,16 @@ Note: In the following, we will use VolumeGroupSnapshot Controller to refer to t
 * Admin creates a VolumeGroupSnapshotClass.
 * User creates a VolumeGroupSnapshot with label selector that matches the label applied to all PVCs to be snapshotted together.
 * This will trigger the VolumeGroupSnapshot controller to create a VolumeGroupSnapshotContent API object. The group snapshot logic in the csi-snapshotter sidecar will call the CreateVolumeGroupSnapshot CSI function.
-* The group snapshot logic in csi-snapshotter will retrieve all volumeSnapshotHandles and their source volumeHandles in the Volume Group Snapshot from the CSI CreateVolumeGroupSnapshotResponse, and populate the VolumeSnapshotHandlePairList field in the VolumeGroupSnapshotContent status.
+* The group snapshot logic in csi-snapshotter will retrieve all volumeSnapshotHandles, their source volumeHandles, and other information in the Volume Group Snapshot from the CSI CreateVolumeGroupSnapshotResponse, and update the VolumeGroupSnapshotContent status.
+  * In v1beta1, it populates the VolumeSnapshotHandlePairList field in the VolumeGroupSnapshotContent status.
+  * In v1beta2, it populates the VolumeSnapshotInfoList field in the VolumeGroupSnapshotContent status for the new v1beta2 API objects. The existing v1beta1 API objects will be converted to the new v1beta2 API objects. The conversion logic will only populate the VolumeHandle and SnapshotHandle fields and will leave the remaining fields empty. A conversion webhook will be developed to make the conversion.
+  * Note: We initially encountered an [issue](https://github.com/kubernetes-csi/external-snapshotter/issues/1271) where the restoreSize is not set for individual VolumeSnapshotContents and VolumeSnapshots if CSI driver does not implement ListSnapshots. We evaluated various options [here](https://docs.google.com/document/d/1LLBSHcnlLTaP6ZKjugtSGQHH2LGZPndyfnNqR1YvzS4/edit?tab=t.0). Making this v1beta2 API change is one option that can solve this problem. There is a second reason for making this v1beta2 API change. We need to make the v1beta2 API change for the pre-provisioned VolumeGroupSnapshot to automatically create individual VolumeSnapshots; otherwise, user has to manually create both group and individual snapshots API objects during static provisioning.
 * The VolumeGroupSnapshot controller will be watching the VolumeGroupSnapshotContent, and create VolumeSnapshotContents pointing to the volumeSnapshotHandles once they are available in the VolumeGroupSnapshotContent status. Then the controller will create VolumeSnapshots pointing to the VolumeSnapshotContents.
 * CreateVolumeGroupSnapshot CSI function response
-  * The CreateVolumeGroupSnapshot CSI function should return a list of snapshots (Snapshot message defined in CSI Spec) in its response. The group snapshot logic in the csi-snapshotter sidecar will update the VolumeSnapshotHandlePairList field in the VolumeGroupSnapshotContent status based on the returned list of snapshots from the CSI call. The VolumeGroupSnapshot controller can use VolumeSnapshotHandles to construct corresponding individual VolumeSnapshotContents and VolumeSnapshots, wait for VolumeSnapshots and VolumeSnapshotContents to be bound, and update PVCVolumeSnapshotRefList in the VolumeGroupSnapshot Status and PVVolumeSnapshotContentList in the VolumeGroupSnapshotContent Status.
+  * The CreateVolumeGroupSnapshot CSI function should return a list of snapshots (Snapshot message defined in CSI Spec) in its response. The group snapshot logic in the csi-snapshotter sidecar will update the VolumeSnapshotInfoList field in the VolumeGroupSnapshotContent status based on the returned list of snapshots from the CSI call. The VolumeGroupSnapshot controller can use VolumeSnapshotInfoList to construct corresponding individual VolumeSnapshotContents and VolumeSnapshots, wait for VolumeSnapshots and VolumeSnapshotContents to be bound.
  * Individual VolumeSnapshots will be named in this format:
-   * <snap>-<hash of VolumeGroupSnapshot UUID+PVC UUID+timestamp>
-   * A label with VolumeGroupSnapshot name will also be added to the VolumeSnapshot
+   * <snap>-<hash of VolumeGroupSnapshot UUID + volume handle>
+   * VolumeGroupSnapshot will also be added as an OwnerReference for the VolumeSnapshot
 
 ```
 apiVersion: snapshot.storage.k8s.io/v1
@@ -156,7 +160,7 @@ status:
   volumeGroupSnapshotName: groupSnapshot1
 ```
 
-* An admissions controller or finalizer should be added to prevent an individual snapshot from being deleted that belongs to a VolumeGroupSnapshot. Note that there is a [KEP](https://github.com/kubernetes/enhancements/pull/2840/files) that is proposing the Liens feature which could potentially be used for this purpose.
+* A finalizer should be added to prevent an individual snapshot from being deleted that belongs to a VolumeGroupSnapshot. Note that there is a [KEP](https://github.com/kubernetes/enhancements/pull/2840/files) that is proposing the Liens feature which could potentially be used for this purpose.
 * In the CSI spec, it is specified that it is required for individual snapshots to be returned along with the group snapshot.
 
 #### Pre-provisioned VolumeGroupSnapshot
@@ -165,6 +169,8 @@ Admin can create a VolumeGroupSnapshotContent, specifying an existing VolumeGrou
 
 The controller will call the CSI GetVolumeGroupSnapshot method to retrieve all volumeSnapshotHandles in the Volume Group Snapshot from the storage system, create VolumeSnapshotContents pointing to the volumeSnapshotHandles. Then the controller will create VolumeSnapshots pointing to the VolumeSnapshotContents.
 
+Note: The automatic creation of individual VolumeSnapshots and VolumeSnapshotContents are not done in Beta. For now, the admin will need to manually construct these individual API objects. We plan to work on this before the feature moves to GA. We have information for all the individual snapshots from CSI [GetVolumeGroupSnapshot](https://github.com/kubernetes-csi/external-snapshotter/blob/release-8.2/pkg/sidecar-controller/groupsnapshot_helper.go#L781). We should be able to populate individual VolumeSnapshots and VolumeSnapshotContents based on this information.
+
 ### Delete VolumeGroupSnapshot
 
 A VolumeGroupSnapshot can be deleted if the CSI driver supports the CREATE_DELETE_GET_VOLUME_GROUP_SNAPSHOT capability.
@@ -202,8 +208,9 @@ N/A
 Integration tests are not needed.
 
 ##### e2e tests
-* e2e tests for external volume group snapshot controller.
-* e2e tests for modified code path of external-snapshotter.
+* e2e tests for external volume group snapshot control logic in the modified code path of external-snapshotter.
+  * https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/testsuites/volume_group_snapshottable.go
+  * https://testgrid.k8s.io/sig-storage-kubernetes#storage-kind-volume-group-snapshots
 * Add stress and scale tests before moving from beta to GA.
 
 ### Graduation Criteria
@@ -373,21 +380,6 @@ Type VolumeGroupSnapshotStatus struct {
 
         // +optional
         Error *VolumeSnapshotError
-
-	// VolumeSnapshotRefList is the list of PVC and VolumeSnapshot pairs that
-	// is part of this group snapshot.
-	// The maximum number of allowed snapshots in the group is 100.
-	// +optional
-	PVCVolumeSnapshotRefList []PVCVolumeSnapshotPair
-}
-
-// PVCVolumeSnapshotPair defines a pair of a PVC reference and a Volume Snapshot Reference
-type PVCVolumeSnapshotPair struct {
-	// PersistentVolumeClaimRef is a reference to the PVC this pair is referring to
-	PersistentVolumeClaimRef core_v1.LocalObjectReference
-
-	// VolumeSnapshotRef is a reference to the VolumeSnapshot this pair is referring to
-	VolumeSnapshotRef core_v1.LocalObjectReference
 }
 ```
 
@@ -464,6 +456,8 @@ type GroupSnapshotHandles struct {
 	VolumeSnapshotHandles []string
 }
 
+// The VolumeSnapshotHandlePair struct is added in v1beta1 but removed in v1beta2
+// It is replaced by the VolumeSnapshotInfo struct
 // VolumeSnapshotHandlePair defines a pair of a source volume handle and a snapshot handle
 type VolumeSnapshotHandlePair struct {
         // VolumeHandle is a unique id returned by the CSI driver to identify a volume
@@ -475,6 +469,31 @@ type VolumeSnapshotHandlePair struct {
         SnapshotHandle string
 }
 
+// The VolumeSnapshotInfo struct is added in v1beta2
+// VolumeSnapshotInfo contains information for a snapshot
+type VolumeSnapshotInfo struct {
+        // VolumeHandle is a unique id returned by the CSI driver to identify a volume
+        // on the storage system
+        VolumeHandle string
+
+        // SnapshotHandle is a unique id returned by the CSI driver to identify a volume
+        // snapshot on the storage system
+        SnapshotHandle string
+
+	// creationTime is the timestamp when the point-in-time snapshot is taken
+	// by the underlying storage system.
+	// +optional
+	CreationTime *int64
+
+	// ReadyToUse indicates if the snapshot is ready to be used to restore a volume.
+	// +optional
+	ReadyToUse *bool
+
+	// RestoreSize represents the complete size of the snapshot in bytes.
+	// +optional
+	RestoreSize *int64
+}
+
 Type VolumeGroupSnapshotContentStatus struct {
         // VolumeGroupSnapshotHandle is a unique id returned by the CSI driver
         // to identify the VolumeGroupSnapshot on the storage system.
@@ -483,28 +502,32 @@ Type VolumeGroupSnapshotContentStatus struct {
         // +optional
         VolumeGroupSnapshotHandle *string
 
+	// This field is introduced in v1beta1 but removed in v1beta2
+	// It is replaced by VolumeSnapshotInfoList
+	// Information in this field from an existing v1beta1 API object
+	// will be copied to VolumeSnapshotInfoList by the conversion logic
         // VolumeSnapshotHandlePairList is a list of CSI "volume_id" and "snapshot_id"
 	// pair returned by the CSI driver to identify snapshots and their source volumes
 	// on the storage system.
         // +optional
-        VolumeSnapshotHandlePairList []VolumeSnapshotHandlePair
+        // VolumeSnapshotHandlePairList []VolumeSnapshotHandlePair
+
+	// This field is introduced in v1beta2
+	// It is replacing VolumeSnapshotHandlePairList
+        // VolumeSnapshotInfoList is a list of snapshot information returned by
+        // by the CSI driver to identify snapshots on the storage system.
+        // +optional
+        VolumeSnapshotInfoList []VolumeSnapshotInfo
 
         // ReadyToUse becomes true when ReadyToUse on all individual snapshots become true
         // +optional
         ReadyToUse *bool
 
         // +optional
-        CreationTime *int64
+        CreationTime *metav1.Time
 
         // +optional
         Error *VolumeSnapshotError
-
-	// NOTE: We will consider removing this field after testing.
-	// PVVolumeSnapshotContentList is the list of pairs of PV and
-	// VolumeSnapshotContent for this group snapshot
-	// The maximum number of allowed snapshots in the group is 100.
-	// +optional
-	PVVolumeSnapshotContentList []PVVolumeSnapshotContentPair
 }
 ```
 
@@ -569,24 +592,23 @@ A new group controller service will be added with a new controller capability CR
   Indicates that the controller plugin supports creating, deleting, and getting details of a snapshot of
   multiple volumes.
 
+A new controller capability GET_SNAPSHOT will also be added. This indicates that the controller plugin supports getting details of a snapshot of multiple volumes.
+
 #### CSI Group Controller RPC
 
 ```
 service Controller {
   …
   rpc CreateVolumeGroupSnapshot(CreateVolumeGroupSnapshotRequest)
     returns (CreateVolumeGroupSnapshotResponse) {
-        option (alpha_method) = true;
     }
 
   rpc DeleteVolumeGroupSnapshot(DeleteVolumeGroupSnapshotRequest)
     returns (DeleteVolumeGroupSnapshotResponse) {
-        option (alpha_method) = true;
     }
 
   rpc GetVolumeGroupSnapshot(GetVolumeGroupSnapshotRequest)
     returns (GetVolumeGroupSnapshotResponse) {
-        option (alpha_method) = true;
     }
   …
 }
@@ -598,8 +620,6 @@ The purpose of this call is to request the creation of a multi-volume snapshot.
 
 ```
 message CreateVolumeGroupSnapshotRequest {
-  option (alpha_message) = true;
-
   // The suggested name for the group snapshot. This field is REQUIRED
   // for idempotency.
   // Any Unicode string that conforms to the length limit is allowed
@@ -627,16 +647,12 @@ message CreateVolumeGroupSnapshotRequest {
 }
 
 message CreateVolumeGroupSnapshotResponse {
-  option (alpha_message) = true;
-
   // Contains all attributes of the newly created group snapshot.
   // This field is REQUIRED.
   VolumeGroupSnapshot group_snapshot = 1;
 }
 
 message VolumeGroupSnapshot {
-  option (alpha_message) = true;
-
   // The identifier for this group snapshot, generated by the plugin.
   // This field MUST contain enough information to uniquely identify
   // this specific snapshot vs all other group snapshots supported by
@@ -673,8 +689,6 @@ message VolumeGroupSnapshot {
 
 ```
 message DeleteVolumeGroupSnapshotRequest {
-  option (alpha_message) = true;
-
   // The ID of the group snapshot to be deleted.
   // This field is REQUIRED.
   string group_snapshot_id = 1;
@@ -706,8 +720,6 @@ message DeleteVolumeGroupSnapshotResponse {
 
 ```
 message GetVolumeGroupSnapshotRequest {
-  option (alpha_message) = true;
-
   // The ID of the group snapshot to fetch current group snapshot
   // information for.
   // This field is REQUIRED.
@@ -725,13 +737,35 @@ message GetVolumeGroupSnapshotRequest {
 }
 
 message GetVolumeGroupSnapshotResponse {
-  option (alpha_message) = true;
-
   // This field is REQUIRED
   VolumeGroupSnapshot group_snapshot = 1;
 }
 ```
 
+#### GetSnapshot
+
+GetSnapshot is an optional controller capability that can help retrieve snapshot information. It can be used by Kubernetes to populate fields in individual VolumeSnapshotContents and VolumeSnapshots API objects that belong to a VolumeGroupSnapshot.
+
+This is introduced because some CSI drivers cannot implement ListSnapshots RPC due to performance concerns. With the introduction of this new RPC, The CSI Snapshotter sidecar will call GetSnapshot if it is implemented. If GetSnapshot is not implemented, the snapshotter will fall back to the current behavior which is to call ListSnapshots instead. If neither GetSnapshot nor ListSnapshots is implemented, the snapshotter will fall back to the current behavior which is to assume the snapshot_id exists without being able to retrieve more information about this snapshot. Currently ListSnapshots is only called to retrieve status for pre-provisioned snapshots. The immediate use case of the new GetSnapshot RPC will also be for pre-provisioned snapshots. In the future, it is possible to use this API for other cases.
+
+```
+message GetSnapshotRequest {
+  // The ID of the snapshot to fetch current snapshot information for.
+  // This field is REQUIRED.
+  string snapshot_id = 1;
+
+  // Secrets required by plugin to complete GetSnapshot request.
+  // This field is OPTIONAL. Refer to the `Secrets Requirements`
+  // section on how to use this field.
+  map<string, string> secrets = 2 [(csi_secret) = true];
+}
+
+message GetSnapshotResponse {
+  // This field is REQUIRED
+  Snapshot snapshot = 1;
+}
+```
+
 ## Production Readiness Review Questionnaire
 
 ### Feature enablement and rollback
@@ -741,9 +775,11 @@ _This section must be completed when targeting alpha to a release._
 * **How can this feature be enabled / disabled in a live cluster?**
   - [x] Other
     - Describe the mechanism:
-      We don't have a feature gate because this feature is out of tree.
-      We will use a flag called enable-volume-group-snapshot to enable this
+      This feature is out of tree.
+      We initially used a flag called enable-volume-group-snapshot to enable this
       feature when the snapshot controller and csi-snapshotter sidecar are started.
+      We changed this flag to a feature gate `VolumeGroupSnapshot` when moving it
+      to Beta.
     - Will enabling / disabling the feature require downtime of the control
       plane?
       From the controller side, it only affects the external controller and sidecars.
@@ -943,6 +979,11 @@ provider?**
 the existing API objects?**
   Describe them, providing:
   - API type(s): A new string field is added to VolumeSnapshot and VolumeSnapshotContent respectively.
+    Update from v1beta1 to v1beta2 API:
+    In v1beta1, the VolumeGroupSnapshotContentStatus contains this field:
+    VolumeSnapshotHandlePairList: [snapshotHandle, volumeHandle]
+    In v1beta1, this is replaced with VolumeSnapshotInfoList: [snapshotHandle, volumeHandle, creationTime (*int64), readyToUse (*bool), restoreSize (*int64)]
+
   - Estimated increase in size: (e.g., new annotation of size 32B):
   - Estimated amount of new objects: (e.g., new Object X for every existing Pod)
 
diff --git a/keps/sig-storage/3476-volume-group-snapshot/kep.yaml b/keps/sig-storage/3476-volume-group-snapshot/kep.yaml
@@ -18,14 +18,14 @@ approvers:
 see-also:
 replaces:
 
-latest-milestone: "v1.32"
+latest-milestone: "v1.34"
 stage: "beta"
 milestone:
   alpha: "v1.27"
   beta: "v1.32"
   stable:
 
-feature-gates:
+feature-gates: VolumeGroupSnapshot (out-of-tree)
 disable-supported: true
 
 metrics:
