VolumeGroupSnapshot

[xing-yang]

Enhancement Description

• One-line enhancement description (can be used as a release note): Introduce a VolumeGroupSnapshot API to take a snapshot of multiple volumes together.

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/3476-volume-group-snapshot/README.md

• Discussion Link:

• Primary contact (assignee): @xing-yang

• Responsible SIGs: sig-storage

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.27
  • Beta release target (x.y): 1.32
  • Stable release target (x.y):

• Alpha

  • KEP (k/enhancements) update PR(s): KEP-3476: Add Volume Group Snapshot KEP #1551
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s): Placeholder blog PR: Add post-release blog article about Volume Group Snapshot alpha website#39415

• Beta

  • KEP (k/enhancements) update PR(s): Update VolumeGroupSnapshot KEP for Beta #4734
  • Code (k/k) update PR(s): Update e2e test script and expand feature focus for volume group snapshots test-infra#33417, Add new E2E job for volume group snapshots in KIND cluster test-infra#33567, Add end-to-end tests for Volume Group Snapshot kubernetes#126326
  • Docs (k/website) update(s): doc PR: Add document link for volume group snapshots website#48419, blog PR: Add a blog for volume group snapshots beta website#48420

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[xing-yang]

/sig storage

[xing-yang]

/milestone v1.26

[ruheenaansari34]

Hello @xing-yang 👋, 1.26 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT on Thursday 6th October 2022.

This enhancement is targeting for stage alpha for 1.26 (correct me, if otherwise).

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has a updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would just need to update the following:

• Update README.md on PR KEP-3476: Add Volume Group Snapshot KEP #1551 to include test plan details as per the latest template
• Include Design Details section and move Test Plan and Graduation Criteria sections into Design Details.
• Merge PR KEP-3476: Add Volume Group Snapshot KEP #1551 which will take care of meeting the above requirements.

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[shecodesmagic]

Hello @xing-yang 👋, v1.32 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage beta for v1.32 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has submitted a production readiness review request for approval and has a reviewer assigned.
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would need to update the following:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP has submitted a production readiness review request for approval and has a reviewer assigned.
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[xing-yang]

Hi @shecodesmagic, I have updated the KEP with the latest template. Can you please verify? Thanks!

[shecodesmagic]

With all the requirements fulfilled this enhancement is now marked as tracked for the upcoming enhancements freeze 🚀

[spurin]

Hi @xing-yang 👋, Great to see you here, I'm contributing as a 1.32 Docs Shadow.

Does this enhancement work planned for 1.32 require any new docs or modifications to the existing docs?

If so, please follows the steps here to open a PR against dev-1.32 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday October 24th 2024 18:00 PDT.

Also, take a look at Documenting for a release to familiarise with the docs requirement for the release.

Thank you!

[xing-yang]

Hi @spurin, great to see you here too! This feature is implemented in the out-of-tree repo https://github.com/kubernetes-csi/external-snapshotter. The doc will be updated in this repo: https://github.com/kubernetes-csi/docs and it shows up here: https://kubernetes-csi.github.io/docs/group-snapshot-restore-feature.html. So we don't have in-tree docs work planned for 1.32. Thank you!

[spurin]

Thanks @xing-yang, the referenced snapshot documentation is great. Can I please query some points areas for consideration and further discussion:

• We're aiming for this to become Beta in the 1.32 release. In the documentation referenced I can see that this is currently listed as Alpha for 1.27. I'm assuming that will be updated to coincide with the 1.32 release. Is this external documentation update requirement being tracked under this KEP?

• How do the in-tree VolumeSnapshotClass and out-of-tree VolumeGroupSnapshotClass features work together? Are there any considerations for users who need to manage both individual and group snapshots and are there any known limitations or dependencies between these two controllers? (Asking from the viewpoint of whether we need to consider this from an in-tree documentation viewpoint)

• Given that this is transitioning to Beta, is it worthwhile cross-referencing this from the in-tree to the out-of-tree, for example: "For advanced use cases, such as creating group snapshots of multiple volumes, see the external CSI Volume Group Snapshot documentation".

[xing-yang]

Thanks @spurin. This feature has a tracked/out-of-tree label. Since the code is completely out-of-tree, the release of the snapshot controller and csi-snapshotter will be done after a Kubernetes release.
That being said, it can be helpful to add a link referencing the out-of-tree csi docs.

[rashansmith]

Hello @xing-yang 👋 from the v1.32 Communications Team!
We'd love for you to consider writing a feature blog about your enhancement! Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt-in, let us know and open a Feature Blog placeholder PR against the website repository by 30th Oct 2024. For more information about writing a blog see the blog contribution guidelines.

[spurin]

Hi again @xing-yang,

Thanks for your feedback. Could you please submit a PR against the dev-1.32 branch in the k/website repo with the changes discussed - A helpful statement and a reference to the out-of-tree CSI documentation.

With this feature moving to Beta, I believe this sets a solid foundation for transitioning it to Stable in the future. I'm sure you're already very familiar with the process but for convenience, the steps are outlined here.

The deadline for this from a documentation viewpoint is Thursday Oct 24 at 18:00 PDT. Thanks! 🚀

[xing-yang]

Hi @spurin, it looks like the doc PR ready to review deadline is 11/19. I can have a placeholder PR, but I think it is unlikely we'll have all docs ready by then.

[spurin]

Hi @xing-yang, you mentioned that the documentation was going to be out-of-tree at the following locations:

repo: https://github.com/kubernetes-csi/docs
webpage: https://kubernetes-csi.github.io/docs/group-snapshot-restore-feature.html

The documentation request that I mentioned was specific to one of the small points we discussed, adding a cross reference from the Kubernetes website to the external feature, along the lines of -

"For advanced use cases, such as creating group snapshots of multiple volumes, see the external CSI Volume Group Snapshot documentation".

Hopefully this makes sense?

[xing-yang]

Hi @spurin, sounds good! How about this? kubernetes/website#48419

[xing-yang]

Hi @rashansmith, I submitted a placeholder PR for a blog here: kubernetes/website#48420

[shecodesmagic]

Hello @xing-yang 👋, Enhancements team here (again 😁 )

With all the implementation(code related) PRs (tracked/out-of-tree code) merged as per the issue description

• Update e2e test script and expand feature focus for volume group snapshots test-infra#33417
• Add new E2E job for volume group snapshots in KIND cluster test-infra#33567
• Add end-to-end tests for Volume Group Snapshot kubernetes#126326

This enhancement is now marked as tracked for code freeze for the v1.32 Code Freeze!

Please note that KEPs targeting stable need to have the status field marked as implemented in the kep.yaml file after code PRs are merged and the feature gates are removed.

[rashansmith]

Hello @xing-yang 👋, v1.32 Comms team here,

Thank you for creating a placeholder PR for a feature blog!

The deadline for completing blog content to be Ready for Review is Monday 25th November 2024 . For more information about writing a blog see the blog contribution guidelines.

Please feel free to reach out if you have any questions!

[dipesh-rawat]

/remove-label lead-opted-in

[jwebster7]

This is directly related to documentation for this feature. I can open a separate issue request if necessary. I saw kubernetes/website#49030, but the last activity on that was in January 2025.

I noticed a discrepancy between the suggested workflows regarding pre-provisioned VolumeGroupSnapshots between the 1.32 blog and the KEP.

For pre-provisioned VolumeGroupSnapshots, the 1.32 blog states:

To import a pre-existing volume group snapshot into Kubernetes, you must also import the corresponding individual volume snapshots.

Identify the individual volume snapshot handles, manually construct a VolumeSnapshotContent object first, then create a VolumeSnapshot object pointing to the VolumeSnapshotContent object. Repeat this for every individual volume snapshot.

Then manually create a VolumeGroupSnapshotContent object, specifying the volumeGroupSnapshotHandle and individual volumeSnapshotHandles already existing on the storage system.

This blog entry suggests that the user must create a VolumeSnapshot and VolumeSnapshotContent for each snapshot individual but "grouped" snapshot within the storage system. Then, the admin would "group" these snapshots (from Kubernetes' perspective) using the VolumeGroupSnapshotContent and VolumeGroupSnapshot objects.

The KEP states:

Admin can create a VolumeGroupSnapshotContent, specifying an existing VolumeGroupSnapshotHandle in the storage system and specifying a VolumeGroupSnapshot name and namespace. Then the user creates a VolumeGroupSnapshot that points to the VolumeGroupSnapshotContent name.

The controller will call the CSI GetVolumeGroupSnapshot method to retrieve all volumeSnapshotHandles in the Volume Group Snapshot from the storage system, create VolumeSnapshotContents pointing to the volumeSnapshotHandles. Then the controller will create VolumeSnapshots pointing to the VolumeSnapshotContents.

The KEP suggests the intended workflow would be that the SP discovers the snapshots in the storage system, then brings them into the K8s cluster via VolumeGroupSnapshotContent and VolumeGroupSnapshot. Then the snapshot controller would create the VolumeSnapshot and VolumeSnapshotContent objects specified within the VolumeGroupSnapshot.

tl;dr

• The 1.32 blog states users must create individual VolumeSnapshot and VolumeSnapshotContents before creating a VolumeGroupSnapshotContent and VolumeGroupSnapshot CR.
• The workflow in the KEP states that admins can use pre-provisioned group snapshots by creating a VolumeGroupSnapshotContent and VolumeGroupSnapshot and the snapshot controller will create the child objects (VolumeSnapshot and VolumeSnapshotContent).

Questions:

• Which of these documented workflows is correct?
• The blog workflow opens the possibility for admins to misrepresent a set of snapshots as a VolumeGroupSnapshot.
  • For example, I could have 1 snapshot taken days before another that I then group in Kubernetes.
  • Would the responsibility fall on the SP to ensure all snapshots are taken at the same point in time when they're grouped?
• Is there a use case where users would want to group of a set of pre-provisioned snapshots in a Kubernetes cluster?
• What's the overall utility of the pre-provisioned workflow? Restores are intended to still happen on the individual snapshot level and it isn't clear why grouping them would be useful without a group restore capability. Is the only benefit restricting snapshot deletions?
  • https://kubernetes.io/blog/2024/12/18/kubernetes-1-32-volume-group-snapshot-beta/#how-to-use-group-snapshot-for-restore-in-kubernetes.

References:

• 1.32 VolumeGroupSnapshot Blog - https://kubernetes.io/blog/2024/12/18/kubernetes-1-32-volume-group-snapshot-beta/#importing-an-existing-group-snapshot-with-kubernetes
• KEP - https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3476-volume-group-snapshot#pre-provisioned-volumegroupsnapshot

[xing-yang]

@jwebster7 Thanks for bringing this up. The steps in the blog are correct. The KEP needs to be updated. I'll do that soon. In this doc, I captured how to test static volume group snapshot. Let me know if you have any questions.
For static provisioning, we do rely on admin to do the right things.