Skip to content

VolumeGroupSnapshot #3476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
7 of 8 tasks
xing-yang opened this issue Aug 25, 2022 · 54 comments · Fixed by #1551
Open
7 of 8 tasks

VolumeGroupSnapshot #3476

xing-yang opened this issue Aug 25, 2022 · 54 comments · Fixed by #1551
Assignees
Labels
sig/storage Categorizes an issue or PR as relevant to SIG Storage. stage/beta Denotes an issue tracking an enhancement targeted for Beta status tracked/out-of-tree Denotes an out-of-tree enhancement issue, which does not need to be tracked by the Release Team
Milestone

Comments

@xing-yang
Copy link
Contributor

xing-yang commented Aug 25, 2022

Enhancement Description

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Aug 25, 2022
@xing-yang
Copy link
Contributor Author

/sig storage

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 25, 2022
@xing-yang xing-yang self-assigned this Aug 30, 2022
@xing-yang
Copy link
Contributor Author

/milestone v1.26

@k8s-ci-robot k8s-ci-robot added this to the v1.26 milestone Sep 7, 2022
@xing-yang xing-yang added the lead-opted-in Denotes that an issue has been opted in to a release label Sep 7, 2022
@xing-yang xing-yang added the stage/alpha Denotes an issue tracking an enhancement targeted for Alpha status label Sep 14, 2022
@msau42 msau42 moved this to Waiting on Review in 1.26 release Sep 20, 2022
@msau42 msau42 linked a pull request Sep 20, 2022 that will close this issue
@msau42 msau42 linked a pull request Sep 20, 2022 that will close this issue
@msau42 msau42 moved this from KEP Waiting on Review to KEP Needs Work in 1.26 release Sep 20, 2022
@rhockenbury rhockenbury added the tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team label Sep 20, 2022
@ruheenaansari34
Copy link

ruheenaansari34 commented Sep 20, 2022

Hello @xing-yang 👋, 1.26 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT on Thursday 6th October 2022.

This enhancement is targeting for stage alpha for 1.26 (correct me, if otherwise).

Here's where this enhancement currently stands:

  • KEP file using the latest template has been merged into the k/enhancements repo.
  • KEP status is marked as implementable
  • KEP has a updated detailed test plan section filled out
  • KEP has up to date graduation criteria
  • KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would just need to update the following:

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

@shecodesmagic
Copy link

Hello @xing-yang 👋, v1.32 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage beta for v1.32 (correct me, if otherwise)

Here's where this enhancement currently stands:

  • KEP readme using the latest template has been merged into the k/enhancements repo.
  • KEP status is marked as implementable for latest-milestone: v1.32.
  • KEP readme has up-to-date graduation criteria
  • KEP has submitted a production readiness review request for approval and has a reviewer assigned.
  • KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would need to update the following:

  • KEP readme using the latest template has been merged into the k/enhancements repo.
  • KEP status is marked as implementable for latest-milestone: v1.32.
  • KEP has submitted a production readiness review request for approval and has a reviewer assigned.
  • KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

@shecodesmagic shecodesmagic moved this to At risk for enhancements freeze in 1.32 Enhancements Tracking Sep 28, 2024
@xing-yang
Copy link
Contributor Author

Hi @shecodesmagic, I have updated the KEP with the latest template. Can you please verify? Thanks!

@shecodesmagic
Copy link

With all the requirements fulfilled this enhancement is now marked as tracked for the upcoming enhancements freeze 🚀

@shecodesmagic shecodesmagic moved this from At risk for enhancements freeze to Tracked for code freeze in 1.32 Enhancements Tracking Oct 11, 2024
@shecodesmagic shecodesmagic moved this from Tracked for code freeze to Tracked for enhancements freeze in 1.32 Enhancements Tracking Oct 11, 2024
@spurin
Copy link

spurin commented Oct 14, 2024

Hi @xing-yang 👋, Great to see you here, I'm contributing as a 1.32 Docs Shadow.

Does this enhancement work planned for 1.32 require any new docs or modifications to the existing docs?

If so, please follows the steps here to open a PR against dev-1.32 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday October 24th 2024 18:00 PDT.

Also, take a look at Documenting for a release to familiarise with the docs requirement for the release.

Thank you!

@xing-yang
Copy link
Contributor Author

Hi @spurin, great to see you here too! This feature is implemented in the out-of-tree repo https://github.com/kubernetes-csi/external-snapshotter. The doc will be updated in this repo: https://github.com/kubernetes-csi/docs and it shows up here: https://kubernetes-csi.github.io/docs/group-snapshot-restore-feature.html. So we don't have in-tree docs work planned for 1.32. Thank you!

@spurin
Copy link

spurin commented Oct 14, 2024

Thanks @xing-yang, the referenced snapshot documentation is great. Can I please query some points areas for consideration and further discussion:

  • We're aiming for this to become Beta in the 1.32 release. In the documentation referenced I can see that this is currently listed as Alpha for 1.27. I'm assuming that will be updated to coincide with the 1.32 release. Is this external documentation update requirement being tracked under this KEP?

  • How do the in-tree VolumeSnapshotClass and out-of-tree VolumeGroupSnapshotClass features work together? Are there any considerations for users who need to manage both individual and group snapshots and are there any known limitations or dependencies between these two controllers? (Asking from the viewpoint of whether we need to consider this from an in-tree documentation viewpoint)

  • Given that this is transitioning to Beta, is it worthwhile cross-referencing this from the in-tree to the out-of-tree, for example: "For advanced use cases, such as creating group snapshots of multiple volumes, see the external CSI Volume Group Snapshot documentation".

@xing-yang
Copy link
Contributor Author

Thanks @spurin. This feature has a tracked/out-of-tree label. Since the code is completely out-of-tree, the release of the snapshot controller and csi-snapshotter will be done after a Kubernetes release.
That being said, it can be helpful to add a link referencing the out-of-tree csi docs.

@rashansmith
Copy link

Hello @xing-yang 👋 from the v1.32 Communications Team!
We'd love for you to consider writing a feature blog about your enhancement! Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt-in, let us know and open a Feature Blog placeholder PR against the website repository by 30th Oct 2024. For more information about writing a blog see the blog contribution guidelines.

@spurin
Copy link

spurin commented Oct 18, 2024

Hi again @xing-yang,

Thanks for your feedback. Could you please submit a PR against the dev-1.32 branch in the k/website repo with the changes discussed - A helpful statement and a reference to the out-of-tree CSI documentation.

With this feature moving to Beta, I believe this sets a solid foundation for transitioning it to Stable in the future. I'm sure you're already very familiar with the process but for convenience, the steps are outlined here.

The deadline for this from a documentation viewpoint is Thursday Oct 24 at 18:00 PDT. Thanks! 🚀

@xing-yang
Copy link
Contributor Author

Hi @spurin, it looks like the doc PR ready to review deadline is 11/19. I can have a placeholder PR, but I think it is unlikely we'll have all docs ready by then.

@spurin
Copy link

spurin commented Oct 18, 2024

Hi @xing-yang, you mentioned that the documentation was going to be out-of-tree at the following locations:

repo: https://github.com/kubernetes-csi/docs
webpage: https://kubernetes-csi.github.io/docs/group-snapshot-restore-feature.html

The documentation request that I mentioned was specific to one of the small points we discussed, adding a cross reference from the Kubernetes website to the external feature, along the lines of -

"For advanced use cases, such as creating group snapshots of multiple volumes, see the external CSI Volume Group Snapshot documentation".

Hopefully this makes sense?

@xing-yang
Copy link
Contributor Author

Hi @spurin, sounds good! How about this? kubernetes/website#48419

@xing-yang
Copy link
Contributor Author

Hi @rashansmith, I submitted a placeholder PR for a blog here: kubernetes/website#48420

@shecodesmagic
Copy link

Hello @xing-yang 👋, Enhancements team here (again 😁 )

With all the implementation(code related) PRs (tracked/out-of-tree code) merged as per the issue description

This enhancement is now marked as tracked for code freeze for the v1.32 Code Freeze!

Please note that KEPs targeting stable need to have the status field marked as implemented in the kep.yaml file after code PRs are merged and the feature gates are removed.

@shecodesmagic shecodesmagic moved this from Tracked for enhancements freeze to Tracked for code freeze in 1.32 Enhancements Tracking Oct 30, 2024
@weshayutin weshayutin added this to OADP Oct 30, 2024
@rashansmith
Copy link

Hello @xing-yang 👋, v1.32 Comms team here,

Thank you for creating a placeholder PR for a feature blog!

The deadline for completing blog content to be Ready for Review is Monday 25th November 2024 . For more information about writing a blog see the blog contribution guidelines.

Please feel free to reach out if you have any questions!

@dipesh-rawat
Copy link
Member

/remove-label lead-opted-in

@k8s-ci-robot k8s-ci-robot removed the lead-opted-in Denotes that an issue has been opted in to a release label Jan 13, 2025
@xing-yang xing-yang reopened this Feb 8, 2025
@jwebster7
Copy link

This is directly related to documentation for this feature. I can open a separate issue request if necessary. I saw kubernetes/website#49030, but the last activity on that was in January 2025.

I noticed a discrepancy between the suggested workflows regarding pre-provisioned VolumeGroupSnapshots between the 1.32 blog and the KEP.

For pre-provisioned VolumeGroupSnapshots, the 1.32 blog states:

To import a pre-existing volume group snapshot into Kubernetes, you must also import the corresponding individual volume snapshots.

Identify the individual volume snapshot handles, manually construct a VolumeSnapshotContent object first, then create a VolumeSnapshot object pointing to the VolumeSnapshotContent object. Repeat this for every individual volume snapshot.

Then manually create a VolumeGroupSnapshotContent object, specifying the volumeGroupSnapshotHandle and individual volumeSnapshotHandles already existing on the storage system.

This blog entry suggests that the user must create a VolumeSnapshot and VolumeSnapshotContent for each snapshot individual but "grouped" snapshot within the storage system. Then, the admin would "group" these snapshots (from Kubernetes' perspective) using the VolumeGroupSnapshotContent and VolumeGroupSnapshot objects.

The KEP states:

Admin can create a VolumeGroupSnapshotContent, specifying an existing VolumeGroupSnapshotHandle in the storage system and specifying a VolumeGroupSnapshot name and namespace. Then the user creates a VolumeGroupSnapshot that points to the VolumeGroupSnapshotContent name.

The controller will call the CSI GetVolumeGroupSnapshot method to retrieve all volumeSnapshotHandles in the Volume Group Snapshot from the storage system, create VolumeSnapshotContents pointing to the volumeSnapshotHandles. Then the controller will create VolumeSnapshots pointing to the VolumeSnapshotContents.

The KEP suggests the intended workflow would be that the SP discovers the snapshots in the storage system, then brings them into the K8s cluster via VolumeGroupSnapshotContent and VolumeGroupSnapshot. Then the snapshot controller would create the VolumeSnapshot and VolumeSnapshotContent objects specified within the VolumeGroupSnapshot.

tl;dr

  • The 1.32 blog states users must create individual VolumeSnapshot and VolumeSnapshotContents before creating a VolumeGroupSnapshotContent and VolumeGroupSnapshot CR.
  • The workflow in the KEP states that admins can use pre-provisioned group snapshots by creating a VolumeGroupSnapshotContent and VolumeGroupSnapshot and the snapshot controller will create the child objects (VolumeSnapshot and VolumeSnapshotContent).

Questions:

  • Which of these documented workflows is correct?
  • The blog workflow opens the possibility for admins to misrepresent a set of snapshots as a VolumeGroupSnapshot.
    • For example, I could have 1 snapshot taken days before another that I then group in Kubernetes.
    • Would the responsibility fall on the SP to ensure all snapshots are taken at the same point in time when they're grouped?
  • Is there a use case where users would want to group of a set of pre-provisioned snapshots in a Kubernetes cluster?
  • What's the overall utility of the pre-provisioned workflow? Restores are intended to still happen on the individual snapshot level and it isn't clear why grouping them would be useful without a group restore capability. Is the only benefit restricting snapshot deletions?

References:

@xing-yang
Copy link
Contributor Author

xing-yang commented Apr 17, 2025

@jwebster7 Thanks for bringing this up. The steps in the blog are correct. The KEP needs to be updated. I'll do that soon. In this doc, I captured how to test static volume group snapshot. Let me know if you have any questions.
For static provisioning, we do rely on admin to do the right things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/storage Categorizes an issue or PR as relevant to SIG Storage. stage/beta Denotes an issue tracking an enhancement targeted for Beta status tracked/out-of-tree Denotes an out-of-tree enhancement issue, which does not need to be tracked by the Release Team
Projects
Status: Net New
Status: Tracked
Status: Tracked for code freeze
Development

Successfully merging a pull request may close this issue.