[FEATURE] Application consistent snapshot/backup, Volume Group

[yasker]

Is your feature request related to a problem? Please describe.
Currently, Longhorn can do crash-consistent snapshot/backup. But for the applications like databases, quiescing is needed to create an application-consistent snapshot, so the application will make sure all the data in the memory has been written to the disk before creating the snapshot.

Describe the solution you'd like
Provide a way for the users to quiesce the workload before taking the snapshot/backup.

Describe alternatives you've considered
Users can also script the backup using CSI snapshotter and run quiescing/unquescing before taking the snapshot. But it won't work with the recurring snapshot/backup which is scheduled by Longhorn.

Additional context
One application is normally composed of multiple workloads, so we need to consider volume group scenario as well.

[innobead]

@jenting Please help with this. Thanks.

[joshimoo]

linking the backup refactor issue #1761

[janeczku]

Application state may be distributed across multiple persistent volumes (e.g. sharded DB, distributed indexes). The application consistent backup must then ensure that snapshots are performed in a point-in-time consistent manner across all volumes.

Offering such feature Longhorn may allow users to create "volume groups" and define on-demand or scheduled backup plans for the whole group instead of individual volumes.

[innobead]

Application state may be distributed across multiple persistent volumes (e.g. sharded DB, distributed indexes). The application consistent backup must then ensure that snapshots are performed in a point-in-time consistent manner across all volumes.

Offering such feature Longhorn may allow users to create "volume groups" and define on-demand or scheduled backup plans for the whole group instead of individual volumes.

Thanks for the comment. The similar idea we had some discussion before. The goal is we need to introduce a mechanism to make the volumes of application atomic aware operations supported and like u said, not just individual volumes.

Please follow up the discussion and proposals in near future.

[innobead]

Hey team! Please add your planning poker estimate with ZenHub @jenting @PhanLe1010 @shuo-wu @joshimoo

[joshimoo]

There is a lot to consider here and high uncertainty, volume groups, pre/post hooks (executing inside of the workload pods), error handling etc. Now that we have backup crds we can also consider exposing (syncing) these crds as csi snapshots via creation of the appropriate csi snapshot and snapshotData resources.

[shuo-wu]

Leave a note here:
Velero relies on annotation hooks to execute the cmd before/after backup. I can do something similar to archive application-consistent snapshot/backup. (run sync and freezes before the snapshot/backup)

The next:
Investigate Kanister

[R-Studio]

This feature would be awesome! 👍🏽😃
@innobead can be estimated when this feature will be implemented? (I ask because it is a open feature request since 21 Dec 2020)

[innobead]

ref: kubernetes/enhancements#3476

[innobead]

ref: kubernetes/enhancements#3476

container-storage-interface/spec#519
https://github.com/container-storage-interface/spec/releases/tag/v1.8.0 (introduce VolumeGroupSnapshot related RPCs under new GroupController services as alpha)

[innobead]

Some interesting project can be referenced as well, https://github.com/kanisterio/kanister.

[c3y1huang]

container-storage-interface/spec#519 https://github.com/container-storage-interface/spec/releases/tag/v1.8.0 (introduce VolumeGroupSnapshot related RPCs under new GroupController services as alpha)

Note:

• The Kubernetes VolumeGroupSnapshot supports PVC only and is triggered by the PVC label group=<group_name>. This means a PVC can only be associated with a single group, this is probably not ideal for RWX PVCs where it might be used by multiple workloads.
• The most recent release of kubernetes-csi/external-snapshotter, version v6.2.2, doesn't have the implementation of the volume group snapshot APIs. This feature currently sits in the master branch.

[innobead]

container-storage-interface/spec#519 https://github.com/container-storage-interface/spec/releases/tag/v1.8.0 (introduce VolumeGroupSnapshot related RPCs under new GroupController services as alpha)

Note:

• The Kubernetes VolumeGroupSnapshot supports PVC only and is triggered by the PVC label group=<group_name>. This means a PVC can only be associated with a single group, this is probably not ideal for RWX PVCs where it might be used by multiple workloads.

This new feature seems similar to the current VolumeSnapshot which means users also can do RWX volume backup by multiple workloads now, so it's not related to this new interface. This is a general interface, so basically we don't need to worry about whether it's ideal or not for different volume access modes.

• The most recent release of kubernetes-csi/external-snapshotter, version v6.2.2, doesn't have the implementation of the volume group snapshot APIs. This feature currently sits in the master branch.

I see, then we have two phases here. In the first phase (1.6), let's introduce the internal mechanism like AppBackup and AppRestore or VolumeGroupBackup and VolumeGroupRestore if we want to make it more general. In the second phase (after 1.6) when VolumeGroupSnapshot is ready upstream, we can do the same backup & restore but the trigger point is from the CSI path instead of creating AppBackup or restore by users or recurring jobs.

To sum up.

• The creator/driver of AppBackup will be a user, recurring job, or CSI. (DR should be naturally the same)
• The creator of AppRestore will be a user or CSI.

[c3y1huang]

Backlog this first to work on higher-priority items:

• [BUG] Somehow the Rebuilding field inside volume.meta is set to true when one replica only, causing the volume into attaching/detaching loop  #6626
• area/negative-testing