RFC: Incremental RBD backup provider (port to 4.23) — design + working prototype

[zidz]

TL;DR

I built an incremental RBD backup provider for CloudStack (port to 4.23.0.0).
It uses Ceph RBD snapshots + export-diff/import-diff so an incremental backup
is ~32x smaller than the full, and a restore replays the full→incremental chain.
Full + incremental + restore are verified end-to-end through the management API
against a real Ceph/KVM host. This is a prototype for design feedback, not a
finished PR — I'd like input on two open questions below.

Motivation

The existing NAS backup provider copies whole disk images every time. On Ceph
RBD primary storage we can do far better: take an RBD snapshot per backup and
export only the diff since the previous snapshot. Backups become small and fast,
and a restore is import (base) + import-diff (chain).

Design

• New plugin plugins/backup/rbd implementing BackupProvider.
• Full = rbd snap create + protect + export → <disk>.<vol>.rbd.
• Incremental = export-diff --from-snap <parent> → <disk>.<vol>.<ts>.rbdiff.
• Restore = import base + import-diff each link, truncated at the
  selected backup (restoring an earlier backup lands on that point-in-time).
• Chain metadata (chain id, position, parent snap) stored as backup details.
• Host-side engine: scripts/vm/hypervisor/kvm/rbdbackup.sh (called by the KVM
  agent wrappers, mirroring the NAS path).

What changed vs 4.23.0.0

• New (additive): plugins/backup/rbd/** (provider + tests + spring module),
  scripts/vm/hypervisor/kvm/rbdbackup.sh.
• Existing code (minimal, null-guarded):
  • core/.../backup/{Take,Delete,Restore}BackupCommand.java — added RBD fields
    (parent snap, base snap, rbd uri/snap, disk types). New Command fields only;
    rolling-upgrade safe (old agents ignore unknown fields).
  • kvm/.../Libvirt{Take,Delete,Restore}BackupCommandWrapper.java — route
    pure-RBD VMs to rbdbackup.sh; NAS path unchanged (guarded branch).
  • plugins/pom.xml + debian/rules — build & package the new module.

Enabling it today

createBackupOffering is currently hard-gated to the KBOSS provider, and
there's no createBackupRepository API, so the offering + repository must be
created via DB for now:

• set backup.framework.enabled=true, backup.framework.provider.plugin=rbd,
  rbd.backup.incremental.enabled=true
• insert a backup_repository (NFS) and a backup_offering (provider=rbd),
  linked by backup_offering.external_id = backup_repository.uuid
  Full step-by-step is in plugins/backup/rbd/README.md.

Verification (real Ceph/KVM host)

Scenario	Result
Full backup	.rbd object, ~5 GB
Incremental (no disk writes)	.rbdiff, ~132 B
Incremental (512 MB written)	.rbdiff, ~512 MB
Restore #1 / #2 / #3 of a 3-backup chain	lands on the correct point-in-time state each time
Restore-to-existing-VM	purges dest image, replays chain, data intact
restore-as-template	rebuilds chain into a new, freely-named image

Limitations / known issues

• Restore is bound to the original VM (framework limitation — restoreBackup
  resolves the VM from backup.getVmId()). For a new, custom-named VM use
  restore-as-template → register as template → deploy.
• Offering/repository creation is manual (KBOSS-gated API) — see open question 1.
• The new module is always built/loaded (plugins/pom.xml, debian/rules), so a
  defect there has global blast radius — see open question 2.

Open questions for the community

1. Offering API: should createBackupOffering be generalized beyond KBOSS so
   RBD (and other providers) can create offerings via API/UI, removing the manual
   DB step? Or is there a preferred existing mechanism I missed?
2. Packaging: is always-on module registration acceptable, or should the RBD
   provider be opt-in (profile/flag) to limit build/startup coupling?

Try it

Branch feature/rbd-incremental-backup-4.23 (6 commits on top of 4.23.0.0): compare link
Docs: plugins/backup/rbd/README.md.

Feel free to get inspiration from, use full or parts of this implementation if anyone have use for it in any way.

[weizhouapache]

@zidz
thanks for the sharing.
If I understand correctly, the same ceph RBD pool will be used as the backup repository, right ?

for your information, I am working on incremental backup of RBD volumes to NAS backup right now.
the code has some conflicts with #14194 and #14203, I will maybe wait until the two PRs to be merged.

[zidz]

The backup will be stored in the secondary storage.
If you are working on it, please go ahead. My fork is more of a working proof of concept to put the idea and examples out there.

[weizhouapache]

The backup will be stored in the secondary storage. If you are working on it, please go ahead. My fork is more of a working proof of concept to put the idea and examples out there.

@zidz
IMHO, all changes should be put in plugins/backup/kboss/. no extra jar is needed.
@JoaoJandre @winterhazel can give some advise

[JoaoJandre]

Hello, @zidz

Regarding your first two questions:

1. The createBackupOffering API can be generalized for other providers. I created it for KBOSS because it was needed. Currently, no other providers have any use for it so it is gated for KBOSS only. If you want to implement a new provider, I think it is a good idea to extend the createBackupOffering API.
2. If you are going to create a new provider, and it does not depend on any non-OSS libs, I would prefer it is always registered. No need for a new flag to build it.

Regarding extending KBOSS with support for RBD incremental backups. I think it would be a great idea, especially if you could make it support all the current workflows that KBOSS supports (backup creation, many types of restore, backup compression (if it makes sense for RBD), backup validation, backup extraction #13901). But even if you make it support only the basics (backup create, restore and delete) it would be great. The other workflows could be extended in the future.

However, if you are going to extend KBOSS, all code should be Java, as I do not believe that maintaining a bash script for a critical feature such as backups is a good idea, so you would have to change a lot of your current PR.

Regarding your process idea, I have one issue: From my testing, when exporting the base images, the whole image seems to be exported, even if most of it is empty. If I have a 100 GB disk with only 10 GB written, this process creates a 100 GB image in the secondary storage; Is there a way to avoid creating thick images?

[JoaoJandre]

Also, @zidz, If you decide to extend KBOSS and have any doubts, feel free to reach out to me.

[zidz]

Note that the rbd export is sparse/thin, it seems to be written as a whole when written to NFS. I guess something is happening when it's written to the secondary storage. This is a big issue actually and breaks the whole idea of the incremental backups... Will look into this.

I've noted your other comments and I will look into a potential refactor of this proof of concept and see if it materialize in something usable.

I will continue to follow up on comments and improve this over time.

[StepBee]

@zidz @weizhouapache Thats awesome work from both of you, exactly what i've been looking for.

At the moment i'm solving the requirement for incremental backups based on Ceph snapshot-diffs using external scripts - which unfortunately have no integration with Cloudstack so far.

So i am very excited about the work of both of you, thanks a lot!

[mathwest]

This is an incredibly important feature for us as well as it is essentially the only feature missing for us to have 100% parity with what we offered our customers in XO and then some with all the extra that cloudstack offers, Cant state enough how exited i am to see that some work is done on this front and i cannot thank you enough for your time.