Volumes: Add remote copy support for snapshots #12045

roosterfish · 2023-07-19T13:48:12Z

Adds support for copying volume snapshots to and from a remote.
Includes tests for local and remote volume snapshot copy scenarios.

Fixes #11112

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

tomponline · 2023-07-19T15:51:17Z

I've pinned curl in main now so you can rebase.

monstermunchkin · 2023-07-19T18:09:04Z

LGTM but code tests are failing.

roosterfish · 2023-07-20T07:44:45Z

LGTM but code tests are failing.

Thanks, cleared. Also the swagger doc update is now added.

tomponline

This is looking great, I like the thorough testing added.

There's a couple of minor changes needed, as this introduces an API change so we need to make an api extension and associated commit.

tomponline · 2023-07-20T15:30:12Z

client/lxd_storage_volumes.go

+			Target:    volume.Target,
+		}
+
+		path = fmt.Sprintf("/storage-pools/%s/volumes/custom/%s/snapshots/%s", url.PathEscape(pool), srcVolParentName, srcVolSnapName)


We should use url.PathEscape() on all path parameters.

I switched to using api.NewURL().Path().String() to remove the burden of having to call url.PathEscape() countless times.

Yes that is preferred way of doing it indeed.

tomponline · 2023-07-20T15:30:19Z

client/lxd_storage_volumes.go

+		path = fmt.Sprintf("/storage-pools/%s/volumes/custom/%s/snapshots/%s", url.PathEscape(pool), srcVolParentName, srcVolSnapName)
+	} else {
+		req = volume
+		path = fmt.Sprintf("/storage-pools/%s/volumes/custom/%s", url.PathEscape(pool), volume.Name)


We should use url.PathEscape() on all path parameters.

tomponline · 2023-07-20T15:33:59Z

shared/api/storage_pool_volume_snapshot.go

+	// Initiate volume snapshot migration
+	// Example: false
+	//
+	// API extension: storage_api_remote_volume_handling


This isnt the API extension that added this field. Instead it should be the name of the new API extension you add in this PR.

I was thinking a bit here but the concept of those extensions wasn't clear to me yet. Thanks. Just saw the client now also needs a check for r.HasExtension. Will add this in now.

We use r.CheckExtension. Please could you add a comment to r.HasExtension that it is considered deprecated.

tomponline · 2023-07-20T15:34:11Z

shared/api/storage_pool_volume_snapshot.go

+
+	// Migration target (for push mode)
+	//
+	// API extension: storage_api_remote_volume_handling


Should be the new API extension name.

test/suites/storage_snapshots.sh

client/lxd_server.go

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

vol.Snapshots() returns an error if the volume is a snapshot. Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

The extension needs to be present when trying to copy volume snapshots between remotes Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

tomponline

LGTM! Thanks!

github-actions bot added the API Changes to the REST API label Jul 19, 2023

roosterfish force-pushed the fix_snapshot_copy branch 5 times, most recently from 81f4777 to 7c5dd7f Compare July 19, 2023 15:37

lxc/volume: Fix docstrings to clarify volume operations

c291a05

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

roosterfish force-pushed the fix_snapshot_copy branch from 7c5dd7f to 45fcba3 Compare July 19, 2023 15:50

roosterfish force-pushed the fix_snapshot_copy branch from 45fcba3 to 33d521d Compare July 19, 2023 16:03

roosterfish force-pushed the fix_snapshot_copy branch 2 times, most recently from deb646b to a974583 Compare July 20, 2023 07:14

github-actions bot added the Documentation Documentation needs updating label Jul 20, 2023

roosterfish force-pushed the fix_snapshot_copy branch from b0cc4e3 to ee5537a Compare July 20, 2023 07:28

roosterfish force-pushed the fix_snapshot_copy branch from ee5537a to 370ed59 Compare July 20, 2023 08:54

tomponline requested changes Jul 20, 2023

View reviewed changes

roosterfish force-pushed the fix_snapshot_copy branch 5 times, most recently from 172549f to eaeca34 Compare July 21, 2023 08:05

monstermunchkin suggested changes Jul 21, 2023

View reviewed changes

client/lxd_server.go Outdated Show resolved Hide resolved

roosterfish added 6 commits July 21, 2023 11:15

lxd/client: Mark HasExtension as deprecated

d137d3f

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

lxd/storage/drivers/zfs: Allow volume snapshot refresh

5a34cdd

vol.Snapshots() returns an error if the volume is a snapshot. Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

api: Add storage_api_remote_volume_snapshot_copy extension

48ace10

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

lxd/shared/api: Extend StorageVolumeSnapshotPost for remote copy

dd23d9a

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

rest-api: Extend StorageVolumeSnapshotPost for remote copy

c94a3c2

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

lxc/volume: Use the right path when copying volume snapshots

92eee56

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

roosterfish added 5 commits July 21, 2023 11:15

lxc/volume: Check for storage_api_remote_volume_snapshot_copy extension

d79715e

The extension needs to be present when trying to copy volume snapshots between remotes Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

lxd/api/storage: Add support to copy volume snapshots between remotes

19524f4

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

lxc/volume: Block --volume-only flag when copying volume snapshots

e6bd9e2

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

tests: Check various volume snapshot copy operations

1e157db

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

tests: Check various volume snapshot remote copy operations

1e5e307

Signed-off-by: Julian Pelizäus <julian.pelizaeus@canonical.com>

roosterfish force-pushed the fix_snapshot_copy branch from eaeca34 to 1e5e307 Compare July 21, 2023 09:15

roosterfish requested review from monstermunchkin and tomponline July 21, 2023 10:16

tomponline approved these changes Jul 24, 2023

View reviewed changes

tomponline merged commit aca9153 into canonical:main Jul 24, 2023
25 checks passed

roosterfish deleted the fix_snapshot_copy branch July 24, 2023 09:13

roosterfish mentioned this pull request Jul 27, 2023

lxd: add all-projects param to the operation API #12078

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Volumes: Add remote copy support for snapshots #12045

Volumes: Add remote copy support for snapshots #12045

roosterfish commented Jul 19, 2023

tomponline commented Jul 19, 2023

monstermunchkin commented Jul 19, 2023

roosterfish commented Jul 20, 2023

tomponline left a comment

tomponline Jul 20, 2023

roosterfish Jul 21, 2023

tomponline Jul 21, 2023

tomponline Jul 20, 2023

tomponline Jul 20, 2023

roosterfish Jul 21, 2023

tomponline Jul 21, 2023

roosterfish Jul 21, 2023

tomponline Jul 20, 2023

tomponline left a comment

Volumes: Add remote copy support for snapshots #12045

Volumes: Add remote copy support for snapshots #12045

Conversation

roosterfish commented Jul 19, 2023

tomponline commented Jul 19, 2023

monstermunchkin commented Jul 19, 2023

roosterfish commented Jul 20, 2023

tomponline left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomponline left a comment

Choose a reason for hiding this comment