Copy from volume snapshot to remote fails: "Error: not found"

[bazaah]

Required information

• Distribution: Arch Linux
• Distribution version: -
• The output of "lxc info" or if that fails:

lxc info

config: <snip>
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: <snip>
  architectures:
  - x86_64
  - i686
  certificate: <snip>
  certificate_fingerprint: <snip>
  driver: qemu | lxc
  driver_version: 7.1.0 | 5.0.1
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.0.7-arch1-1
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Arch Linux
  os_version: ""
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: <snip>
  server_pid: 603
  server_version: "5.7"
  storage: ceph
  storage_version: 17.2.5
  storage_supported_drivers:
  - name: lvm
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.47.0
    remote: false
  - name: btrfs
    version: "6.0"
    remote: false
  - name: ceph
    version: 17.2.5
    remote: true
  - name: cephfs
    version: 17.2.5
    remote: true
  - name: dir
    version: "1"
    remote: false

Coping from a volume snapshot does not work

I am attempting to copy a volume snapshot to another LXD instance, using the schema provided by the CLI:

Usage:
  lxc storage volume copy [<remote>:]<pool>/<volume>[/<snapshot>] [<remote>:]<pool>/<volume> [flags]

Specially: lxc storage volume copy $A:$pool/$volume/snap0 $B:$pool/$volume

This fails because it attempts to hit an endpoint that doesn't exist (or, isn't documented at any rate):

DEBUG  [2022-11-12T15:59:16Z] Connected to the websocket: wss://$A/1.0/events
DEBUG  [2022-11-12T15:59:16Z] Sending request to LXD                        etag= method=POST url="https://$A/1.0/storage-pools/$pool/volumes/custom/$volume/snap0"

See https://linuxcontainers.org/lxd/docs/master/api/#/storage, search storage-pools/{name}/volumes/{type}/{volume} and note the lack of a /{snapshot} endpoint.

Steps to reproduce

1. Create 2 LXD clusters
2. Create a custom volume
3. Snapshot volume
4. Attempt copy snapshot to second cluster: lxc storage volume copy $A:$pool/$volume/snap0 $B:$pool/$volume

Information to attach

--debug output of copy attempt

❯ lxc storage volume copy $A:$pool/$volume/snap0 $B:$pool/$volume
DEBUG  [2022-11-12T16:13:59Z] Connecting to a remote LXD over HTTPS
DEBUG  [2022-11-12T16:13:59Z] Sending request to LXD                        etag= method=GET url="https://$A/1.0"
DEBUG  [2022-11-12T16:13:59Z] Got response struct from LXD
DEBUG  [2022-11-12T16:13:59Z]
        {
                "config": "<snip>",
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode",
                        "resources_pci_vpd",
                        "qemu_raw_conf",
                        "storage_cephfs_fscache",
                        "network_load_balancer",
                        "vsock_api",
                        "instance_ready_state",
                        "network_bgp_holdtime",
                        "storage_volumes_all_projects",
                        "metrics_memory_oom_total",
                        "storage_buckets",
                        "storage_buckets_create_credentials",
                        "metrics_cpu_effective_total",
                        "projects_networks_restricted_access",
                        "storage_buckets_local",
                        "loki",
                        "acme",
                        "internal_metrics",
                        "cluster_join_token_expiry",
                        "remote_token_expiry"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [
                                "<snip>"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "<snip>",
                        "certificate_fingerprint": "<snip>",
                        "driver": "lxc | qemu",
                        "driver_version": "5.0.1 | 7.1.0",
                        "firewall": "xtables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "idmapped_mounts": "true",
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "shiftfs": "false",
                                "uevent_injection": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "6.0.7-arch1-1",
                        "lxc_features": {
                                "cgroup2": "true",
                                "core_scheduling": "true",
                                "devpts_fd": "true",
                                "idmapped_mounts_v2": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "Arch Linux",
                        "os_version": "",
                        "project": "default",
                        "server": "lxd",
                        "server_clustered": false,
                        "server_event_mode": "full-mesh",
                        "server_name": "<snip>",
                        "server_pid": 603,
                        "server_version": "5.7",
                        "storage": "ceph",
                        "storage_version": "17.2.5",
                        "storage_supported_drivers": [
                                {
                                        "Name": "lvm",
                                        "Version": "2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.47.0",
                                        "Remote": false
                                },
                                {
                                        "Name": "btrfs",
                                        "Version": "6.0",
                                        "Remote": false
                                },
                                {
                                        "Name": "ceph",
                                        "Version": "17.2.5",
                                        "Remote": true
                                },
                                {
                                        "Name": "cephfs",
                                        "Version": "17.2.5",
                                        "Remote": true
                                },
                                {
                                        "Name": "dir",
                                        "Version": "1",
                                        "Remote": false
                                }
                        ]
                }
        }
DEBUG  [2022-11-12T16:13:59Z] Connecting to a remote LXD over HTTPS
DEBUG  [2022-11-12T16:13:59Z] Sending request to LXD                        etag= method=GET url="https://$B/1.0"
DEBUG  [2022-11-12T16:13:59Z] Got response struct from LXD
DEBUG  [2022-11-12T16:13:59Z]
        {
                "config": "<snip>",
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode",
                        "resources_pci_vpd",
                        "qemu_raw_conf",
                        "storage_cephfs_fscache",
                        "network_load_balancer",
                        "vsock_api",
                        "instance_ready_state",
                        "network_bgp_holdtime",
                        "storage_volumes_all_projects",
                        "metrics_memory_oom_total",
                        "storage_buckets",
                        "storage_buckets_create_credentials",
                        "metrics_cpu_effective_total",
                        "projects_networks_restricted_access",
                        "storage_buckets_local",
                        "loki",
                        "acme",
                        "internal_metrics",
                        "cluster_join_token_expiry",
                        "remote_token_expiry"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": "<snip>",
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "<snip>",
                        "certificate_fingerprint": "<snip>",
                        "driver": "qemu | lxc",
                        "driver_version": "7.1.0 | 5.0.1",
                        "firewall": "nftables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "idmapped_mounts": "true",
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "shiftfs": "false",
                                "uevent_injection": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "6.0.7-arch1-1",
                        "lxc_features": {
                                "cgroup2": "true",
                                "core_scheduling": "true",
                                "devpts_fd": "true",
                                "idmapped_mounts_v2": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "Arch Linux",
                        "os_version": "",
                        "project": "default",
                        "server": "lxd",
                        "server_clustered": false,
                        "server_event_mode": "full-mesh",
                        "server_name": "<snip>",
                        "server_pid": 675,
                        "server_version": "5.7",
                        "storage": "ceph",
                        "storage_version": "17.2.5",
                        "storage_supported_drivers": [
                                {
                                        "Name": "btrfs",
                                        "Version": "6.0",
                                        "Remote": false
                                },
                                {
                                        "Name": "ceph",
                                        "Version": "17.2.5",
                                        "Remote": true
                                },
                                {
                                        "Name": "cephfs",
                                        "Version": "17.2.5",
                                        "Remote": true
                                },
                                {
                                        "Name": "dir",
                                        "Version": "1",
                                        "Remote": false
                                },
                                {
                                        "Name": "lvm",
                                        "Version": "2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.47.0",
                                        "Remote": false
                                }
                        ]
                }
        }
DEBUG  [2022-11-12T16:13:59Z] Sending request to LXD                        etag= method=GET url="https://$A/1.0/storage-pools/$pool/volumes/custom/$volume"
DEBUG  [2022-11-12T16:13:59Z] Got response struct from LXD
DEBUG  [2022-11-12T16:13:59Z]
        {
                "config": "<snip>",
                "description": "",
                "name": "$volume",
                "type": "custom",
                "used_by": "<snip>",
                "location": "",
                "content_type": "filesystem",
                "project": "default"
        }
DEBUG  [2022-11-12T16:13:59Z] Sending request to LXD                        etag= method=GET url="https://$A/1.0/storage-pools/$pool/volumes/custom/$volume/snapshots/snap0"
DEBUG  [2022-11-12T16:13:59Z] Got response struct from LXD
DEBUG  [2022-11-12T16:13:59Z]
        {
                "description": "",
                "expires_at": "0001-01-01T00:00:00Z",
                "name": "snap0",
                "config": "<snip>",
                "content_type": "filesystem"
        }
DEBUG  [2022-11-12T16:13:59Z] Connected to the websocket: wss://$A/1.0/events
DEBUG  [2022-11-12T16:13:59Z] Sending request to LXD                        etag= method=POST url="https://$A/1.0/storage-pools/$pool/volumes/custom/$volume/snap0"
DEBUG  [2022-11-12T16:13:59Z]
        {
                "name": "$volume/snap0",
                "pool": "$pool",
                "migration": true,
                "target": null,
                "volume_only": false
        }
Error: not found

[tomponline]

Confirmed reproducer.

[tomponline]

Ah:

lxc/incus@70614bd#diff-29bc81fbd030cb3842d453034fe80625a788f0813d99453a0cbabe30c5b58055R348

[bazaah]

I guess this is a documentation issue then.

I don't specifically need this feature, as I can (and have) do just fine with faking LXD out via lxc storage volume create ... + rbd clone ... + rbd rename ..., swapping the underlying volume out from under LXD.

[tomponline]

No its a bug. As it works when doing between storage pools on same server.

[tomponline]

It seems the entire infrastructure for copying custom volume snapshots to a remote server is missing from the migration layer. So its not a small fix.