Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lxd.migrate leaves snapshots in broken state #5828

Closed
lhorace opened this issue Jun 7, 2019 · 5 comments · Fixed by #5846

Comments

@lhorace
Copy link

commented Jun 7, 2019

Required information

  • Distribution: Ubuntu
  • Distribution version: 18.04
  • The output of "lxc info" or if that fails:
config:
  core.https_address: '[::]'
  core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 10.0.1.47:8443
  - 10.0.3.65:8443
  - 10.37.168.1:8443
  - 192.168.122.1:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFYTCCA0mgAwIBAgIQbFGsjT7WdyF+++2ViYbE0DANBgkqhkiG9w0BAQsFADA7
    MRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRswGQYDVQQDDBJyb290QGxp
    bnV4LWRlc2t0b3AwHhcNMTkwMzI4MTM0NDMwWhcNMjkwMzI1MTM0NDMwWjA7MRww
    GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRswGQYDVQQDDBJyb290QGxpbnV4
    LWRlc2t0b3AwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDmHFehF1Co
    hf4YT4XW1bbjS1i5Z2ikjCnxUc1A2/K1u++ED7En3oGvc7aqjjMiKdkeKy15FM7T
    MdW8JvogwN2ov2SUgdD5QrBfG8mm54Kgbsa1xRqtG6GqZyDMCEhMy8UezD3Uf5LP
    J7QCf7VcGzSbhWTJ02e8dOGhjxuXq6LIJnckgF22tNVl9DG6I1W5neORGjHbtTKI
    GzJR71pOQAawasuj/YEtgIq4BPMYWcLkhoaWz+9uhxvaJBWEC3SRrHmQbfA6F2+G
    hOkG+RhJ5s2aYRrXUhJTvbn0tojrgUi43gEqSMB3iE2XFRLB+S/v7FP5EyWJU/mf
    I6yQwsD67dH8HPdc7GOdqFt1GEf3lJSosbcc0ZiSVCmHIm/79SjMT9RgpxnYSSkw
    /aRBc+7rLJU5wR5K+UWqmi6Wl667OEXwsaTEfGmtne0OR1pXWhqcp8a+khXcZ9zF
    mz3Uucx5aSF44lf4OMt83ZRRsSdOZ6er0lXAek5nwPc6+gC2lohvQmuu6tIycMLc
    lITUBnXJVvwQdF73tft5JsVi3RBAv+o3BDH1a4ni6yvkihz1rinBFnwVHQZi1pbw
    AsUta1GN9BpYeByfGCwuTi3xdMGu5Y2vekLD/DzNmUHzQarqLnrSFBT4WYU2e/W2
    qsnkvWolrOAfiOib7CT7K2azSpGw0jjOOwIDAQABo2EwXzAOBgNVHQ8BAf8EBAMC
    BaAwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAqBgNVHREEIzAh
    gg1saW51eC1kZXNrdG9whwQKAAEvhwQKAAM8hwTAqHoBMA0GCSqGSIb3DQEBCwUA
    A4ICAQBRLSjAZQux+Bg02hbfgHNIirPo2uBjMIw3FsSGDVt7rWFmxdpZyybTgyHk
    E+a3GZt4pT9c8CTexP5Td9biaT3sw99uI4bExcurQNGBUaNY94B31lmQFVxvthrP
    +Ov5J7B8+hnPP+Im9iK0aGZNQT6q2oyl+Hpy0M6tpmVT8p+3kcvtmcUVTw8d/l5y
    b3q2uY6gJQLHgYP9AfVFLF8b2PyeCixsu5PT7vtfVuefgFPb/Q5qZvmRa8T6Min3
    ECSRgj8h64mSs/xvxUMFdkuc3cGiFYnpN/x6nZsPizPiXZgMUi9bdN1tV5/M1V5d
    WQhK7gCpPs0hZ6FdGLblJvXGlBRKPKwt1FlIyln94tyeWvACLjvnly1alBZmZSVV
    VXw1W2MtghkV1G1zbsF8d4fYHxtvpLblipS/nqMuTO0WeB2gweGGVNAZx8j+y8QH
    8c1+NigL7VVGCqGbrmxdwOoaYBUYDtT1JBCz4bLf6yKqW6shz+KawL1z304ao0EU
    8lSJrbqTtnrPI8RXzTqhRs64V5fYHUmY1ghAziMBUxcj26ryun7bznF1JkOJA7rD
    Gun+VRK6rmjm425QX4/mpqb6DM00Cd8RJ/dai1cs3rpLr2sNeN5/aD6k0GlzqrCy
    sdYhDVrsfTXIOWo/1WZgZywaNksm2pVwhWZ3TYfF/AspPkdXgA==
    -----END CERTIFICATE-----
  certificate_fingerprint: 40649f6d73d904fdee6bd2f36c73f13f4a6b895d87b428b6d996e0b58f40f7a2
  driver: lxc
  driver_version: 3.1.0
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.1.4-xanmod3
  lxc_features:
    mount_injection_file: "true"
    network_gateway_device_route: "false"
    network_ipvlan: "false"
    network_l2proxy: "false"
    seccomp_notify: "false"
  project: default
  server: lxd
  server_clustered: false
  server_name: linux-desktop
  server_pid: 5940
  server_version: "3.13"
  storage: btrfs
  storage_version: "4.4"

Issue description

Ran lxd.migrage after I installed LXD from snap and didn't observer any problem during the operation. However, when I was working with a container a moment ago and attempted to restore to an earlier snapshot:

Error: rename /var/snap/lxd/common/lxd/storage-pools/storage/containers-snapshots/alpine39-dev/pre-setprep /var/snap/lxd/common/lxd/storage-pools/storage/containers-snapshots/alpine39-dev/pre-setprep.ro: no such file or directory
architecture: x86_64
config:
  boot.autostart: "no"
  image.architecture: amd64
  image.description: Alpine 3.9 amd64 (20190414_13:00)
  image.os: Alpine
  image.release: "3.9"
  image.serial: "20190414_13:00"
  limits.cpu: "2"
  limits.memory: 1740MB
  volatile.base_image: 197c02508fb0f3cee1fa2829e414172f722a17372e861dd913a876271ab27742
  volatile.eth0.hwaddr: 00:16:3e:65:2e:6c
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":165536,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":165536,"Nsid":0,"Maprange":65536}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":165536,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":165536,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":165536,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":165536,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: STOPPED
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: virbr1
    type: nic
  root:
    path: /
    pool: storage
    type: disk
name: default
used_by:
- /1.0/containers/debian9-dev
- /1.0/containers/debian9-dev/before-sysprep
- /1.0/containers/ubuntu1804-dev
- /1.0/containers/oracle7-dev
- /1.0/containers/oracle7-dev/before-sysprep
- /1.0/containers/oracle7-dev/after-sysprep
- /1.0/containers/debian9-dev/before-sysprep-fixedhostname
- /1.0/containers/debian9-dev/after-sysprep
- /1.0/containers/ubuntu1804-dev/before-sysprep
- /1.0/containers/ubuntu1804-dev/after-sysprep-fixed
- /1.0/containers/alpine39-dev
- /1.0/containers/alpine39-dev/pre-setprep
- /1.0/containers/rainloop-stag
- /1.0/containers/rainloop-stag/before-sysprep
- /1.0/containers/rainloop-stag/after-sysprep
- /1.0/containers/archlinux-dev
- /1.0/containers/archlinux-dev/before-sysprep

+---------+-------------+--------+--------------------------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |                   SOURCE                   | USED BY |
+---------+-------------+--------+--------------------------------------------+---------+
| lxd     |             | btrfs  | /var/snap/lxd/common/lxd/storage-pools/lxd | 0       |
+---------+-------------+--------+--------------------------------------------+---------+
| storage |             | btrfs  | /var/lib/snapd/hostfs/storage/lxd          | 35      |
+---------+-------------+--------+--------------------------------------------+---------+
config:
  source: /var/lib/snapd/hostfs/storage/lxd
  volatile.initial_source: /storage/lxd
description: ""
name: storage
driver: btrfs
used_by:
- /1.0/containers/alpine39-dev
- /1.0/containers/alpine39-dev/snapshots/pre-setprep
- /1.0/containers/archlinux-dev
- /1.0/containers/archlinux-dev/snapshots/before-sysprep
- /1.0/containers/debian9-dev
- /1.0/containers/debian9-dev/snapshots/after-sysprep
- /1.0/containers/debian9-dev/snapshots/before-sysprep
- /1.0/containers/debian9-dev/snapshots/before-sysprep-fixedhostname
- /1.0/containers/lxdmosaic
- /1.0/containers/lxdmosaic/snapshots/After sys_prep
- /1.0/containers/lxdmosaic/snapshots/after-lxdmosaic-configuration
- /1.0/containers/lxdmosaic/snapshots/commit-13ccb65-dnssslfix
- /1.0/containers/lxdmosaic/snapshots/commit-13ccb65-mysqlfix-with-remotegit
- /1.0/containers/lxdmosaic/snapshots/commit-13ccb65c07b18641768d9f970401d5d10e39afe6
- /1.0/containers/lxdmosaic/snapshots/commit-591d7d6
- /1.0/containers/lxdmosaic/snapshots/commit-591d7d6-from-v0.3.0
- /1.0/containers/lxdmosaic/snapshots/v0.2.0-from-commit-13ccb65-dnssslfix
- /1.0/containers/lxdmosaic/snapshots/v0.3.0-from-v0.2.0
- /1.0/containers/lxdmosaic/snapshots/v020
- /1.0/containers/oracle7-dev
- /1.0/containers/oracle7-dev/snapshots/after-sysprep
- /1.0/containers/oracle7-dev/snapshots/before-sysprep
- /1.0/containers/rainloop-stag
- /1.0/containers/rainloop-stag/snapshots/after-sysprep
- /1.0/containers/rainloop-stag/snapshots/before-sysprep
- /1.0/containers/ubuntu1804-dev
- /1.0/containers/ubuntu1804-dev/snapshots/after-sysprep-fixed
- /1.0/containers/ubuntu1804-dev/snapshots/before-sysprep
- /1.0/images/071cb9ea1f69458e7626ecb507c7d30cf1cf5a545bab381ac444d09d324d2b8f
- /1.0/images/197c02508fb0f3cee1fa2829e414172f722a17372e861dd913a876271ab27742
- /1.0/images/6700bee14eb3034ba4bd0c3d0165f938faa161d2690e919465aab2946490689b
- /1.0/images/b3fa37f55bb4244619202f9f38171efbacf133f6a2b339ad469ef68e92869f91
- /1.0/images/c4672d3f782457ae2369f89c45f78838875c2298e92b76b1fbb2f936bf9b3a38
- /1.0/profiles/default
- /1.0/profiles/staticip
status: Created
locations:
- none

Steps to reproduce

  1. snap install --stable lxd
  2. lxd.migrate
  3. lxc restore

Information to attach

The problem seem to stem from, there is no container-snapshots under /storage/lxd ? Workaround for this is to symlink container-snapshots to snapshots? Is this intended behaviour?

Regards

@stgraber

This comment has been minimized.

Copy link
Member

commented Jun 7, 2019

Hmm, kinda feels like a missing migration step when upgrading from 3.0.x to 3.13.

@monstermunchkin can you look into this one? I think we should:

  • Setup a LXD 3.13 with btrfs and confirm that snapshots land in containers-snapshots
  • Setup a LXD 3.0.3 with btrfs and confirm that snapshots land in snapshots
  • Upgrade the 3.0.3 to 3.13 the "normal" way (so no snap involved, just start the 3.13 binary) and confirm we see this issue post-upgrade for containers that have snapshots pre-upgrade
  • Add needed patch to rename snapshots to containers-snapshots on upgrade
  • Re-try the upgrade from 3.0.3 and confirm that snapshots behave then

@stgraber stgraber added Bug Easy labels Jun 7, 2019

@stgraber stgraber added this to the lxd-3.14 milestone Jun 7, 2019

@monstermunchkin

This comment has been minimized.

Copy link
Member

commented Jun 7, 2019

Setup a LXD 3.13 with btrfs and confirm that snapshots land in containers-snapshots

Confirmed.

Setup a LXD 3.0.3 with btrfs and confirm that snapshots land in snapshots

Confirmed.

Upgrade the 3.0.3 to 3.13 the "normal" way (so no snap involved, just start the 3.13 binary) and confirm we see this issue post-upgrade for containers that have snapshots pre-upgrade

I cannot confirm this. The directory snapshots is renamed to containers-snapshots, and the symlink is updated accordingly.

@lhorace

This comment has been minimized.

Copy link
Author

commented Jun 7, 2019

Upgrade the 3.0.3 to 3.13 the "normal" way (so no snap involved, just start the 3.13 binary) and confirm we see this issue post-upgrade for containers that have snapshots pre-upgrade

I cannot confirm this. The directory snapshots is renamed to containers-snapshots, and the symlink is updated accordingly.

Are there differences between migrating from repository to snap with lxd.migrate and the normal way?

ls -l /storage/lxd
drwxr-xr-x 1 root root 166 Jun  7 04:28 containers
lrwxrwxrwx 1 root root   9 Jun  7 03:24 containers-snapshots -> snapshots
drwxr-xr-x 1 root root   0 Mar 28 10:39 custom
drwxr-xr-x 1 root root 640 Jun  7 04:11 images
drwxr-xr-x 1 root root 166 Jun  7 03:46 snapshots

Bear in mind I had to manually create the symlink above.

With that said, I like to point out possible differences from your setup: a) /storage is a mounted disk and b) /storage/lxd is a btrfs subvolume. Why you may not be able to replicate it? If I was aware snapshots were broken when I completed lxd.migrate. I would have kept the output because something seems to be amiss here, a silent fail?

@stgraber

This comment has been minimized.

Copy link
Member

commented Jun 10, 2019

There shouldn't really be a difference. The way we handle the migration is by moving /var/lib/lxd over to /var/snap/lxd/common/lxd, rewriting a few database entries which included hardcoded paths and then starting LXD, LXD on startup then runs all the DB upgrades and patches as a normal update would.

stgraber added a commit to stgraber/lxd that referenced this issue Jun 13, 2019
lxd/patches: Fix handling of containers-snapshots
Closes lxc#5828

Signed-off-by: Stéphane Graber <stgraber@ubuntu.com>
@stgraber

This comment has been minimized.

Copy link
Member

commented Jun 13, 2019

@monstermunchkin the problem is that the migration patches do not trigger mounting the storage pools, as a result only those which were mounted at the time get transitioned.

Got a fix for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
You can’t perform that action at this time.