MicroCloud should consciously set the `instances.migration.stateful` key

[roosterfish]

latest/edge MicroCloud always sets instances.migration.stateful=true which makes custom storage volumes from non-remote storage pools (dir, zfs, btrfs) unusable. This is especially painful for the local storage pool.

A user would need to explicitly set migration.stateful=false on every instance to which a custom storage volume from a non-remote pool should be attached.

To work around this there are several options:

• (1) Set instances.migration.stateful=true only in case no local storage got configured.
  • No additional action required by the user
  • Live migration is enabled on server level
  • Drawback:
    • If a local storage pool gets configured, a user would have to manually enable live migration on server level or per instance
• (2) Set instances.migration.stateful=true only in case remote storage got configured.
  • No additional action required by the user
  • Live migration is enabled on server level
  • Drawback:
    • No volumes from the local storage pool (if created) can be attached to any instance
    • If no remote storage pool gets configured, a user would have to manually enable live migration on server level or per instance
• (3) Don't set instances.migration.stateful
  • A user would have to manually enable live migration on server level or per instance (current state in MicroCloud 2 LTS)
• (4) Always set instances.migration.stateful=true (this is the current state in MicroCloud edge)
  • No additional action required by the user
  • Live migration is enabled on server level
  • Drawback:
    • No volumes from the local storage pool (if created) can be attached to any instance
    • Breaks lxc cluster recover (Setting instances.migration.stateful breaks cluster restore lxd#15218)

[tomponline]

Breaks lxc cluster recover (canonical/lxd#15218)

Perhaps we can modify the LXD logic to detect local volumes and instead of doing live migration do a non-live migration?

[tomponline]

unusable

Can you define unusable? It allows using the volumes, but prevents live migration right?

[tomponline]

@mionaalex wanted to ensure live migration was enabled by default so we need to be careful not to regress this product feature.

[roosterfish]

Can you define unusable? It allows using the volumes, but prevents live migration right?

IIRC if instances.migration.stateful=true is set, you cannot even attach a custom storage vol from any local storage pool as this will already raise an error. So you have to manually set migration.stateful=false on the instance itself.

[tomponline]

@roosterfish ah right, do you have reproducer steps so we can confirm not a bug?

[tomponline]

So you have to manually set migration.stateful=false on the instance itself.

but that sounds reasonable, if we want to ensure the VM is live migrateable by default, then doing anything to compromise that ability (like adding local volumes) would make sense to reject, and then one has to consider manually overriding the default of migration.stateful.

[roosterfish]

@roosterfish ah right, do you have reproducer steps so we can confirm not a bug?

The reproducer steps are already in canonical/lxd#15218. This one got closed as you can now skip the instances when restoring cluster members (canonical/lxd#15267).

but that sounds reasonable

I agree, but we should probably indicate this somewhere in the docs as it doesn't seem obvious. Maybe the LXD error message can propose this?

[minaelee]

Added the above PR to document the restriction in the How to manage storage volumes doc in LXD.