SMAPIv1 improvements for XCP-ng 8.2

[stormi]

There are several features that require improvements to the SMAPIv1 so they don't add technical debt and can be maintained confidently.

Features:

• Add Gluster driver
• Add CephFS driver
• Add linstor driver and related modifications
• Add a ZFS basic driver to avoid losing metadata : Create a dedicated ZFS SR driver to avoid losing VDI metadata #401

Related required improvements:

• Create maintenance branches in our sm fork, that will be the basis for creating the patches we'll apply to the sm RPM.
• Document their naming: Update develprocess.md xcp-ng-org#16
• Merge sm-additional-drivers into sm (source repo and RPM -- I can do the RPM part (@stormi))
• Update driver error messages to hint at missing packages (xfsprogs, glusterfs, ceph, linstor stuff...), since the drivers can be present without their dependencies. This is not a regression for the new drivers we are adding, but can be seen as one for the experimental xfs driver (sm-additional-drivers did pull xfs-progs). The situations where we can expect errors due to missing packages are:
  • attempts at creating a new SR
  • trying to use a SR after an upgrade (additional RPMs are not kept after an ISO upgrade)
    • Very important: when a required package is missing, we MUST make sure scanning the SR explicitly fails. It must not "believe" that the filesystem is appropriately mounted and consider that it's empty, else all metadata is wiped and lost.
      List of dependencies:
  • xfs: needs xfsprogs
  • gluster: needs glusterfs-server
  • cephfs: needs ceph-common
  • linstor: needs python-linstor (Package is direclty used, and if LINSTOR is not connected, an exception is always thrown. So It's not useful to test it.)
  • zfs: needs zfs
• Patch sm to let storage drivers give their cleanupType themselves rather than using a hardcoded correspondance list. That list is a pain to change each time we add a driver. => not easy to to because cleanup.py doesn't import the driver classes themselves.
  • So we'll just remove the monkey patching from the Gluster, Ceph and maybe other drivers and keep modifying the hardcoded correspondance in cleanup.normalizeType
• Update the list of supported drivers in xapi.conf. Ideally make that default list NOT belong in a user-modifiable config file like it is currently, because it's painful to update with rpm (need to modify it through a scriplet since it's %config(noreplace)). The * value seems to be a possible solution to let XAPI accept any SM driver that is present. See [question] Are there drawbacks to setting sm-plugins=* in xapi.conf? xapi-project/xen-api#4159. Else let's continue to use a RPM scriptlet and make sure we don't mess up.
• Rename classes in GlusterSR.py (they are not consistent with the file name and the driver identifier)
• Also check sm source code for other changes required to really 100% support the additional drivers. There were for ext4 that we never made, and could be for other drivers too. Example: get_o_direct_capability => moved to sm code update for the added drivers #423

• Contribute what can be contributed upstream to the XAPI project. => nothing probably as sm is "the past" and they're probably not very interested in adding drivers to it

[stormi]

For 8.2 beta, it would be good to test each driver at least once: zfs, glusterfs, xfs, cephfs. Basic features and specific changes from 8.2.

Non-exhaustive list of tests:

zfs

• SR creation fails if the zfs package is not present
• SR creation works if it's present
• SR activated after reboot
• migrate or import a VM to the SR and start it
• SR activation and scan fails if zfs packages are not present (start from a working setup then remove packages). The scan must not appear like it succeeded and the SR was empty.
• SR activation and scan fails if the ZFS mountpoint is not mounted (start from a working setup then "break" the mount). The scan must not appear like it succeeded and the SR was empty.
• Think of other situations where the scan could "work" despite there being an issue with the underlying FS and thus remove all VDI metadata because it thinks it's empty. Test them.
• SR destroy

glusterfs

• SR creation fails if the glusterfs-server package is not present
• SR creation works if it's present
• SR activated after reboot
• migrate or import a VM to the SR and start it
• SR activation and scan fails if glusterfs packages are not present (start from a working setup then remove packages). The scan must not appear like it succeeded and the SR was empty.
• Think of other situations where the scan could "work" despite there being an issue with the underlying FS and thus remove all VDI metadata because it thinks it's empty. Test them.
• SR destroy

xfs

• SR creation fails if the xfsprogs package is not present
• SR creation works if it's present
• SR activated after reboot
• migrate or import a VM to the SR and start it
• SR activation and scan fails if the xfsprogs package is not present (start from a working setup then remove the package). The scan must not appear like it succeeded and the SR was empty.
• SR activation and scan fails if the XFS mountpoint is not mounted (start from a working setup then "break" the mount). The scan must not appear like it succeeded and the SR was empty.
• SR destroy

cephfs

CC @rushikeshjadhav

• SR creation fails if the ceph-common package is not present
• SR creation works if the requirements are present
• SR activated after reboot
• migrate or import a VM to the SR and start it
• SR activation and scan fails if ceph-common packages are not present (start from a working setup then remove packages). The scan must not appear like it succeeded and the SR was empty.
• SR activation and scan fails if the CephFS mountpoint is not mounted (start from a working setup then "break" the mount). The scan must not appear like it succeeded and the SR was empty.
• Think of other situations where the scan could "work" despite there being an issue with the underlying FS and thus remove all VDI metadata because it thinks it's empty. Test them.
• SR destroy

ext4

• SR creation refused
• Existing SR created in 8.0 or earlier still works after an ISO upgrade
• SR destroy