mdraid member status not updated and transition to multi use Disk.role field. Fixes #1214

[phillxnet]

The initial commit in this pr addresses the issue of prior mdraid member's db entries being updated once they are no longer mdraid members. The rest of the pr is concerned with migrating the 'role' Disk field in the db to use the json format and to open it up to alternative purposes than simply as an mdraid member labelling systems.. Previously it was expected to contain either null (db default) or one of the following strings 'isw_raid_member' and 'linux_raid_member'. This pr prepares the role field for future use such as labelling drives for special purposes eg as backup or import devices where a role will be needed to label devices appropriately. So far the mdraid member status is fully maintained as this can be assessed via the current (and unchanged) scan_disks method by way of it's FSTYPE probe but provision has been made to preserve existing role entries, along side those of mdraid, over re-scans and device disconnect / reconnect cycles.

I will specify the testing employed to prove this mechanism in a separate post on this pr.

[phillxnet]

In order to prove that existing roles are maintained while mdraid role info is added / removed the following testing was carried out. It was necessary to 'hand' modify the db role field to mimic elements of Rockstor that do no yet exist, ie facilities that add or remove role labels. As currently scan_disks() and _update_disk_state() only cater for mdraid member status as the truth of this can be had from each scan_disks() run via the FSTYPE field of lsblk.

vda vdb test drives on a kvm.
Give one an additional (currently bogus) role via:
UPDATE storageadmin_disk SET role = '{"backup-task": "syncthing-backup-task"}' WHERE name = 'vdb';
The other has the db default of null for role.
Here system drive also has custom role to serve as a test member for non mdraid member devices.

select name, size, serial, role from storageadmin_disk;
 name |  size   |  serial  |                   role                   
------+---------+----------+------------------------------------------
 vda  | 2097152 | serial-7 | 
 sda3 | 7025459 | QM00005  | {"backup-task": "syncthing-backup-task"}
 vdb  | 2097152 | serial-9 | {"backup-task": "syncthing-backup-task"}
(3 rows)

Setup mdraid mirror of vda and vdb - both drives labelled as mdraid and existing role on one of the now mdraid members was maintained:
mdraid created via:

mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/vda /dev/vdb

and db contents checked via:

select name, size, serial, role from storageadmin_disk;
 name |  size   |               serial                |                                  role                                   
------+---------+-------------------------------------+-------------------------------------------------------------------------
 sda3 | 7025459 | QM00005                             | {"backup-task": "syncthing-backup-task"}
 vda  | 2097152 | serial-7                            | {"mdraid": "linux_raid_member"}
 md0  | 2097152 | 2d921a48:a4e5e7f5:f6091f18:903b6a04 | 
 vdb  | 2097152 | serial-9                            | {"mdraid": "linux_raid_member", "backup-task": "syncthing-backup-task"}

Remove the raid member with no other role (stopping the md0 first to allow this)
mdadm --stop /dev/md0
mdadm --zero-superblock /dev/vda
db updated accordingly to null as mdraid was the only member.

select name, size, serial, role from storageadmin_disk;
               name               |  size   |               serial                |                                  role                                   
----------------------------------+---------+-------------------------------------+-------------------------------------------------------------------------
 sda3                             | 7025459 | QM00005                             | {"backup-task": "syncthing-backup-task"}
 b8f1910a335443259698946222c0a950 | 2097152 | 2d921a48:a4e5e7f5:f6091f18:903b6a04 | 
 vda                              | 2097152 | serial-7                            | 
 vdb                              | 2097152 | serial-9                            | {"mdraid": "linux_raid_member", "backup-task": "syncthing-backup-task"}
(4 rows)

Then remove the other raid member’s superblock:-
mdadm --zero-superblock /dev/vdb
and Rescan disks.

select name, size, serial, role from storageadmin_disk;
               name               |  size   |               serial                |                   role                   
----------------------------------+---------+-------------------------------------+------------------------------------------
 vda                              | 2097152 | serial-7                            | 
 vdb                              | 2097152 | serial-9                            | {"backup-task": "syncthing-backup-task"}
 242d551bc30b4d95b98da3ac782d2b52 | 2097152 | 2d921a48:a4e5e7f5:f6091f18:903b6a04 | 
 sda3                             | 7025459 | QM00005                             | {"backup-task": "syncthing-backup-task"}
(4 rows)

Successfully left in place existing role on vdb.
Then re-create mdraid array using vda and vdb:

mdadm --create /dev/md0 --level=mirror --raid-devices=2 /dev/vda /dev/vdb

and then after Rescan of disks:

select name, size, serial, role from storageadmin_disk;
               name               |  size   |               serial                |                                  role                                   
----------------------------------+---------+-------------------------------------+-------------------------------------------------------------------------
 vda                              | 2097152 | serial-7                            | {"mdraid": "linux_raid_member"}
 md0                              | 2097152 | d3f61db7:48d5bfa5:a69b5857:898fc66d | 
 vdb                              | 2097152 | serial-9                            | {"mdraid": "linux_raid_member", "backup-task": "syncthing-backup-task"}
 c0ffdbfc1d7c45d786e7325b81a75807 | 2097152 | 2d921a48:a4e5e7f5:f6091f18:903b6a04 | 
 sda3                             | 7025459 | QM00005                             | {"backup-task": "syncthing-backup-task"}
(5 rows)

Mdraid role added to both members with one retaining it’s original example ‘backup-task’ role.
In this example we have the memory of the previously created mdraid array as our new one has a new uuid / serial.
Throughout all tests our sda3 role has been maintained as an example of role persistence in non mdraid members. Further check this feature for non system disk:

Added another drive vdc (virtio with serial=’serial-13’) and gave it a role:

UPDATE storageadmin_disk SET role = '{"backup-task": "data-backup-task"}' WHERE name = 'vdc';
UPDATE 1
select name, size, serial, role from storageadmin_disk;
               name               |  size   |               serial                |                                  role                                   
----------------------------------+---------+-------------------------------------+-------------------------------------------------------------------------
 sda3                             | 7025459 | QM00005                             | {"backup-task": "syncthing-backup-task"}
 0796ea2f86284f5691be5a38130ec32c | 2097152 | 2d921a48:a4e5e7f5:f6091f18:903b6a04 | 
 vdc                              | 3145728 | serial-13                           | {"backup-task": "data-backup-task"}
 vda                              | 2097152 | serial-7                            | {"mdraid": "linux_raid_member"}
 md0                              | 2097152 | d3f61db7:48d5bfa5:a69b5857:898fc66d | 
 vdb                              | 2097152 | serial-9                            | {"mdraid": "linux_raid_member", "backup-task": "syncthing-backup-task"}
(6 rows)

so the db now contains for this serial='serial-13' disk:

select name, size, serial, role from storageadmin_disk WHERE serial = 'serial-13';
 name |  size   |  serial   |                role                 
------+---------+-----------+-------------------------------------
 vdc  | 3145728 | serial-13 | {"backup-task": "data-backup-task"}
(1 row)

Removed the device, rescan, and we have:

select name, size, serial, role from storageadmin_disk WHERE serial = 'serial-13';
               name               |  size   |  serial   |                role                 
----------------------------------+---------+-----------+-------------------------------------
 36ab2856762843d793650585d4d49e13 | 3145728 | serial-13 | {"backup-task": "data-backup-task"}
(1 row)

Re-attach device, and rescan:
(N.B. this required a power down as VMM can’t add the device and set it’s serial in one go)

select name, size, serial, role from storageadmin_disk WHERE serial = 'serial-13';
 name |  size   |  serial   |                role                 
------+---------+-----------+-------------------------------------
 vdc  | 3145728 | serial-13 | {"backup-task": "data-backup-task"}
(1 row)

So previous role retained and proves disconnection / reconnection persistence of non system disk role info in db.

Test to convert from non json format in role field to json format:-
Mdraid on system disk install.

select name, size, serial, role from storageadmin_disk;
 name  |   size   |               serial                |       role        
-------+----------+-------------------------------------+-------------------
 vda   | 20971520 | 13579-serial                        | 
 md127 |  5976883 | a7c5c1e1:fb0ba5cd:bc9984cc:8d8f145b | 
 sda   |  8388608 | QM00005                             | linux_raid_member
 sdb   |  8388608 | QM00007                             | linux_raid_member
 sdc   |  2097152 | another-serial-6789                 | 
(5 rows)

Apply patch, re-scan (after restart):-

select name, size, serial, role from storageadmin_disk;
 name  |   size   |               serial                |              role               
-------+----------+-------------------------------------+---------------------------------
 vda   | 20971520 | 13579-serial                        | 
 md127 |  5976883 | a7c5c1e1:fb0ba5cd:bc9984cc:8d8f145b | 
 sda   |  8388608 | QM00005                             | {"mdraid": "linux_raid_member"}
 sdb   |  8388608 | QM00007                             | {"mdraid": "linux_raid_member"}
 sdc   |  2097152 | another-serial-6789                 | 
(5 rows)

So old plain string role format updated to json format for the same.

Rockstor version 3.8-13.09 - applied patch and successfully changed from legacy to json format.

select name, size, serial, role from storageadmin_disk;
 name  |   size   |               serial                |              role               
-------+----------+-------------------------------------+---------------------------------
 vda   | 20971520 | 13579-serial                        | 
 md127 |  5976883 | a7c5c1e1:fb0ba5cd:bc9984cc:8d8f145b | 
 sda   |  8388608 | QM00005                             | {"mdraid": "linux_raid_member"}
 sdb   |  8388608 | QM00007                             | {"mdraid": "linux_raid_member"}
 sdc   |  2097152 | another-serial-6789                 | 
(5 rows)

Also confirmed expected function on existing partitioned drives eg with ext4 fs on, ie cog in webUI.

Testing on bios raid install, ie rockstor installed on 2x80GB bios raid disks:
Before:

storageadmin=# select name, size, serial, role from storageadmin_disk;
  name   |   size   |               serial                |      role       
---------+----------+-------------------------------------+-----------------
 md126p3 | 73610035 | d6245d0c:94128b48:374717e6:c1fd5ca0 | 
 sda     | 78118912 | WD-WMAM9X991187                     | 
 sdb     | 78118912 | WD-WMAET2305538                     | isw_raid_member
 sdc     | 78118912 | 6RW17C9A                            | isw_raid_member
(4 rows)

Apply patch and afterwards we have:

select name, size, serial, role from storageadmin_disk;
  name   |   size   |               serial                |             role              
---------+----------+-------------------------------------+-------------------------------
 sda     | 78118912 | WD-WMAM9X991187                     | 
 sdb     | 78118912 | WD-WMAET2305538                     | {"mdraid": "isw_raid_member"}
 sdc     | 78118912 | 6RW17C9A                            | {"mdraid": "isw_raid_member"}
 md126p3 | 73610035 | d6245d0c:94128b48:374717e6:c1fd5ca0 | 
(4 rows)

So successful bios raid plain string to json format in role field.

In all cases the WebUI Disks page displayed the expected icons.

[phillxnet]

@schakrava patch set ready for review.

[schakrava]

Thanks for explaining test scenarios in detail @phillxnet . Nicely done as always!

[phillxnet]

@schakrava Cheers.