Revise internal use and format of device names #1320

Closed
phillxnet opened this Issue May 21, 2016 · 10 comments

Projects

None yet

3 participants

@phillxnet
Contributor
phillxnet commented May 21, 2016 edited

In order to increase the flexibility and utility of the db Disk.name field it is proposed that it be re-deployed to hold by-id device names as opposed to the current ‘sd’ type names. This in combination with the Disk.role field is intended to aid in supporting additional storage options such as LUKS full disk encryption.

Ie change:
/dev/sdX derived from a db entry of sdX
to:
/dev/disk/by-id/wwid-for-device derived from a db entry of wwid-for-device.

A World Wide Identifier (WWID) is a persistent and system independent identifier required of all SCSI standards compliant devices.
Reference:-
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/persistent_naming.html

@phillxnet
Contributor

I am currently working on this issue.

@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 22, 2016
@phillxnet phillxnet update Disk.name model comment to reflect new content #1320 a874106
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 22, 2016
@phillxnet phillxnet extend device by-id name helper to strip path if needed #1320
So as not to deal with paths in Disk.name db field we can
strip them first in our helper by way of a default flag that
leaves existing function intact.
583e395
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 22, 2016
@phillxnet phillxnet extend api/disk regex to include additional chars #1320
As /dev/disk/by-id names contain colons, underscores and
dashes we need to include these in our extended disk_regex
to accommodate the move to by-id names in the Disk.name
db field.
5df33e1
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 22, 2016
@phillxnet phillxnet use by-id disk names in db and consequent btrfs commands #1320
Initial conversion of disk name entries in db from 'sda' type to
those created by udev in the /dev/disk/by-id directory. Also
involves changing hard coded paths in low level btrfs functions.
12f193c
@phillxnet
Contributor
phillxnet commented May 22, 2016 edited
  • convert btrfs import mechanism to by-id name use.
  • convert APM level reporting to by-id name use.
  • convert Power State / spin down config and reporting to by-id name use.
  • convert SMART subsystem to by-id name use.
  • revise fake dev name for missing devices mechanism as now legitimate device name (by-id) and fake dev name collision is far more likely than before.
  • convert resize_pool to by-id name use.
  • resolve get_dev_byid_name returning null names ie for virtio without serial, this trips the db's "violates not-null constraint"
  • don't offer to use drives with no serial in pool creation wizard, as we can't track them.
  • don't collect SMART data on devices with no serial as since we key this to a name and with no serial that name is transient we end up with misdirected and consequently inappropriate / misleading data.
  • improve stability in multi by-id names for the same device as in some circumstances the names are reported in varying order. They always point to the same underlying (though transient) device name but greater utility would be had by stabilising the one we use: ie the longest.
  • debug disk removal from pool not 'sticking' ie disk gets removed but then shortly after the db is updated with on disk state which was not changed in the meantime. ie remove disk from pool not working.
  • don't offer to use drives with no serial in pool resize (add disks to pool) wizard.
  • address inefficiency in get_base_device_byid avoid using double conversion if we already have a by-id type device name, ie use surface syntactic method first then fall back to double conversion.
  • address various todo's concerning function location to better segregate by functional area and aid in future testing.
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 23, 2016
@phillxnet phillxnet update disk power state and hdparm setting readers to by-id #1320
Since the db now contains a by-id name convention we modify these
readers accordingly. Variable name re-factoring included for clarity.
f5bd733
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 23, 2016
@phillxnet phillxnet update disk pause function to by-id device name type #1320 ca15cc3
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 23, 2016
@phillxnet phillxnet update disk APM reader to by-id device name type #1320 1f8bf83
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 23, 2016
@phillxnet phillxnet update disk spin down config wrapper to by-id device name type #1320
Also update is_rotational comments with possible improvement and
make existing comments more consistent with current function.
2158553
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 23, 2016
@phillxnet phillxnet include by-id name in skipping hdparm settings message #1320
Previously we just logged a generic message with no device name
embedded. Improve log message by adding in dev name by-id
14d3f83
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 23, 2016
@phillxnet phillxnet update SMART options wrapper to by-id device name type #1320
Initially achieved by adding an additional wrapper to translate the
original get_base_device function so that it can be used with the new
by-id type name format.
Also corrected capitalization issue with recently added local variables.
c007d2e
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 23, 2016
@phillxnet phillxnet update pool import mechanism to by-id device name type #1320 05a8d8e
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 23, 2016
@phillxnet phillxnet update resize_pool mechanism to by-id device name type #1320 6f531d2
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 23, 2016
@phillxnet phillxnet remove some logging used in early stage of issue #1320 587219e
@phillxnet
Contributor
phillxnet commented May 24, 2016 edited

From fs/btrfs/scan_disks we have:
"# 12 chars (fake-serial-) + 36 chars (uuid4) = 48 chars"
Given our move to much long by-id type names we should no longer just use the length of this serial as a flag for a device with no original serial number.
Checked by handlebar helper checkSerialStatus storageadmin/static/storageadmin/js/views/disks.js

@phillxnet
Contributor
phillxnet commented May 24, 2016 edited

get_dev_byid_name looks the DEVLINKS line eg:-
DEVLINKS=/dev/disk/by-id/virtio-test-serial22222
in for example (for a virtio of vdd):-
udevadm info --query=property --name vdd
This is created by udev from the model "virtio" and the serial "test-serial22222"
In the case of a virtio without a serial ascribed to it we have no DEVLINKS as there is no serial for it to be drawn from!
This causes get_dev_byid_name to return None (by design) which has created the "violates not-null constraint" on Disk.name field. Previously we used the vdd device name but due to this missing DEVLINKS entry in the output of at least virtio devices without serial we need a graceful and meaningful failover.
I am leaning towards failing back, in this circumstance, to the original name.
Will pick this up again soon.

@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 24, 2016
@phillxnet phillxnet improve robustness of fake-serial- detection #1320
Previously we only checked for a length of 48, improve
this by looking specifically for scan_disks stamp on all
fake-serial-uuid4's that it generates.
e5d90da
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 24, 2016
@phillxnet phillxnet add debug logging for issue #1320
Used to monitor get_dev_byid_name function.
5a545d2
@phillxnet
Contributor

Disk.name by-id conversion mostly done and all disk and pool functions appear to be working, bar dealing with corner case serial issues.
I now just need to debugging (see above) and clean a few things up.

@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 25, 2016
@phillxnet phillxnet if no by-id type disk name then fail over to caller name #1320
Disks with no serial number are not given a DEVLINKS entry by
udev ie virtio with no serial. This causes an issue as we need a
kernel name of sorts for every disk. Failing over to the calling
devices as found by scan_disks and flagging via an additional
return value (a boolean) associated via a tuple return type.
We can't use these no-serial devices anyway but do need to
have 'a' name in order to store and flag them.
Minor formatting updates and comment updates included.
a7c7839
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 25, 2016
@phillxnet phillxnet remove redundant import #1320 1b4a8d3
@holmesb
holmesb commented May 25, 2016 edited

This will be a great improvement. For one thing it will make installing in a XenServer VM possible using the normal method. Xen uses /dev/xvdx rather /dev/sdx device names, so currently have to use a workaround.

@phillxnet
Contributor

@holmesb Thanks for chipping in more info on this one; but I'm afraid this won't have any effect on the actual install, sorry. But I'm hoping it will have other 'good' effects. At least that's my aim.

@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 25, 2016
@phillxnet phillxnet disable SMART function for drives with transient names #1320
As we index the smart data to drive names and now these names
are less transient we have a chance to ensure that the SMART
data collected for a given drive will stick with that drive by using
only non transient names to index that data. So when we are
unable to retrieve a non transient name, ie when there is no serial
number and so no by-id name we disable the SMART data function.
969d7e5
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 27, 2016
@phillxnet phillxnet update pool device list generator to by-id device name type #1320
Minor variable name refactoring to aid readability and comments added to
assist in further refactoring if need be.
d577a3e
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 28, 2016
@phillxnet phillxnet improve robustness of fake-serial detection in db backend #1320
Previously we only looked for a length of 48 to confirm a fake serial
but this could give false positives so instead match for serials beginning
with 'fake-serial-' independent of length.
b18220f
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 28, 2016
@phillxnet phillxnet make temporary placeholder names for detached devices more obvious #1320


By adding "detached-" to the beginning of placeholder device names we make
them more obvious in logs etc. Especially given the move to by-id type names
which are now much longer. Also easier to identify / parse in UI code as length
no longer a differentiating factor and contents was previously all random.
9ac2f8b
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 28, 2016
@phillxnet phillxnet add 12 zeros as unreliable disk serial on Disks page #1320
An Orico 4 disk usb 3.0 enclosure was found to obfuscate
all it's drives serial numbers with 12 zeros. This makes them
non unique but the first was labeled as OK. Given this is not
reliable flag all drives with this serial as having an unreliable
serial number.
4a948fa
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 28, 2016
@phillxnet phillxnet add disk serial viability check during pool creation #1320
When creating a pool don't offer up disks that have fake-serial
numbers assigned by scan_disks() as they will have unreliable
device names in the db. Also block 12 zeros as serial.
Code reformatted to use spaces not tabs.
360cc07
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 28, 2016
@phillxnet phillxnet minor rewording on disks_table to match detached device naming #1320
In the future the term offline in the context of btrfs may be more confusing
so for the time being at least move to attached / detached / reattached as
we then move closer to removable / backup / import type device roles.
Also detached device names are renamed to start with 'detached-' to
indicate their status in logs and current table views.
6825a48
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 29, 2016
@phillxnet phillxnet trivial code reformat to replace tabs with spaces in models.js #1320
Some extraneous trailing spaces also removed.
1fd1f68
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 29, 2016
@phillxnet phillxnet add serial viability property to backbone disk model #1320 0b86e44
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 29, 2016
@phillxnet phillxnet only offer disks to add to a pool if their serial passes self validat…
…ion #1320

This way a disk with a fake-serial- added by scan_disks() because
none was found is not included in the list of disks offered when
adding to an existing pool.
Also removed console.log line.
007db1f
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 29, 2016
@phillxnet phillxnet add comment on possibly redundant code duplication #1320 b75d45f
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 30, 2016
@phillxnet phillxnet rationalize and improve stability of by-id name finder #1320
Ensure we actually find a by-id and always return the same one
in circumstances where there are multiple by-id entries by
always returning the longest or first if multiple by-id entries are of
equal length. Also improve code readability and rationalize
redundant pattern repetition.
25832ce
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue May 30, 2016
@phillxnet phillxnet restore prior direct db Disk.name matching to conditionally disable S…
…MART #1320

Updates name matches to reflect by-id type names, removed temporary work
around that involved (expensive) conversion back to the previous name types
prior to by-id name conversion. This is not needed as by-id type names for the
target devices are equally as suited to direct name matching.
24f34d8
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 1, 2016
@phillxnet phillxnet reduce number of system calls required to gain base device name #1320
As the db field Disk.name now uses by-id type names we can skip
the prior lsblk system call and output parsing to find our base device
as by-id holds this information within it's surface syntax; so parse that
instead.
Also improve separation of SMART command pre-processor and
get base device system by moving the path addition to the
get_dev_options pre-processor and leaving the lower level process
with simple sting io.
f283386
@phillxnet
Contributor

Almost there now I think, just need to address re-arrangement todo's and maybe slip a few more clean-ups in and a fair bit more testing then this should be ready for review. Currently working as expected but with loads of logging.

@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 2, 2016
@phillxnet phillxnet rewording no serial message to indicate no pool viability status #1320 55a5eac
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 2, 2016
@phillxnet phillxnet move remount from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
5b34543
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 2, 2016
@phillxnet phillxnet move convert_to_kib from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
Added simple docstrings and renamed to comply with PEP8.
05543db
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 2, 2016
@phillxnet phillxnet move root_disk() from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
e8f8667
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 2, 2016
@phillxnet phillxnet move wipe_disk() from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
Added docstrings and did minor variable name refactoring.
c4680a9
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 2, 2016
@phillxnet phillxnet move blink_disk() from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
Added docstrings and did minor variable name refactoring.
bf766d9
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 2, 2016
@phillxnet phillxnet move scan_disk() from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
Switched to using existing const for lsblk binary location.
310299d
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 2, 2016
@phillxnet phillxnet remove some redundant import statements #1320
Also normalize import multi-line syntax in this file.
ceb13c9
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 2, 2016
@phillxnet phillxnet correct some todo codetags syntax according to language #1320 58364aa
@phillxnet
Contributor
phillxnet commented Jun 2, 2016 edited
  • Investigate apparent anomaly in current logging which suggests a blank base name returned in SMART query, yet no malfunction!
@phillxnet
Contributor

Disk's and Pools pages with by-id type names:-
disks_by-id

pools_by-id

@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 4, 2016
@phillxnet phillxnet disable SMART calls on nvme devices #1320
As reported by @snafu on the Rockstor forum (Thanks)
the smartmontools project has only recently added
nvme support: https://www.smartmontools.org/ticket/657
So fixed in upstream but CentOS7's version of
smartmontools doesn't yet include this fix.
N.B. May require additional -d option filters.
adce222
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 4, 2016
@phillxnet phillxnet remove some debug logging used in issue #1320
Also some minor PEP8 formatting of long code lines.
fabac50
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 5, 2016
@phillxnet phillxnet add todo on approach to fix disk activity widget #1320 0ce3c7f
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 10, 2016
@phillxnet phillxnet initial commit of transient to by-id name map function #1320
Lightweight method to create a map of all attached devices
sda type names to by-id type names. Intended primarily for
use by the Disk Activity widget.
9992964
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 10, 2016
@phillxnet phillxnet update Disk Activity widget to by-id name use #1320
By utilizing a name map created on every Disk Activity
widget instantiation the widget can quickly convert the
/proc/diskstats sda type names to the new by-id names
used in the db via a dict lookup.
6ed4a07
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 11, 2016
@phillxnet phillxnet ensure the longest by-id name is used in get_byid_name_map() #1320
This removes the previous assumption that the longest by-id name for
a given device is always listed first as this was found to be incorrect.
Brings this function in line with the behaviour of get_dev_byid_name()
so as to maintain consistency. N.B. the context here is that some
devices have multiple by-id names so we take care to pick the longest
as it is more descriptive and gives repeatable naming.
cf0c390
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 11, 2016
@phillxnet phillxnet remove debug logging for issue #1320 05be66a
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 12, 2016
@phillxnet phillxnet update snapshot rollback mechanism to by-id device name type #1320 7753100
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 12, 2016
@phillxnet phillxnet minor comment clean-up relating to issue #1320 9fb3d2b
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 12, 2016
@phillxnet phillxnet remove redundant constants from system/osi #1320 5710185
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 12, 2016
@phillxnet phillxnet add TODO's for proposed code move to system/hdparm #1320
To split appart the now very large system/osi these TODO's
highlight a subset of code associated with drive APM and spin down
that can be moved to for example system/hdparm in another pr.
Included here for review.
46c2c19
@schakrava schakrava added this to the Looney Bean milestone Jun 13, 2016
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update Disk.name model comment to reflect new content #1320 bc62a11
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet extend device by-id name helper to strip path if needed #1320
So as not to deal with paths in Disk.name db field we can
strip them first in our helper by way of a default flag that
leaves existing function intact.
a484b15
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet extend api/disk regex to include additional chars #1320
As /dev/disk/by-id names contain colons, underscores and
dashes we need to include these in our extended disk_regex
to accommodate the move to by-id names in the Disk.name
db field.
1312b1a
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet use by-id disk names in db and consequent btrfs commands #1320
Initial conversion of disk name entries in db from 'sda' type to
those created by udev in the /dev/disk/by-id directory. Also
involves changing hard coded paths in low level btrfs functions.
aeba752
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update disk power state and hdparm setting readers to by-id #1320
Since the db now contains a by-id name convention we modify these
readers accordingly. Variable name re-factoring included for clarity.
e152837
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update disk pause function to by-id device name type #1320 295ad1d
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update disk APM reader to by-id device name type #1320 46198b0
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update disk spin down config wrapper to by-id device name type #1320
Also update is_rotational comments with possible improvement and
make existing comments more consistent with current function.
64176c2
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet include by-id name in skipping hdparm settings message #1320
Previously we just logged a generic message with no device name
embedded. Improve log message by adding in dev name by-id
bb57283
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update SMART options wrapper to by-id device name type #1320
Initially achieved by adding an additional wrapper to translate the
original get_base_device function so that it can be used with the new
by-id type name format.
Also corrected capitalization issue with recently added local variables.
57e0052
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update pool import mechanism to by-id device name type #1320 1d24a8f
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update resize_pool mechanism to by-id device name type #1320 cc95af5
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet remove some logging used in early stage of issue #1320 4b6f82a
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet improve robustness of fake-serial- detection #1320
Previously we only checked for a length of 48, improve
this by looking specifically for scan_disks stamp on all
fake-serial-uuid4's that it generates.
4fec9d2
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet add debug logging for issue #1320
Used to monitor get_dev_byid_name function.
f664edf
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet if no by-id type disk name then fail over to caller name #1320
Disks with no serial number are not given a DEVLINKS entry by
udev ie virtio with no serial. This causes an issue as we need a
kernel name of sorts for every disk. Failing over to the calling
devices as found by scan_disks and flagging via an additional
return value (a boolean) associated via a tuple return type.
We can't use these no-serial devices anyway but do need to
have 'a' name in order to store and flag them.
Minor formatting updates and comment updates included.
d4baea5
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet remove redundant import #1320 afa0a37
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet disable SMART function for drives with transient names #1320
As we index the smart data to drive names and now these names
are less transient we have a chance to ensure that the SMART
data collected for a given drive will stick with that drive by using
only non transient names to index that data. So when we are
unable to retrieve a non transient name, ie when there is no serial
number and so no by-id name we disable the SMART data function.
f2a0439
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update pool device list generator to by-id device name type #1320
Minor variable name refactoring to aid readability and comments added to
assist in further refactoring if need be.
5f2998a
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet improve robustness of fake-serial detection in db backend #1320
Previously we only looked for a length of 48 to confirm a fake serial
but this could give false positives so instead match for serials beginning
with 'fake-serial-' independent of length.
b68d0a8
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet make temporary placeholder names for detached devices more obvious #1320


By adding "detached-" to the beginning of placeholder device names we make
them more obvious in logs etc. Especially given the move to by-id type names
which are now much longer. Also easier to identify / parse in UI code as length
no longer a differentiating factor and contents was previously all random.
8fb957d
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet add 12 zeros as unreliable disk serial on Disks page #1320
An Orico 4 disk usb 3.0 enclosure was found to obfuscate
all it's drives serial numbers with 12 zeros. This makes them
non unique but the first was labeled as OK. Given this is not
reliable flag all drives with this serial as having an unreliable
serial number.
4786d62
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet add disk serial viability check during pool creation #1320
When creating a pool don't offer up disks that have fake-serial
numbers assigned by scan_disks() as they will have unreliable
device names in the db. Also block 12 zeros as serial.
Code reformatted to use spaces not tabs.
cb98a45
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet minor rewording on disks_table to match detached device naming #1320
In the future the term offline in the context of btrfs may be more confusing
so for the time being at least move to attached / detached / reattached as
we then move closer to removable / backup / import type device roles.
Also detached device names are renamed to start with 'detached-' to
indicate their status in logs and current table views.
bb9cec3
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet trivial code reformat to replace tabs with spaces in models.js #1320
Some extraneous trailing spaces also removed.
5edd63e
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet add serial viability property to backbone disk model #1320 2b1eec7
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet only offer disks to add to a pool if their serial passes self validat…
…ion #1320

This way a disk with a fake-serial- added by scan_disks() because
none was found is not included in the list of disks offered when
adding to an existing pool.
Also removed console.log line.
bfba146
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet add comment on possibly redundant code duplication #1320 07d71c0
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet rationalize and improve stability of by-id name finder #1320
Ensure we actually find a by-id and always return the same one
in circumstances where there are multiple by-id entries by
always returning the longest or first if multiple by-id entries are of
equal length. Also improve code readability and rationalize
redundant pattern repetition.
0b21412
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet restore prior direct db Disk.name matching to conditionally disable S…
…MART #1320

Updates name matches to reflect by-id type names, removed temporary work
around that involved (expensive) conversion back to the previous name types
prior to by-id name conversion. This is not needed as by-id type names for the
target devices are equally as suited to direct name matching.
27eff39
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet reduce number of system calls required to gain base device name #1320
As the db field Disk.name now uses by-id type names we can skip
the prior lsblk system call and output parsing to find our base device
as by-id holds this information within it's surface syntax; so parse that
instead.
Also improve separation of SMART command pre-processor and
get base device system by moving the path addition to the
get_dev_options pre-processor and leaving the lower level process
with simple sting io.
465d39f
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet rewording no serial message to indicate no pool viability status #1320 6725936
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet move remount from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
db70916
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet move convert_to_kib from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
Added simple docstrings and renamed to comply with PEP8.
ff88166
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet move root_disk() from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
42c2593
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet move wipe_disk() from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
Added docstrings and did minor variable name refactoring.
8ca805f
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet move blink_disk() from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
Added docstrings and did minor variable name refactoring.
31ca3c0
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet move scan_disk() from fs/btrfs to system/osi #1320
Not directly related to btrfs so moving to more general location.
Switched to using existing const for lsblk binary location.
af8fe74
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet remove some redundant import statements #1320
Also normalize import multi-line syntax in this file.
03b827a
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet correct some todo codetags syntax according to language #1320 6ad48b9
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet disable SMART calls on nvme devices #1320
As reported by @snafu on the Rockstor forum (Thanks)
the smartmontools project has only recently added
nvme support: https://www.smartmontools.org/ticket/657
So fixed in upstream but CentOS7's version of
smartmontools doesn't yet include this fix.
N.B. May require additional -d option filters.
201b8a9
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet remove some debug logging used in issue #1320
Also some minor PEP8 formatting of long code lines.
06d75e8
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet add todo on approach to fix disk activity widget #1320 7b259d5
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet initial commit of transient to by-id name map function #1320
Lightweight method to create a map of all attached devices
sda type names to by-id type names. Intended primarily for
use by the Disk Activity widget.
8bba211
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update Disk Activity widget to by-id name use #1320
By utilizing a name map created on every Disk Activity
widget instantiation the widget can quickly convert the
/proc/diskstats sda type names to the new by-id names
used in the db via a dict lookup.
6c8c08d
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet ensure the longest by-id name is used in get_byid_name_map() #1320
This removes the previous assumption that the longest by-id name for
a given device is always listed first as this was found to be incorrect.
Brings this function in line with the behaviour of get_dev_byid_name()
so as to maintain consistency. N.B. the context here is that some
devices have multiple by-id names so we take care to pick the longest
as it is more descriptive and gives repeatable naming.
69db137
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet remove debug logging for issue #1320 ef5d7df
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet update snapshot rollback mechanism to by-id device name type #1320 564cf44
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet minor comment clean-up relating to issue #1320 0f85180
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet remove redundant constants from system/osi #1320 ebe4f64
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet add TODO's for proposed code move to system/hdparm #1320
To split appart the now very large system/osi these TODO's
highlight a subset of code associated with drive APM and spin down
that can be moved to for example system/hdparm in another pr.
Included here for review.
d1a227b
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 24, 2016
@phillxnet phillxnet restore trivial quotes change in handlebars helper #1320
This was lost in a rebase / merge so restoring to match
upstream master.
e26ee6a
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jul 1, 2016
@phillxnet phillxnet remove btrfs_label() as unused function #1320 3b79fb3
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jul 1, 2016
@phillxnet phillxnet remove btrfs_importable() as unused function #1320 2009a6c
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jul 1, 2016
@phillxnet phillxnet improve comment consistency in get_byid_name() #1320 4623a44
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jul 2, 2016
@phillxnet phillxnet add get_dev_temp_name() function #1320
This function returns the the temporary canonical name
of a given by-id type name, or if an OSError is encountered
will fail over to returning the passed device name unchanged.
c0ab0a2
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jul 2, 2016
@phillxnet phillxnet add temp_name property to Disk model and serializer #1320 933d121
@phillxnet phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jul 2, 2016
@phillxnet phillxnet remove repeat import statement #1320 18f68f1
@schakrava schakrava closed this in #1357 Jul 4, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment