Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disk I/O graph are missing for SAN disk after a disable/reenable of a Fiber card #4670

Closed
mickael1026 opened this issue Sep 29, 2016 · 18 comments

Comments

@mickael1026
Copy link

Hello

On a Dell PowerEdge R730 with redhat 6, I have several disk attached to the server through multipathd.
After added the server in librenms, I see all disk (sdXX) in the Disk I/O graph part. Everything is OK.

We disabled and reenabled a fiber channel card on the server for a maintenance.
When we disable the fiber channel, the multipathdd process automatically disable the corresponding path.
On linux each path correspond to a disk device /dev/sdXX.
When we reenable the fiber channel, the multipath, reenable the path and the disk have exactly the same name /dev/sd..

After this action, the device that was disable/enable by multipath are missing in librenms in the part Healt, Disk I/O graph,

The usage of /data/librenms/discovery.php script didn't help.

Here the information of my librenms installation :

Version info:
[WARN] Your install is out of date: edbb6ae (Wed, 28 Sep 2016 13:19:28 +0000)
DB Schema: 140
PHP: 5.4.16
MySQL: 5.5.50-MariaDB
RRDTool: 1.4.8
SNMP: NET-SNMP version: 5.7.2

[OK] Database connection successful

Regards
Mickael

@laf laf added the Needs-Info label Oct 1, 2016
@laf
Copy link
Member

laf commented Oct 1, 2016

Did you wait for 6-12 hours for a rediscovery?

If so, please provide the output of:

./discovery.php -h HOSTNAME -d

@mickael1026
Copy link
Author

Hi

I discovered some disq was missing fiew days after.

I putted in attached the result of the 2 commands I run when I had this issue.

discovery.php -v -d -m ucd-diskio -h Hostname >> /tmp/ucd-diskio1
discovery.php -d -m storage -h Hostname >> /tmp/storage1

storage1.txt
ucd-diskio1.txt

I resolved my issue doing this

mv /opt/librenms/rrd/HOSTNAME /opt/librenms/rrd/HOSTNAME.old
discovery.php -h HOSTNAME -d
mv /opt/librenms/rrd/HOSTNAME.old /opt/librenms/rrd/HOSTNAME

after this I could see the graph again.

Now everything is ok, but here the result of a discovery.php -h HOSTNAME -d

discovery.txt

@mickael1026
Copy link
Author

Hi

I put in attached the result of the command ./discovery.php -h HOSTNAME -d

it misses the disk from sdb to sdu

discovery.txt

@laf
Copy link
Member

laf commented Mar 7, 2017

Can't see anything wrong with the discovery which is probably due to it being fixed.

@laf laf closed this as completed Mar 7, 2017
@mickael1026
Copy link
Author

The discovery is OK, but 2 disks are still missing in the interface (Disk I/O graph).
Sdb is missing in the rrd graph even it is in the discovery.

@laf
Copy link
Member

laf commented Mar 8, 2017

Please update your install and then provide the info from http://docs.librenms.org/Support/FAQ/#what-information-do-you-need-to-add-a-new-os

@laf laf reopened this Mar 8, 2017
@mickael1026
Copy link
Author

Update done.
info in attached.

In the poller file, it's the mess for the sdX Array part.
For exemple :

[RRD Disabled]sdh Array
(
[diskIOIndex] => 30
[diskIODevice] => sdl
[diskIONRead] => 1508491776
[diskIONWritten] => 663834624
[diskIOReads] => 27245942
[diskIOWrites] => 71794990
[diskIOLA1] => 20
[diskIOLA5] => 11
[diskIOLA15] => 9
[diskIONReadX] => 1929948807680
[diskIONWrittenX] => 2186802188288
)
RRD[update /data/librenms/rrd/dtcosmdbd01.dtc.rccad.net/ucd_diskio-sdh.rrd N:1929948807680:2186802188288:27245942:71794990]

Here the info from the messsage log of the server, just to show you multipath removed/add the device but no switch was done regarding the sdX and mapping device :

messages: multipathd: sdh: remove path (uevent)
messages: multipathd: sdh [8:112]: path removed from map mpathg
messages: multipathd: sdh: add path (uevent)
messages: multipathd: sdh [8:112]: path added to devmap mpathg

messages : multipathd: sdl [8:176]: path removed from map mpathk
messages: multipathd: sdl: add path (uevent)
messages: multipathd: sdl [8:176]: path added to devmap mpathk

discovery.txt
poller.txt

snmpwalk.zip

@laf
Copy link
Member

laf commented Mar 8, 2017

Please provide the output of SELECT * FROM ucd_diskioWHEREdevice_id = 69\G

@mickael1026
Copy link
Author

MariaDB [librenms]> SELECT * FROM ucd_diskio WHERE device_id= 69\G;
*************************** 1. row ***************************
diskio_id: 32810
device_id: 69
diskio_index: 25
diskio_descr: sda
*************************** 2. row ***************************
diskio_id: 32811
device_id: 69
diskio_index: 26
diskio_descr: sda1
*************************** 3. row ***************************
diskio_id: 32812
device_id: 69
diskio_index: 27
diskio_descr: sda2
*************************** 4. row ***************************
diskio_id: 32813
device_id: 69
diskio_index: 28
diskio_descr: sda3
*************************** 5. row ***************************
diskio_id: 32815
device_id: 69
diskio_index: 30
diskio_descr: sdh
*************************** 6. row ***************************
diskio_id: 32816
device_id: 69
diskio_index: 31
diskio_descr: sde
*************************** 7. row ***************************
diskio_id: 32817
device_id: 69
diskio_index: 32
diskio_descr: sdc
*************************** 8. row ***************************
diskio_id: 32818
device_id: 69
diskio_index: 33
diskio_descr: sdg
*************************** 9. row ***************************
diskio_id: 32819
device_id: 69
diskio_index: 34
diskio_descr: sdi
*************************** 10. row ***************************
diskio_id: 32820
device_id: 69
diskio_index: 35
diskio_descr: sdk
*************************** 11. row ***************************
diskio_id: 32821
device_id: 69
diskio_index: 36
diskio_descr: sdf
*************************** 12. row ***************************
diskio_id: 32822
device_id: 69
diskio_index: 37
diskio_descr: sdj
*************************** 13. row ***************************
diskio_id: 32823
device_id: 69
diskio_index: 38
diskio_descr: sdd
*************************** 14. row ***************************
diskio_id: 32824
device_id: 69
diskio_index: 39
diskio_descr: sdaf
*************************** 15. row ***************************
diskio_id: 32825
device_id: 69
diskio_index: 40
diskio_descr: sds
*************************** 16. row ***************************
diskio_id: 32826
device_id: 69
diskio_index: 41
diskio_descr: sdr
*************************** 17. row ***************************
diskio_id: 32827
device_id: 69
diskio_index: 42
diskio_descr: sdu
*************************** 18. row ***************************
diskio_id: 32828
device_id: 69
diskio_index: 43
diskio_descr: sdl
*************************** 19. row ***************************
diskio_id: 32829
device_id: 69
diskio_index: 44
diskio_descr: sdp
*************************** 20. row ***************************
diskio_id: 32830
device_id: 69
diskio_index: 45
diskio_descr: sdt
*************************** 21. row ***************************
diskio_id: 32831
device_id: 69
diskio_index: 46
diskio_descr: sdq
*************************** 22. row ***************************
diskio_id: 32832
device_id: 69
diskio_index: 47
diskio_descr: sdm
*************************** 23. row ***************************
diskio_id: 32833
device_id: 69
diskio_index: 48
diskio_descr: sdo
*************************** 24. row ***************************
diskio_id: 32835
device_id: 69
diskio_index: 50
diskio_descr: sdal
*************************** 25. row ***************************
diskio_id: 32836
device_id: 69
diskio_index: 51
diskio_descr: sdah
*************************** 26. row ***************************
diskio_id: 32837
device_id: 69
diskio_index: 52
diskio_descr: sdai
*************************** 27. row ***************************
diskio_id: 32838
device_id: 69
diskio_index: 53
diskio_descr: sdaj
*************************** 28. row ***************************
diskio_id: 32839
device_id: 69
diskio_index: 54
diskio_descr: sdam
*************************** 29. row ***************************
diskio_id: 32840
device_id: 69
diskio_index: 55
diskio_descr: sdw
*************************** 30. row ***************************
diskio_id: 32841
device_id: 69
diskio_index: 56
diskio_descr: sdak
*************************** 31. row ***************************
diskio_id: 32842
device_id: 69
diskio_index: 57
diskio_descr: sdx
*************************** 32. row ***************************
diskio_id: 32843
device_id: 69
diskio_index: 58
diskio_descr: sdv
*************************** 33. row ***************************
diskio_id: 32844
device_id: 69
diskio_index: 59
diskio_descr: sdy
*************************** 34. row ***************************
diskio_id: 32845
device_id: 69
diskio_index: 60
diskio_descr: sdz
*************************** 35. row ***************************
diskio_id: 32846
device_id: 69
diskio_index: 61
diskio_descr: sdaa
*************************** 36. row ***************************
diskio_id: 32847
device_id: 69
diskio_index: 62
diskio_descr: sdab
*************************** 37. row ***************************
diskio_id: 32848
device_id: 69
diskio_index: 63
diskio_descr: sdac
*************************** 38. row ***************************
diskio_id: 32849
device_id: 69
diskio_index: 64
diskio_descr: sdad
*************************** 39. row ***************************
diskio_id: 32850
device_id: 69
diskio_index: 65
diskio_descr: sdae
*************************** 40. row ***************************
diskio_id: 32851
device_id: 69
diskio_index: 66
diskio_descr: sdan
*************************** 41. row ***************************
diskio_id: 32852
device_id: 69
diskio_index: 67
diskio_descr: sdao
*************************** 42. row ***************************
diskio_id: 32853
device_id: 69
diskio_index: 68
diskio_descr: sdag
*************************** 43. row ***************************
diskio_id: 39307
device_id: 69
diskio_index: 49
diskio_descr: dm-19
*************************** 44. row ***************************
diskio_id: 39319
device_id: 69
diskio_index: 70
diskio_descr: sdv
*************************** 45. row ***************************
diskio_id: 39320
device_id: 69
diskio_index: 71
diskio_descr: sdw
*************************** 46. row ***************************
diskio_id: 39322
device_id: 69
diskio_index: 73
diskio_descr: sdy
*************************** 47. row ***************************
diskio_id: 39323
device_id: 69
diskio_index: 74
diskio_descr: sdz
*************************** 48. row ***************************
diskio_id: 39324
device_id: 69
diskio_index: 75
diskio_descr: sdaa
*************************** 49. row ***************************
diskio_id: 39325
device_id: 69
diskio_index: 76
diskio_descr: sdab
*************************** 50. row ***************************
diskio_id: 39326
device_id: 69
diskio_index: 77
diskio_descr: sdac
*************************** 51. row ***************************
diskio_id: 39327
device_id: 69
diskio_index: 78
diskio_descr: sdad
*************************** 52. row ***************************
diskio_id: 39328
device_id: 69
diskio_index: 79
diskio_descr: sdae
*************************** 53. row ***************************
diskio_id: 39329
device_id: 69
diskio_index: 80
diskio_descr: sdaf
*************************** 54. row ***************************
diskio_id: 39330
device_id: 69
diskio_index: 81
diskio_descr: sdag
*************************** 55. row ***************************
diskio_id: 39331
device_id: 69
diskio_index: 82
diskio_descr: sdah
*************************** 56. row ***************************
diskio_id: 39332
device_id: 69
diskio_index: 83
diskio_descr: sdai
*************************** 57. row ***************************
diskio_id: 39333
device_id: 69
diskio_index: 84
diskio_descr: sdaj
*************************** 58. row ***************************
diskio_id: 39334
device_id: 69
diskio_index: 85
diskio_descr: sdak
*************************** 59. row ***************************
diskio_id: 39335
device_id: 69
diskio_index: 86
diskio_descr: sdal
*************************** 60. row ***************************
diskio_id: 39336
device_id: 69
diskio_index: 87
diskio_descr: sdam
*************************** 61. row ***************************
diskio_id: 39337
device_id: 69
diskio_index: 88
diskio_descr: sdan
*************************** 62. row ***************************
diskio_id: 39338
device_id: 69
diskio_index: 89
diskio_descr: sdao
*************************** 63. row ***************************
diskio_id: 44245
device_id: 69
diskio_index: 29
diskio_descr: sdm
*************************** 64. row ***************************
diskio_id: 44249
device_id: 69
diskio_index: 72
diskio_descr: dm-2
64 rows in set (0.00 sec)

@laf
Copy link
Member

laf commented Mar 8, 2017

Your storage index has changed, want to try a patch?

@mickael1026
Copy link
Author

Yes of course, let's try a patch.

@laf
Copy link
Member

laf commented Mar 8, 2017

@mickael1026
Copy link
Author

I put the patch, run discovery command, but it's still the same.

I put new poller, discovery in attanched.

linux.zip
poller.txt
discovery.txt

@laf
Copy link
Member

laf commented Mar 9, 2017

Can you re-run the sql query as that data must now have changed.

@mickael1026
Copy link
Author

No change

MariaDB [librenms]> SELECT * FROM ucd_diskio WHERE device_id= 69\G;
*************************** 1. row ***************************
diskio_id: 32810
device_id: 69
diskio_index: 25
diskio_descr: sda
*************************** 2. row ***************************
diskio_id: 32811
device_id: 69
diskio_index: 26
diskio_descr: sda1
*************************** 3. row ***************************
diskio_id: 32812
device_id: 69
diskio_index: 27
diskio_descr: sda2
*************************** 4. row ***************************
diskio_id: 32813
device_id: 69
diskio_index: 28
diskio_descr: sda3
*************************** 5. row ***************************
diskio_id: 32815
device_id: 69
diskio_index: 30
diskio_descr: sdh
*************************** 6. row ***************************
diskio_id: 32816
device_id: 69
diskio_index: 31
diskio_descr: sde
*************************** 7. row ***************************
diskio_id: 32817
device_id: 69
diskio_index: 32
diskio_descr: sdc
*************************** 8. row ***************************
diskio_id: 32818
device_id: 69
diskio_index: 33
diskio_descr: sdg
*************************** 9. row ***************************
diskio_id: 32819
device_id: 69
diskio_index: 34
diskio_descr: sdi
*************************** 10. row ***************************
diskio_id: 32820
device_id: 69
diskio_index: 35
diskio_descr: sdk
*************************** 11. row ***************************
diskio_id: 32821
device_id: 69
diskio_index: 36
diskio_descr: sdf
*************************** 12. row ***************************
diskio_id: 32822
device_id: 69
diskio_index: 37
diskio_descr: sdj
*************************** 13. row ***************************
diskio_id: 32823
device_id: 69
diskio_index: 38
diskio_descr: sdd
*************************** 14. row ***************************
diskio_id: 32824
device_id: 69
diskio_index: 39
diskio_descr: sdaf
*************************** 15. row ***************************
diskio_id: 32825
device_id: 69
diskio_index: 40
diskio_descr: sds
*************************** 16. row ***************************
diskio_id: 32826
device_id: 69
diskio_index: 41
diskio_descr: sdr
*************************** 17. row ***************************
diskio_id: 32827
device_id: 69
diskio_index: 42
diskio_descr: sdu
*************************** 18. row ***************************
diskio_id: 32828
device_id: 69
diskio_index: 43
diskio_descr: sdl
*************************** 19. row ***************************
diskio_id: 32829
device_id: 69
diskio_index: 44
diskio_descr: sdp
*************************** 20. row ***************************
diskio_id: 32830
device_id: 69
diskio_index: 45
diskio_descr: sdt
*************************** 21. row ***************************
diskio_id: 32831
device_id: 69
diskio_index: 46
diskio_descr: sdq
*************************** 22. row ***************************
diskio_id: 32832
device_id: 69
diskio_index: 47
diskio_descr: sdm
*************************** 23. row ***************************
diskio_id: 32833
device_id: 69
diskio_index: 48
diskio_descr: sdo
*************************** 24. row ***************************
diskio_id: 32835
device_id: 69
diskio_index: 50
diskio_descr: sdal
*************************** 25. row ***************************
diskio_id: 32836
device_id: 69
diskio_index: 51
diskio_descr: sdah
*************************** 26. row ***************************
diskio_id: 32837
device_id: 69
diskio_index: 52
diskio_descr: sdai
*************************** 27. row ***************************
diskio_id: 32838
device_id: 69
diskio_index: 53
diskio_descr: sdaj
*************************** 28. row ***************************
diskio_id: 32839
device_id: 69
diskio_index: 54
diskio_descr: sdam
*************************** 29. row ***************************
diskio_id: 32840
device_id: 69
diskio_index: 55
diskio_descr: sdw
*************************** 30. row ***************************
diskio_id: 32841
device_id: 69
diskio_index: 56
diskio_descr: sdak
*************************** 31. row ***************************
diskio_id: 32842
device_id: 69
diskio_index: 57
diskio_descr: sdx
*************************** 32. row ***************************
diskio_id: 32843
device_id: 69
diskio_index: 58
diskio_descr: sdv
*************************** 33. row ***************************
diskio_id: 32844
device_id: 69
diskio_index: 59
diskio_descr: sdy
*************************** 34. row ***************************
diskio_id: 32845
device_id: 69
diskio_index: 60
diskio_descr: sdz
*************************** 35. row ***************************
diskio_id: 32846
device_id: 69
diskio_index: 61
diskio_descr: sdaa
*************************** 36. row ***************************
diskio_id: 32847
device_id: 69
diskio_index: 62
diskio_descr: sdab
*************************** 37. row ***************************
diskio_id: 32848
device_id: 69
diskio_index: 63
diskio_descr: sdac
*************************** 38. row ***************************
diskio_id: 32849
device_id: 69
diskio_index: 64
diskio_descr: sdad
*************************** 39. row ***************************
diskio_id: 32850
device_id: 69
diskio_index: 65
diskio_descr: sdae
*************************** 40. row ***************************
diskio_id: 32851
device_id: 69
diskio_index: 66
diskio_descr: sdan
*************************** 41. row ***************************
diskio_id: 32852
device_id: 69
diskio_index: 67
diskio_descr: sdao
*************************** 42. row ***************************
diskio_id: 32853
device_id: 69
diskio_index: 68
diskio_descr: sdag
*************************** 43. row ***************************
diskio_id: 39307
device_id: 69
diskio_index: 49
diskio_descr: dm-19
*************************** 44. row ***************************
diskio_id: 39319
device_id: 69
diskio_index: 70
diskio_descr: sdv
*************************** 45. row ***************************
diskio_id: 39320
device_id: 69
diskio_index: 71
diskio_descr: sdw
*************************** 46. row ***************************
diskio_id: 39322
device_id: 69
diskio_index: 73
diskio_descr: sdy
*************************** 47. row ***************************
diskio_id: 39323
device_id: 69
diskio_index: 74
diskio_descr: sdz
*************************** 48. row ***************************
diskio_id: 39324
device_id: 69
diskio_index: 75
diskio_descr: sdaa
*************************** 49. row ***************************
diskio_id: 39325
device_id: 69
diskio_index: 76
diskio_descr: sdab
*************************** 50. row ***************************
diskio_id: 39326
device_id: 69
diskio_index: 77
diskio_descr: sdac
*************************** 51. row ***************************
diskio_id: 39327
device_id: 69
diskio_index: 78
diskio_descr: sdad
*************************** 52. row ***************************
diskio_id: 39328
device_id: 69
diskio_index: 79
diskio_descr: sdae
*************************** 53. row ***************************
diskio_id: 39329
device_id: 69
diskio_index: 80
diskio_descr: sdaf
*************************** 54. row ***************************
diskio_id: 39330
device_id: 69
diskio_index: 81
diskio_descr: sdag
*************************** 55. row ***************************
diskio_id: 39331
device_id: 69
diskio_index: 82
diskio_descr: sdah
*************************** 56. row ***************************
diskio_id: 39332
device_id: 69
diskio_index: 83
diskio_descr: sdai
*************************** 57. row ***************************
diskio_id: 39333
device_id: 69
diskio_index: 84
diskio_descr: sdaj
*************************** 58. row ***************************
diskio_id: 39334
device_id: 69
diskio_index: 85
diskio_descr: sdak
*************************** 59. row ***************************
diskio_id: 39335
device_id: 69
diskio_index: 86
diskio_descr: sdal
*************************** 60. row ***************************
diskio_id: 39336
device_id: 69
diskio_index: 87
diskio_descr: sdam
*************************** 61. row ***************************
diskio_id: 39337
device_id: 69
diskio_index: 88
diskio_descr: sdan
*************************** 62. row ***************************
diskio_id: 39338
device_id: 69
diskio_index: 89
diskio_descr: sdao
*************************** 63. row ***************************
diskio_id: 44245
device_id: 69
diskio_index: 29
diskio_descr: sdm
*************************** 64. row ***************************
diskio_id: 44249
device_id: 69
diskio_index: 72
diskio_descr: dm-2
64 rows in set (0.00 sec)

@mickael1026
Copy link
Author

mickael1026 commented Mar 10, 2017

I 'v check the patch and replace a coma with a "AND" in the mysql select, I think there is a mistake.

Now, the discovery added missing devices but some clean need to be done. Now I have duplicate row with the same diskio_descr.

I can have 2 row for the same diskio_descr.

MariaDB [librenms]> SELECT * FROM ucd_diskio WHERE device_id= 69 and diskio_descr ="sdl"\G;
*************************** 1. row ***************************
diskio_id: 32828
device_id: 69
diskio_index: 43
diskio_descr: sdl
*************************** 2. row ***************************
diskio_id: 44353
device_id: 69
diskio_index: 30
diskio_descr: sdl

@mickael1026
Copy link
Author

mickael1026 commented Mar 13, 2017

Hi

I modified ucd-diskio.inc.php to remove duplicate diskio_descr.
I upload the file.

ucd-diskio.inc.php.zip

@lock
Copy link

lock bot commented May 17, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed.

@lock lock bot locked as resolved and limited conversation to collaborators May 17, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants