Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workaround for bad entSensorPrecision values #11757

Merged
merged 2 commits into from Jun 5, 2020
Merged

Workaround for bad entSensorPrecision values #11757

merged 2 commits into from Jun 5, 2020

Conversation

ryanheffernan
Copy link
Contributor

@ryanheffernan ryanheffernan commented Jun 1, 2020

We are experiencing frequent crashes of librenms due to a router bug in some Cat9k switches that are reporting a garbage high value for entSensorPrecision. Example:

iso.3.6.1.4.1.9.9.91.1.1.1.1.3.2126 = INTEGER: 1615384784

This causes librenms to padd the sensor value with 1615384784 zeroes which obviously fails to be written into the database and prevents the discovery of this interface. Librenms then logs this failure and fills up the disk space in /opt causing the application to crash. See the below log message and imagine the '0's keep going until the log file is 85GB long.

[2020-05-23 02:47:07] production.ERROR: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away (SQL: INSERT IGNORE INTO `sensors` (`poller_type`,`sensor_class`,`device_id`,`sensor_oid`,`sensor_index`,`sensor_type`,`sensor_descr`,`sensor_divisor`,`sensor_multiplier`,`sensor_limit`,`sensor_limit_low`,`sensor_current`,`entPhysicalIndex`,`entPhysicalIndex_measured`)  VALUES (snmp,voltage,9,.1.3.6.1.4.1.9.9.91.1.1.1.1.4.2141,2141,cisco-entity-sensor, - Te2/0/17 Bias Current ,10000000000000

This is obviously a router bug that is being reported to Cisco however I think librenms should have some prevention for this instead of simply accepting the precision value and padding it to whatever it receives. I arbitrarily selected 100k as a reasonable upper limit for how many decimal places to pad since I can't imagine any sensor with that level of precision in a realistic scenario. I don't exactly like this solution and would be open to alternate suggestions.
DO NOT DELETE THE UNDERLYING TEXT

Please note

Please read this information carefully. You can run ./scripts/pre-commit.php to check your code before submitting.

  • Have you followed our code guidelines?
  • If my Pull Request does some changes/fixes/enhancements in the WebUI, I have inserted a screenshot of it.

Testers

If you would like to test this pull request then please run: ./scripts/github-apply <pr_id>, i.e ./scripts/github-apply 5926
After you are done testing, you can remove the changes with ./scripts/github-remove. If there are schema changes, you can ask on discord how to revert.

@kkrumm1 kkrumm1 added the Device 🖥️ New or added device support label Jun 1, 2020
@murrant murrant merged commit 801d9a1 into librenms:master Jun 5, 2020
@ryanheffernan ryanheffernan deleted the ent-sensor-precision branch June 8, 2020 17:25
@murrant
Copy link
Member

murrant commented Jul 3, 2020

This pull request has been mentioned on LibreNMS Community. There might be relevant details there:

https://community.librenms.org/t/v1-65-release-changelog-june-2020/12687/1

1 similar comment
@murrant
Copy link
Member

murrant commented Jul 3, 2020

This pull request has been mentioned on LibreNMS Community. There might be relevant details there:

https://community.librenms.org/t/v1-65-release-changelog-june-2020/12687/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Device 🖥️ New or added device support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants