Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On large systems, Primary ID usage on heavily used tables will overflow due to default MySQL variable size #4092

Closed
TheWitness opened this issue Jan 23, 2021 · 12 comments
Labels
blocker Blocking release bug Undesired behaviour resolved A fixed issue
Milestone

Comments

@TheWitness
Copy link
Member

Describe the bug

On Cacti systems that have been running for years and who have experienced quite a bit of change, the database will run out of auto_increment variables for several tables. I've just come across this and the damage is pretty significant.

Expected behavior

Cacti should be able to have millions of graphs over years of use without issue.

@TheWitness TheWitness added bug Undesired behaviour blocker Blocking release labels Jan 23, 2021
@TheWitness TheWitness added this to the v1.2.17 milestone Jan 23, 2021
@bmfmancini
Copy link
Member

Would this affect systems that started from 1.2.x or systems that started from 0.8 and upgraded since then ?

@bmfmancini
Copy link
Member

Quick question would mysql safe just fail to do the insert and it would error out or is cacti not showing any signs of the issue unless you look at the db side I have a few large systems so want to check on what the impact would be

@TheWitness
Copy link
Member Author

It starts erroring somewhere about 3m graphs over it's lifetime.

TheWitness added a commit that referenced this issue Jan 25, 2021
- On large systems several Cacti integer value will overflow
@TheWitness TheWitness added the resolved A fixed issue label Jan 25, 2021
@TheWitness
Copy link
Member Author

@netniV, the only question on this one is do we do it be default in the 1.2.17 upgrade for all systems. For new systems, it won't be required.

@bmfmancini
Copy link
Member

@TheWitness

I just ran a quick test on this and found a small bug

root@cacti-1:/var/www/html/cacti/cli# php fix_mediumint.php -h
!#/usr/bin/env php
PHP Warning:  include(../../include/cli_check.php): failed to open stream: No such file or directory in /var/www/html/cacti/cli/fix_mediumint.php on line 26
PHP Warning:  include(): Failed opening '../../include/cli_check.php' for inclusion (include_path='.:/usr/share/php') in /var/www/html/cacti/cli/fix_mediumint.php on line 26
PHP Fatal error:  Uncaught Error: Call to undefined function cacti_sizeof() in /var/www/html/cacti/cli/fix_mediumint.php:36
Stack trace:
#0 {main}
  thrown in /var/www/html/cacti/cli/fix_mediumint.php on line 36

I had to change

include('../../include/cli_check.php');
to
include('../include/cli_check.php');

@bmfmancini
Copy link
Member

Actually I think that fixes the error but breaks the script

This is the output I get

root@cacti-1:/var/www/html/cacti/cli# php fix_mediumint.php --debug
!#/usr/bin/env php

And the DB still looks the same

MariaDB [cacti]> describe data_input;
+--------------+-----------------------+------+-----+---------+----------------+
| Field        | Type                  | Null | Key | Default | Extra          |
+--------------+-----------------------+------+-----+---------+----------------+
| id           | mediumint(8) unsigned | NO   | PRI | NULL    | auto_increment |
| hash         | varchar(32)           | NO   |     |         |                |
| name         | varchar(200)          | NO   | MUL |         |                |
| input_string | varchar(512)          | YES  |     | NULL    |                |
| type_id      | tinyint(2)            | NO   |     | 0       |                |
+--------------+-----------------------+------+-----+---------+----------------+

Unless I am missing a file or something

@TheWitness
Copy link
Member Author

lol, bangsh!

@TheWitness
Copy link
Member Author

Try again.

@TheWitness
Copy link
Member Author

TheWitness commented Jan 25, 2021

Should look like this on the 2nd pass anyway

[root@vmhost3 cli]# php -q fix_mediumint.php --debug
Fixing Column widths
DEBUG: Column data_template_data_id in Table data_input_data already converted.
DEBUG: Column id in Table data_template_data already converted.
DEBUG: Column local_data_template_data_id in Table data_template_data already converted.
DEBUG: Column local_data_id in Table data_template_data already converted.
DEBUG: Column id in Table data_template_rrd already converted.
DEBUG: Column local_data_template_rrd_id in Table data_template_rrd already converted.
DEBUG: Column local_data_id in Table data_template_rrd already converted.
DEBUG: Column id in Table graph_local already converted.
DEBUG: Column id in Table data_local already converted.
DEBUG: Column local_data_id in Table data_source_purge_action already converted.
DEBUG: Column local_data_id in Table data_source_purge_temp already converted.
DEBUG: Column local_data_id in Table data_source_stats_daily already converted.
DEBUG: Column local_data_id in Table data_source_stats_hourly already converted.
DEBUG: Column local_data_id in Table data_source_stats_hourly_cache already converted.
DEBUG: Column local_data_id in Table data_source_stats_hourly_last already converted.
DEBUG: Column local_data_id in Table data_source_stats_monthly already converted.
DEBUG: Column local_data_id in Table data_source_stats_weekly already converted.
DEBUG: Column local_data_id in Table data_source_stats_yearly already converted.
DEBUG: Column id in Table graph_templates_graph already converted.
DEBUG: Column local_graph_id in Table graph_templates_graph already converted.
DEBUG: Column local_graph_template_graph_id in Table graph_templates_graph already converted.
DEBUG: Column graph_template_item_id in Table graph_template_input_defs already converted.
DEBUG: Column id in Table graph_templates_item already converted.
DEBUG: Column local_graph_template_item_id in Table graph_templates_item already converted.
DEBUG: Column local_graph_id in Table graph_templates_item already converted.
DEBUG: Column task_item_id in Table graph_templates_item already converted.
DEBUG: Column local_graph_id in Table graph_tree_items already converted.
DEBUG: Column local_data_id in Table poller_item already converted.
DEBUG: Column local_data_id in Table poller_output already converted.
DEBUG: Column local_data_id in Table poller_output_boost already converted.
DEBUG: Column local_data_id in Table poller_output_realtime already converted.
DEBUG: Column graph_tree_item_id in Table settings_tree already converted.
DEBUG: Column data_template_rrd_id in Table snmp_query_graph_rrd already converted.
Column widths adjusted on 0 Tables!

@bmfmancini
Copy link
Member

LOL Yea right after I posted that output I was like ooooo I get it now lol

Yes here is what I get now first pass

root@cacti-1:/var/www/html/cacti/cli# php fix_mediumint.php --debug
Fixing Column widths
DEBUG: Updating Table data_input_data.
DEBUG: Updating Table data_template_data.
DEBUG: Updating Table data_template_rrd.
DEBUG: Updating Table graph_local.
DEBUG: Updating Table data_local.
DEBUG: Updating Table data_source_purge_action.
DEBUG: Updating Table data_source_purge_temp.
DEBUG: Updating Table data_source_stats_daily.
DEBUG: Updating Table data_source_stats_hourly.
DEBUG: Updating Table data_source_stats_hourly_cache.
DEBUG: Updating Table data_source_stats_hourly_last.
DEBUG: Updating Table data_source_stats_monthly.
DEBUG: Updating Table data_source_stats_weekly.
DEBUG: Updating Table data_source_stats_yearly.
DEBUG: Updating Table graph_templates_graph.
DEBUG: Updating Table graph_template_input_defs.
DEBUG: Updating Table graph_templates_item.
DEBUG: Updating Table graph_tree_items.
DEBUG: Updating Table poller_item.
DEBUG: Updating Table poller_output.
DEBUG: Updating Table poller_output_boost.
DEBUG: Updating Table poller_output_realtime.
DEBUG: Updating Table settings_tree.
DEBUG: Updating Table snmp_query_graph_rrd.
DEBUG: Column local_graph_id in Table aggregate_graphs already converted.
DEBUG: Column local_graph_id in Table aggregate_graphs_items already converted.
DEBUG: Updating Table plugin_thold_log.
DEBUG: Column local_data_id in Table poller_output_boost_local_data_ids already converted.
DEBUG: Column local_graph_id in Table reports_items already converted.
DEBUG: Updating Table thold_data.
DEBUG: Updating Table thold_data.
DEBUG: Updating Table thold_data.
Column widths adjusted on 28 Tables!

@bmfmancini
Copy link
Member

MariaDB [cacti]> describe data_input_data
    -> ;
+-----------------------+-----------------------+------+-----+---------+-------+
| Field                 | Type                  | Null | Key | Default | Extra |
+-----------------------+-----------------------+------+-----+---------+-------+
| data_input_field_id   | mediumint(8) unsigned | NO   | PRI | 0       |       |
| data_template_data_id | int(10) unsigned      | NO   | PRI | 0       |       |
| t_value               | char(2)               | YES  | MUL | NULL    |       |
| value                 | text                  | YES  |     | NULL    |       |
+-----------------------+-----------------------+------+-----+---------+-------+
4 rows in set (0.002 sec)

Diff from DB dump

<   `data_template_rrd_id` mediumint(8) unsigned NOT NULL DEFAULT 0,
---
>   `data_template_rrd_id` int(10) unsigned NOT NULL DEFAULT 0,
3763c3763

@bmfmancini
Copy link
Member

Second pass test

root@cacti-1:/var/www/html/cacti/cli# php fix_mediumint.php 
Fixing Column widths
DEBUG: Column data_template_data_id in Table data_input_data already converted.
DEBUG: Column id in Table data_template_data already converted.
DEBUG: Column local_data_template_data_id in Table data_template_data already converted.
DEBUG: Column local_data_id in Table data_template_data already converted.
DEBUG: Column id in Table data_template_rrd already converted.
DEBUG: Column local_data_template_rrd_id in Table data_template_rrd already converted.
DEBUG: Column local_data_id in Table data_template_rrd already converted.
DEBUG: Column id in Table graph_local already converted.
DEBUG: Column id in Table data_local already converted.
DEBUG: Column local_data_id in Table data_source_purge_action already converted.
DEBUG: Column local_data_id in Table data_source_purge_temp already converted.
DEBUG: Column local_data_id in Table data_source_stats_daily already converted.
DEBUG: Column local_data_id in Table data_source_stats_hourly already converted.
DEBUG: Column local_data_id in Table data_source_stats_hourly_cache already converted.
DEBUG: Column local_data_id in Table data_source_stats_hourly_last already converted.
DEBUG: Column local_data_id in Table data_source_stats_monthly already converted.
DEBUG: Column local_data_id in Table data_source_stats_weekly already converted.
DEBUG: Column local_data_id in Table data_source_stats_yearly already converted.
DEBUG: Column id in Table graph_templates_graph already converted.
DEBUG: Column local_graph_id in Table graph_templates_graph already converted.
DEBUG: Column local_graph_template_graph_id in Table graph_templates_graph already converted.
DEBUG: Column graph_template_item_id in Table graph_template_input_defs already converted.
DEBUG: Column id in Table graph_templates_item already converted.
DEBUG: Column local_graph_template_item_id in Table graph_templates_item already converted.
DEBUG: Column local_graph_id in Table graph_templates_item already converted.
DEBUG: Column task_item_id in Table graph_templates_item already converted.
DEBUG: Column local_graph_id in Table graph_tree_items already converted.
DEBUG: Column local_data_id in Table poller_item already converted.
DEBUG: Column local_data_id in Table poller_output already converted.
DEBUG: Column local_data_id in Table poller_output_boost already converted.
DEBUG: Column local_data_id in Table poller_output_realtime already converted.
DEBUG: Column graph_tree_item_id in Table settings_tree already converted.
DEBUG: Column data_template_rrd_id in Table snmp_query_graph_rrd already converted.
Column widths adjusted on 0 Tables!

All looking good !

TheWitness added a commit that referenced this issue Feb 7, 2021
This moves the mediumint fix into the upgrade script.  Will leave the CLI there for people who have upgraded to the 1.2.x branch prior to formal release.
@netniV netniV changed the title On large systems several Cacti integer value will overflow On large systems, Primary ID usage on heavily used tables will overflow due to default MySQL variable size Apr 30, 2021
@github-actions github-actions bot locked and limited conversation to collaborators Jul 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
blocker Blocking release bug Undesired behaviour resolved A fixed issue
Projects
None yet
Development

No branches or pull requests

2 participants