1.2.24 -> 1.2.25 Upgrade process throws Error 500 at the end, stops. Possible DB data damage. #5482
Comments
misterjmw
added
bug
|
If you could zip up your database backup and send it my way, it'll help with Diagnosis. I hope it's not MariaDB 10.5.22! |
|
SendTo: thewitness@cacti.net |
I don't follow; It is "Server version: 10.5.22-MariaDB MariaDB Server" |
mysqldump cacti > dump_for_larry.sql
gzip dump_for_larry.sqlSend that my way. I have to use your setup to reproduce. |
I Understood that portion, I just didn't follow the mariadb comment. I will export tomorrow. |
|
The reason for the MariaDB comment was that something had to have corrupted you database. So, that's the bit that I'm concerned about. |
Ah. Yes, there's something funny going on. But as it works 95% of the time, I've never been able to nail things down; It's only relating to upgrading that tables get dropped and things break. I'm not a DBA so all I can see is that it looks fine in phpMyAdmin. I am exporting the snapshot now to perform the export; While we don't have directly sensitive information in it, are there any steps I need to take for security/sensitivity sake? |
|
I don't know how it might be related; But I'm also seeing a deluge of Thold errors at regular intervals: |
|
@misterjmw, that thold bug has been reported. I'll see about fixing before the morning. |
|
If you installed Thold 1.8 before Cacti 1.2.25, you are likely missing a few columns in the database. |
|
As a workaround, run this query, and then goto the Thold tab to 're-upgrade'. UPDATE plugin_config SET version='1.7' WHERE directory='thold';After that, in tables thold_data, and thold_template, you should find these columns: | syslog_facility | int(2) | YES | | NULL | |
| syslog_enabled | char(3) | NO | | | |
| data_type | int(12) | NO | | 0 | |
| show_units | char(3) | NO | | | |
| units_suffix | varchar(10) | NO | | | |
| decimals | int(11) | NO | | -1 | | |
Ahh, gotcha. I 'rolled' back to 1.2.24, which probably didn't revert Thold. Good to know, and apologies for polluting the topic with a second unrelated bug. |
|
BTW, Nothing in cacti would drop those tables. So, we have a mystery on our hands... |
|
Had a hardware failure in my lab. Waiting on replacement hardware. So, there will be a delay in my analysis. |
Understood. Thanks!
I'm not trying to pollute this ticket; However, every upgrade, 1.2.20 -> 1.2.21, 1.2.21 -> 1.2.22, etc. Those tables disappeared and I had to go through the same import process. It happens every time, and it only occurs during the upgrade process. For the rest of the time, it runs fine without any issue. So, it is correlation but it's a strong one. |
|
What plugins do you have installed? |
|
something something picture worth something something
|
|
Quick Followup; There was clearly DB corruption. I can't say how, or why. I also am not aware of any reason why that would cause the issues being seen here; They are repeatable and consistent, always at the same steps. But I am also not a dba. |
|
Finally got my lab back and running today. Took a day off to catch up on Cacti. Needed the break. |
No worries. Take the time you need. I appreciate the support. I have not had a chance to snapshot, then reattempt the upgrade after optimizing the various DB tables. |
|
I just upgraded your schema, installed all the updated templates (not recommended on existing installs) with no issues. So, I suspect your VM/Database is causing this. PHP version was 8.0.27 (Rock9.2 default). |
2023/09/23 12:00:14 - INSTALL: always: Installation was started at 2023-09-23 15:59:22, completed at 2023-09-23 16:00:14
--
2023/09/23 12:00:14 - INSTALL-SYNC: always: No Remote Data Collectors found for full synchronization
2023/09/23 12:00:14 - INSTALL-SYNC: always: Found 0 poller(s) to sync
2023/09/23 12:00:14 - INSTALL: always: Finished UPGRADE Process for v1.2.25
2023/09/23 12:00:14 - CMDPHP Disabling gexport version 1.4.2 as it is not compatible with Cacti 1.2.25
2023/09/23 12:00:14 - CMDPHP Disabling weathermap version 1.0 as it is not compatible with Cacti 1.2.25
2023/09/23 12:00:14 - CMDPHP Disabling quicktree version 2.0 as it is not compatible with Cacti 1.2.25
2023/09/23 12:00:14 - CMDPHP Disabling webseer version 3.0 as it is not compatible with Cacti 1.2.25
2023/09/23 12:00:14 - CMDPHP Disabling thold version 1.7 as it is not compatible with Cacti 1.2.25
2023/09/23 12:00:14 - CMDPHP Disabling monitor version 2.3.6 as it is not compatible with Cacti 1.2.25
2023/09/23 12:00:13 - INSTALL: always: Mapping Automation Template for Device Template 'Cisco Router'
2023/09/23 12:00:13 - INSTALL: always: Mapping Automation Template for Device Template 'Windows Device'
2023/09/23 12:00:13 - INSTALL: always: Mapping Automation Template for Device Template 'Net-SNMP Device'
2023/09/23 12:00:13 - INSTALL: always: Repairing Automation Rules
2023/09/23 12:00:13 - INSTALL: always: Import of Package #23 'Windows_Device.xml.gz' under Profile '1' succeeded
2023/09/23 12:00:13 - INSTALL: always: About to import Package #23 'Windows_Device.xml.gz'.
2023/09/23 12:00:13 - INSTALL: always: Import of Package #22 'Synology_NAS.xml.gz' under Profile '1' succeeded
2023/09/23 12:00:10 - INSTALL: always: About to import Package #22 'Synology_NAS.xml.gz'.
2023/09/23 12:00:10 - INSTALL: always: Import of Package #21 'SNMP_UPS.xml.gz' under Profile '1' succeeded
2023/09/23 12:00:09 - INSTALL: always: About to import Package #21 'SNMP_UPS.xml.gz'.
2023/09/23 12:00:09 - INSTALL: always: Import of Package #20 'SNMP_Printer.xml.gz' under Profile '1' succeeded
2023/09/23 12:00:08 - INSTALL: always: About to import Package #20 'SNMP_Printer.xml.gz'.
2023/09/23 12:00:08 - INSTALL: always: Import of Package #19 'PING_Advanced_Ping.xml.gz' under Profile '1' succeeded
2023/09/23 12:00:08 - INSTALL: always: About to import Package #19 'PING_Advanced_Ping.xml.gz'.
2023/09/23 12:00:08 - INSTALL: always: Import of Package #18 'NetSNMP_Device.xml.gz' under Profile '1' succeeded
2023/09/23 12:00:05 - INSTALL: always: About to import Package #18 'NetSNMP_Device.xml.gz'.
2023/09/23 12:00:05 - INSTALL: always: Import of Package #17 'Motorola_SB6141.xml.gz' under Profile '1' succeeded
2023/09/23 12:00:03 - INSTALL: always: About to import Package #17 'Motorola_SB6141.xml.gz'.
2023/09/23 12:00:03 - INSTALL: always: Import of Package #16 'MikroTik_Switch_SWOS.xml.gz' under Profile '1' succeeded
2023/09/23 12:00:02 - INSTALL: always: About to import Package #16 'MikroTik_Switch_SWOS.xml.gz'.
2023/09/23 12:00:02 - INSTALL: always: Import of Package #15 'MikroTik_Device.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:55 - INSTALL: always: About to import Package #15 'MikroTik_Device.xml.gz'.
2023/09/23 11:59:55 - INSTALL: always: Import of Package #14 'Local_Linux_Machine.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:54 - INSTALL: always: About to import Package #14 'Local_Linux_Machine.xml.gz'.
2023/09/23 11:59:54 - INSTALL: always: Import of Package #13 'Generic_SNMP_Device.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:54 - INSTALL: always: About to import Package #13 'Generic_SNMP_Device.xml.gz'.
2023/09/23 11:59:54 - INSTALL: always: Import of Package #12 'Fortigate.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:52 - INSTALL: always: About to import Package #12 'Fortigate.xml.gz'.
2023/09/23 11:59:52 - INSTALL: always: Import of Package #11 'ESXi_Device.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:51 - INSTALL: always: About to import Package #11 'ESXi_Device.xml.gz'.
2023/09/23 11:59:51 - INSTALL: always: Import of Package #10 'Citrix_NetScaler_VPX.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:39 - INSTALL: always: About to import Package #10 'Citrix_NetScaler_VPX.xml.gz'.
2023/09/23 11:59:39 - INSTALL: always: Import of Package #9 'Cisco_Router.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:38 - INSTALL: always: About to import Package #9 'Cisco_Router.xml.gz'.
2023/09/23 11:59:38 - INSTALL: always: Import of Package #8 'Cacti_Stats.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:37 - INSTALL: always: About to import Package #8 'Cacti_Stats.xml.gz'.
2023/09/23 11:59:37 - INSTALL: always: Import of Package #7 'BayTech_PDU.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:32 - INSTALL: always: About to import Package #7 'BayTech_PDU.xml.gz'.
2023/09/23 11:59:32 - INSTALL: always: Import of Package #6 'Aruba_Instant_AP_Cluster.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:31 - INSTALL: always: About to import Package #6 'Aruba_Instant_AP_Cluster.xml.gz'.
2023/09/23 11:59:31 - INSTALL: always: Import of Package #5 'APC_InfraStruXure_PDU.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:29 - INSTALL: always: About to import Package #5 'APC_InfraStruXure_PDU.xml.gz'.
2023/09/23 11:59:29 - INSTALL: always: Import of Package #4 'APC_InfraStruXure_InRow_CRAC.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:26 - INSTALL: always: About to import Package #4 'APC_InfraStruXure_InRow_CRAC.xml.gz'.
2023/09/23 11:59:26 - INSTALL: always: Import of Package #3 'Apache_Webserver.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:25 - INSTALL: always: About to import Package #3 'Apache_Webserver.xml.gz'.
2023/09/23 11:59:25 - INSTALL: always: Import of Package #2 'AKCP_Device.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:24 - INSTALL: always: About to import Package #2 'AKCP_Device.xml.gz'.
2023/09/23 11:59:24 - INSTALL: always: Import of Package #1 'ACME.xml.gz' under Profile '1' succeeded
2023/09/23 11:59:22 - INSTALL: always: About to import Package #1 'ACME.xml.gz'.
2023/09/23 11:59:22 - INSTALL: always: Found 23 templates to install
2023/09/23 11:59:22 - INSTALL: always: Upgrading from v1.2.24 (DB 1.2.25 (DB: 1.2.24)) to v1.2.25
2023/09/23 11:59:22 - INSTALL: always: NOTE: Using temporary file for db cache: /tmp/cduvjKWu4
2023/09/23 11:59:22 - INSTALL: always: Switched from to /tmp/cduvjKWu4
2023/09/23 11:59:22 - INSTALL: always: No tables where found or selected for conversion
2023/09/23 11:59:22 - INSTALL: always: Starting UPGRADE Process for v1.2.25
2023/09/23 11:59:22 - INSTALL: always: Setting PHP Option memory_limit = -1
2023/09/23 11:59:22 - INSTALL: always: Setting PHP Option max_execution_time = 0
2023/09/23 11:59:22 - INSTALL: Checking arguments
2023/09/23 11:59:22 - INSTALL: always: Spawning background process: /usr/bin/php '/var/www/html/cacti/install/background.php' 1695484762.3372 |
|
This is likely due to me not having these installed: |
|
I think that's a bug too... |
|
With regard to your upgrade issues, it might have something to do with your Weathermap. Maybe you should upgrade to the Cacti version of it. |
Is that integrated (like realtime is now)? |
|
That's the right one, but the plugin table had the wrong data. Have to fix that. |
@TheWitness Unfortunately my most recent attempt is just as borked as when I opened this ticket. 500 Error is always at the same spot, always right at or right after 100%. Here's a gif of the whole thing (GH being GH, Remember to hit the play button or else it'll stay on that first frame): I rolled back to 1.2.24 again, I couldn't confirm that nothing further was damaged. |
|
This just happened in my roll-back of 1.2.24, this has only happened after my 1.2.2.5 upgrade then roll back;
|
|
Please try it without importing the packages. |
Same 500 after 'completion', same permissions issues afterwards:
I haven't seen a smoking gun in the logs, but the debug logging necessary is deeper digging than I normally have to go. |
|
Do you want to do a zoom on Thursday? |
I'll shoot you an email. There's a possibility of that working, but I have a very mixed schedule. |
Tried to catch you for a zoom call today, doesn't seem to have worked out. |
|
I totally missed this. Was a busy day at the office. Are you awake? |
|
Working with @misterjmw, we were able to fix some corruption in his database, but it's still a mystery as to how the User Group tables are being removed. |
|
I think I found the issue. Did you have the User Group plugin installed at one point in time? SELECT * FROM plugin_db_changes WHERE table LIKE '%user%';If you find your tables there, node the plugin and then go through the following tables and remove any rows you find. plugin_config;
plugin_realms;
plugin_db_changes;
plugin_hooks; |
|
I think I wrote a user group plugin years ago and totally forgot about it. |
I honestly do not recall; It's been a long time since I spun it up, lots of iterations. I'll dig through SQL and look. |
|
Confirmed. You need to delete the ugroup directory.
|
|
@TheWitness One last thing; I was going to open the feature request for the sql colors work we did, but I wasn't sure exactly how to phrase what we did. Suggestions? |
|
How about Cacti colors table repair fails when duplicate color hex values exist in colors table due to missing primary key. |
|
Was that color issue opened? |
It had not, I have barely had a chance to get back to cacti other than to write my last forum post. Just fell off the urgency list. |
|
Let me address that one. Not sure how far we have to go back to resolve though. |
|
Fixed here: #5581 |
|
@TheWitness Afternoon, I know you wanted to loop back to this if the next upgrade still had issues; I just upgraded to 1.2.26 and the issue still occurs. I get a 500 error, auth tables go poof and I have to push them back in (via my sql) to make auth work properly. |
|
Re-opened. I'll be working on Cacti on Sunday too. Watching stuff on Rumble now. |
|
@TheWitness E.g. here's a graph that breaks because the CDEF for a 24hr trend is zero'd out. Initial graph (used to work): Looking at CDEFs being used: CDEF for 24h Trend: |
|
My brain..... |
|
@misterjmw, I'm going to close this one. Reach out to me at thewitness at cacti dot net and we can do a conference call. It may have to be the week of 2/26. I'm off tomorrow and on the Left coast the rest of the week. |
Sure thing, thanks again. |
Describe the bug
Upon completion of the Upgrade process from 1.2.24 to 1.2.25., The auto-refreshing pages throws an 'Error 500' and the page quits responding.
Upon refresh, the installer does not recognize the existing install (suggests a new install) and there appears to be corruption of some sort in the DB as errors are shown indicating that CDEFs are unknown.
To Reproduce
CLI:
GUI:
The page stops responding.
Clicking Refresh; The installer restarts but does not recognize the existing Cacti installation.
If templates were imported, attempting to use the Cacti install shows various error messages that did not exist previously:
If templates were not imported, cacti is mostly usable but there are odd errors in /var/log/messages:
Expected behavior
Upgrade process should complete, an error should not be shown, DB data should not be corrupted/damaged.
Screenshots
This is a recording of the install process:
Server (please complete the following information)
Snapshots of the VM were taken before/after issue so I can repeat/test easily.
Related Forum Topic: https://forums.cacti.net/viewtopic.php?t=62808
Notes
I fully expect that this is a db error somewhere/somehow. The server has several other instances of odd behavior that have/do occur. However, there are no log entries that I can find indicating what fails. Cacti's log has no entries post-update, Apache log is empty, etc.
The text was updated successfully, but these errors were encountered: