Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When upgrading to 1.2.11, installer can appear to stop at 42% #3438

Closed
misterjmw opened this issue Apr 8, 2020 · 37 comments
Closed

When upgrading to 1.2.11, installer can appear to stop at 42% #3438

misterjmw opened this issue Apr 8, 2020 · 37 comments
Assignees
Labels
bug Undesired behaviour resolved A fixed issue

Comments

@misterjmw
Copy link

On two different CentOS 7 systems the installer hangs at 42%.
The upgrade page keeps refreshing, but no progress is made.

The seems to be caused by importing template packages (which I can find no other way to import).

From the Forums: https://forums.cacti.net/viewtopic.php?f=2&t=60847

[2020-04-06 20:58:15] [ always  ] Converting Table #191 'version'
[2020-04-06 20:58:15] [ always  ] Converting Table #192 'weathermap_auth'
[2020-04-06 20:58:16] [ always  ] Converting Table #193 'weathermap_data'
[2020-04-06 20:58:16] [ always  ] Converting Table #194 'weathermap_groups'
[2020-04-06 20:58:16] [ always  ] Converting Table #195 'weathermap_maps'
[2020-04-06 20:58:16] [ always  ] Converting Table #196 'weathermap_settings'
[2020-04-06 20:58:16] [ always  ] Switched from  to /tmp/cduF1KTV7
[2020-04-06 20:58:16] [ always  ] NOTE: Using temporary file for db cache: /tmp/cduF1KTV7
[2020-04-06 20:58:16] [ always  ] Upgrading from v1.2.10 (DB 1.2.11 (DB: 1.2.10)) to v1.2.11
[2020-04-06 20:58:16] [ always  ] Found 6 templates to install
[2020-04-06 20:58:16] [ always  ] About to import Package #1 'Cacti_Stats.xml.gz'.
[2020-04-06 20:58:18] [ always  ] Import of Package #1 'Cacti_Stats.xml.gz' under Profile '1' succeeded
[2020-04-06 20:58:18] [ always  ] About to import Package #2 'Cisco_Router.xml.gz'.

Eventually, I reboot the server. The Install runs through converting the tables again;

[2020-04-06 20:58:16] [ always  ] Switched from  to /tmp/cduF1KTV7
[2020-04-06 20:58:16] [ always  ] NOTE: Using temporary file for db cache: /tmp/cduF1KTV7
[2020-04-06 20:58:16] [ always  ] Upgrading from v1.2.10 (DB 1.2.11 (DB: 1.2.10)) to v1.2.11
[2020-04-06 20:58:16] [ always  ] Found 6 templates to install
[2020-04-06 20:58:16] [ always  ] About to import Package #1 'Cacti_Stats.xml.gz'.
[2020-04-06 20:58:18] [ always  ] Import of Package #1 'Cacti_Stats.xml.gz' under Profile '1' succeeded
[2020-04-06 20:58:18] [ always  ] About to import Package #2 'Cisco_Router.xml.gz'.
-- Reboot here --
[2020-04-06 21:23:21] [ always  ] Spawning background process: /bin/php '/usr/share/cacti/install/background.php' 1586229801.2444
[2020-04-06 21:23:21] [ always  ] Setting PHP Option max_execution_time = 0
[2020-04-06 21:23:21] [ always  ] Setting PHP Option memory_limit = -1
[2020-04-06 21:23:22] [ always  ] Starting INSTALL Process for v1.2.11
[2020-04-06 21:23:22] [ always  ] Found 196 tables to convert
[2020-04-06 21:23:22] [ always  ] Converting Table #1 'aggregate_graphs'
[2020-04-06 21:23:22] [ always  ] Converting Table #2 'aggregate_graphs_graph_item'
[2020-04-06 21:23:22] [ always  ] Converting Table #3 'aggregate_graphs_items'
[2020-04-06 21:23:22] [ always  ] Converting Table #4 'aggregate_graph_templates'

This time it succeeds as the import is not performed at the end of the process.

Also; It should also be noted that the timestamps are in local time and the installation times (started/completed) are seemingly in UTC.

[2020-04-06 21:23:51] [ always  ] Converting Table #194 'weathermap_groups'
[2020-04-06 21:23:51] [ always  ] Converting Table #195 'weathermap_maps'
[2020-04-06 21:23:52] [ always  ] Converting Table #196 'weathermap_settings'
[2020-04-06 21:23:52] [ always  ] Finished INSTALL Process for v1.2.11
[2020-04-06 21:23:52] [ always  ] Installation was started at 2020-04-07 03:23:21, completed at 2020-04-07 03:23:52
@misterjmw misterjmw added bug Undesired behaviour unverified Some days we don't have a clue labels Apr 8, 2020
@gbonny1982
Copy link

Same here, ubuntu 18.04, i tried twice with templates. Deselected them and installed ok.

@TheWitness
Copy link
Member

I've tested again and again latest CentOS 7 on 1.2.11 both Apache and NGINX and can not reproduce this.

@netniV
Copy link
Member

netniV commented Apr 9, 2020

The timestamps come from PHP so are the timezone applied by the appropriate PHP.INI

@netniV netniV self-assigned this Apr 10, 2020
@misterjmw
Copy link
Author

The timestamps come from PHP so are the timezone applied by the appropriate PHP.INI

Obviously not terribly concerned by the times, just a bit odd that the timestamp and the referenced time didn't use the same TZ.

Both of the logs I've got seem to indicate that it hung installing template packages. I've tried the CLI and "Import" in the GUI; Is there a way to import them by hand?

@netniV
Copy link
Member

netniV commented Apr 10, 2020

There is the CLI program that the installer used to run, https://github.com/Cacti/cacti/blob/develop/cli/import_template.php

@F-i-f
Copy link
Contributor

F-i-f commented Apr 11, 2020

I ran into the same problem.
After interrupting the stuck upgrade, I was able to login, but the failed template import completely messed up my graphs. I had modified the stock templates a long time ago, and the update ended up having some graphs referencing the old template (pre-import) which I couldn't edit anymore as "Edit Graph Template" was leading me to the new imported template.
I restored from a DB backup, didn't import the templates and it worked fine.

@vperezescorcia
Copy link

I ran into the same problem.
stuck y 42% after upgrade 1.2.10 to 1.2.11 I have installed cacti over Windows server 2016 datacenter.

I rollback to 1.2.10

@ISekan
Copy link

ISekan commented Apr 14, 2020

Have the same problem on Centos7

[2020-04-14 13:36:39] [ always ] Spawning background process: /bin/php '/var/www/html/cacti/install/background.php' 1586860599.2826
[2020-04-14 13:36:39] [ always ] Setting PHP Option max_execution_time = 0
[2020-04-14 13:36:39] [ always ] Setting PHP Option memory_limit = -1
[2020-04-14 13:36:40] [ always ] Starting UPGRADE Process for v1.2.11
[2020-04-14 13:36:40] [ always ] No tables where found or selected for conversion
[2020-04-14 13:36:40] [ always ] Switched from to /tmp/cdurXZ9oZ
[2020-04-14 13:36:40] [ always ] NOTE: Using temporary file for db cache: /tmp/cdurXZ9oZ
[2020-04-14 13:36:40] [ always ] Upgrading from v1.2.8 (DB 1.2.11 (DB: 1.2.8)) to v1.2.11
[2020-04-14 13:36:40] [ always ] Found 5 templates to install
[2020-04-14 13:36:40] [ always ] About to import Package #1 'Cacti_Stats.xml.gz'.
[2020-04-14 13:36:44] [ always ] Import of Package #1 'Cacti_Stats.xml.gz' under Profile '1' succeeded
[2020-04-14 13:36:44] [ always ] About to import Package #2 'Cisco_Router.xml.gz'.

@waclaw66
Copy link

How to recover from it?
I stucked at...

2020/04/16 12:57:47 - INSTALL: always: About to import Package #4 'Local_Linux_Machine.xml.gz'.
2020/04/16 12:57:47 - INSTALL: info: Progress: 44
2020/04/16 12:57:47 - INSTALL: always: Import of Package #3 'Generic_SNMP_Device.xml.gz' under Profile '1' succeeded
2020/04/16 12:57:45 - INSTALL: always: About to import Package #3 'Generic_SNMP_Device.xml.gz'.
2020/04/16 12:57:45 - INSTALL: info: Progress: 43

the log says repeatedly...

2020/04/16 13:15:49 - INSTALL-JSON: debug: Using supplied data
2020/04/16 13:15:49 - INSTALL-JSON: notice: Start: {"Step":"97","Eula":"1"}
2020/04/16 13:15:49 - INSTALL-STEP: notice: Install Parameters: array ( 'Runtime' => 'Web', 'Step' => '97', 'Eula' => '1', )
2020/04/16 13:15:49 - INSTALL-STEP: notice: Initial: '97'
2020/04/16 13:15:49 - INSTALL-STEP: notice: Previously complete: NULL
2020/04/16 13:15:49 - INSTALL-STEP: debug: Does match: '1.2.11'
2020/04/16 13:15:49 - INSTALL-STEP: notice: After: '97'
2020/04/16 13:15:49 - INSTALL-STEP: debug: setStep(): '97'
2020/04/16 13:15:49 - INSTALL-STEP: notice: setStep(): stepError false < '97'
2020/04/16 13:15:49 - INSTALL-LANGUAGE: info: getLanguage(): en-US
2020/04/16 13:15:49 - INSTALL-LANGUAGE: debug: setLanguage(): en-US
2020/04/16 13:15:49 - INSTALL-THEME: info: getTheme(): modern
2020/04/16 13:15:49 - INSTALL-THEME: info: Checking theme: modern
2020/04/16 13:15:49 - INSTALL-THEME: debug: setTheme(): modern
2020/04/16 13:15:49 - INSTALL-STEP: debug: Error: false
2020/04/16 13:15:49 - INSTALL-STEP: debug: Done: '97'
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundTime = 1587034648.4288
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundNeeded =
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: ---------------- Check Expire ----------------
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundDateStarted = 2020-04-16 10:57:28
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundLast = 1587034648.4288
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundExpire = 1587034249
2020/04/16 13:15:49 - INSTALL-JSON: notice:   End: {"Mode":null,"Step":"97","Errors":[],"Eula":"1","Prev":{"Text":"Previous","Step":96,"Enabled":true,"Visible":false},"Next":{"Text":"Next","Step":98,"Enabled":true,"Visible":false},"Test":{"Text":"Test Connection","Step":-
4,"Enabled":true,"Visible":false},"Theme":"modern","StepData":{"Current":"44","Total":100}}

@ParkaMark
Copy link

ParkaMark commented Apr 17, 2020

I can confirm the same problem (hanging) on CentOS 8 under Chrome. After many re-attempts, I then switched to using Firefox and the install did actually complete (no hang), but then I was not able to login on the cacti login screen. The login failure is not the same as just a wrong username/invalid password (which it would print in red text if this was the case) - instead, the login just doesn't work, the page refreshes/resets with no red text. And clearing cookies doesn't resolve this problem.

After spending hours on this, I've reverted back to 1.2.10.

Incidentally, during my cookie resetting etc between versions, I did clock that version 1.2.11 does not appear to set the Cacti cookie, but this is present on 1.2.10. At least I think that's what I saw - I don't have the time or patience to troubleshoot this any further.

@misterjmw
Copy link
Author

@ParkaMark I have also experienced the 'no login' issue. I have found that I can bypass it by logging in (which redirects back to the login page) then going to a cacti link in my browser (e.g. graph), from there I'm let in. Certainly not a 'real' fix.

@TheWitness
Copy link
Member

Unfortunately, I have been unable to reproduce this on either CentOS 8 or CentOS 7 either using Apache or Nginx. Could it be that you guys have some apache configuration setup that is not the default? Are you using https and have you inspected the cookie domain or the cookie settings? You say that there is not Cacti cookie, well that means it has to be environment related.

So, I asked these questions elsewhere:

  1. What version of PHP
  2. What version of Apache
  3. Any customization's to Apache configuration
  4. http or https self signed or legit.
  5. Cookie domains enabled?
  6. Good dns that matches the cookie domain?

@netniV
Copy link
Member

netniV commented Apr 17, 2020

If you have copied the sources over and replaced your existing include/cacti.conf file, you will have the cacti_cookie_domain setting present incorrectly and should have been commented out. If you comment that out, it should then work OK. The reason is that the cookie isn't set properly after the upgrade so it is effectively wiped out. If the domain had been set to your websites domain, this would not have been an issue for logging in I believe.

We do still have the outstanding package issue when hitting 42% (local machine package).

@ParkaMark
Copy link

@netniV Ok, success. That cacti_cookie_domain setting looks to be absent in the old configuration in version 1.2.10 (I did a quick grep). When I upgrade, I never copy across old configuration files, I re-configure the new ones, and I was not aware of the existence of this new setting. I set cacti_cookie_domain to be my actual domain and everything has worked perfectly, upgraded to 1.2.11 with no issues, using Chrome.

So my advice to others is to make sure cacti_cookie_domain is set correctly - at the moment, that setting seems to be un-commented and set to cacti.net by default, which probably breaks things and might explain this upgrade problem others are experiencing.

@TheWitness
Copy link
Member

Cool! Yup that's my fault. Too much late night coding and not enough Zinfandel. Wife is trying to curb my both my consumption during this COVID19 pandemic and well, nope, she's always telling me "go to your office". I guess that's good for Cacti anyway.

@arno-st
Copy link
Contributor

arno-st commented Apr 21, 2020

Sorry to be rude, but where is this cacti.conf is coming from ?
Are you talking about config.php ?

@ParkaMark
Copy link

@arno-st Yes, I pretty sure @netniV meant config.php, not cacti.conf - config.php is the file that requires modification and where the cacti_cookie_domain setting is located (in version 1.2.11). I'm fairly sure, as has already been said, you can comment this setting out and the code will just use what ever virtual host you're running on, but I've chosen to configure it directly, matching my website's virtual host.

@misterjmw
Copy link
Author

While this issue has now addressed the login issue/cookie_domain bug, did the 42%/Package import hang get addressed?

@arno-st
Copy link
Contributor

arno-st commented Apr 22, 2020

The login issue yes, the action to do after the process hang at 42% no!

@netniV
Copy link
Member

netniV commented Apr 22, 2020

@arno-st Yes, I pretty sure @netniV meant config.php, not cacti.conf

I did, my apologies. Have my head deep in a massive migration using lots of conf files for service configuration so I guess that bleed through unfortunately.

@TheWitness
Copy link
Member

I'll take credit for the package issue too. It'll be better in 1.2.12

@hiddenicon
Copy link

I have the same issue. 1.2.10 going to 1.2.11. Stuck on Cisco_Rotuer.xml.gz import package. I'm assuming these packages haven't changed for a while, so since I'm working on 1.2.10, I can skip all of the package imports on the 1.2.11 upgrade?

I'm hitting issue#3373 currently with 1.2.10 so would like to upgrade to 1.2.11 soon as I can.

image

@hiddenicon
Copy link

I think I spoke too soon. I came back and the login prompt was on the screen. On version 1.2.11 now. I was able to log in. I will test more. It may have just taken forever to do that import? I wasn't watching it, but it was at least 30 mins.

@TheWitness
Copy link
Member

Yea, there is a reason. Best not to re-import the packages unless you want all your graphs updated to the new (or in some cases) old format.

@TheWitness TheWitness added resolved A fixed issue and removed unverified Some days we don't have a clue labels Apr 23, 2020
@TheWitness
Copy link
Member

This should be fixed now.

TheWitness added a commit that referenced this issue Apr 24, 2020
Upgrade from 1.2.10 to 1.2.11 - Cacti Upgrade hangs at 42%
@armend-i
Copy link

I had the same issue, upgrade hanged on 42% on Centos 7 and no matter how long I waited it did not finish its installation so I had to stop this process by closing the window, since I have many objects in my cacti I hoped I did not loose them, and it looks that it finished its process, at list it works and the version has been upgraded, I'm just wondering if it has left any problems that can occur at some later stages?
Is there a way how I could tshoot its database or cacti itself?

@netniV
Copy link
Member

netniV commented Apr 27, 2020

The installer doesn't actually do that much work. Most of it is sanity checks and then the few processes it does in the background (not limited to but most):

  • Convert tables if necessary
  • Upgrade the database if required
  • Import packages
  • Setup automatino
  • Let you loose

If it was at the import package stage, if you haven't seen any issues by now, I think you should be OK. You could reimport all the packages by hand, but the likelihood is that they were the same anyway, so it wouldn't have changed them.

You should ALWAYS take a database backup before the upgrade though. If you do this via a VM snapshot or a database dump, it's up to you. But with any upgrade, there can be issues - though hopefully we have nailed down these recent ones as we dislike the disruption to you guys as much as you do.

@TheWitness
Copy link
Member

It should have finished. You would see errors in the Cacti log otherwise. If you are uncertain, or you find a graph that does not match the template, you can always 'Sync Graphs' from the Graph Templates page to finish the Orphan cleaning a template at a time.

@netniV netniV changed the title Upgrade from 1.2.10 to 1.2.11 - Cacti Upgrade hangs at 42% When upgrading to 1.2.11, installer can appear to stop at 42% May 3, 2020
@mdella
Copy link

mdella commented May 30, 2020

Just did a pkg upgrade via the cacti package in the EPEL. During the upgrade, it got stuck at the 42% mark. After reloading the server and trying again, it "finished" (ie, it skipped where it hung) and completed. However now ALL of my graphs (other than aggregrates) are completely messed up with single colors, missing information, etc. I'm assuming the templates installed incorrectly and now are messing everything else up with them.

Unfortunately I don't know how to go backwards with this one (there is no "uninstaller" other than pkg remove) so was wondering if there was a documented manual method of importing these templates and then resetting all the graphs to use them? Any pointers on this one?

image

This is an example of all my "in/out" graphs now. Dark green and blue are both missing as well as all the descriptions (ie, in, out). At this point I'm not sure what I'm looking at on any of my graphs anymore.

@bmfmancini
Copy link
Member

ah poop that sucks since you are at 1.2.11 how about trying to go to 1.2.12 via source?
that would at least get you to the latest I know that 1.2.11 had a bunch of issues that was patched in 1.2.12

@mdella
Copy link

mdella commented May 30, 2020

Been trying to keep things at a package level but if I can't figure out the current issue soon, I might have to go to an all manual method. Blah

@netniV
Copy link
Member

netniV commented May 30, 2020

all the packages are in the install/templates folder and can be imported via the gui or the cli tool import_package.php

@mdella
Copy link

mdella commented May 30, 2020

Have already tried importing all 6 packages using the import_package tool. Of the 6, 5 had changes that were imported. One apparently had already been processed. However once importing all 6 packages, nothing has changed on the output of the graphs. I have verified however that the packages are now in place as processing a second time comes up with no changes (unlike the first time).

@netniV
Copy link
Member

netniV commented May 30, 2020

It is likely then you had previously customised the core templates. This reimporting has reset these customisations back to our defaults.

@mdella
Copy link

mdella commented May 30, 2020

This was a vanilla install. As far as I know, no one here ever touched templates before. Anyway, at this point (see image above) all of our images that are not aggregates (those images seem ok). The importing of the newer templates did not seem to change things at all. Another thing we tried was the sync graphs option. however that one once we hit "continue" basically just sat there and did nothing. never completed and as far as I could tell from ps, etc, never really started anything either.

@mdella
Copy link

mdella commented May 31, 2020

Ok, checking the logs, every time I try to do a graph sync to get things correct, I get the following error in the cacti.log file:

2020/05/31 00:41:40 - ERROR PHP ERROR: Uncaught Error: Call to undefined function api_data_source_cache_crc_update() in /usr/share/cacti/lib/utility.php:745 Stack trace: #0 /usr/share/cacti/lib/template.php(547): push_out_host('218', '4952') #1 /usr/share/cacti/lib/template.php(882): update_graph_data_source_output_type('5009', 0) #2 /usr/share/cacti/lib/template.php(622): change_graph_template('5009', '41', true) #3 /usr/share/cacti/graph_templates.php(305): retemplate_graphs('41') #4 /usr/share/cacti/graph_templates.php(49): form_actions() #5 {main} thrown in file: /usr/share/cacti/lib/utility.php on line: 745
2020/05/31 00:41:40 - CMDPHP PHP ERROR Backtrace: (CactiShutdownHandler())

Searching back far enough shows that this error was happening during the install update as well.

@netniV
Copy link
Member

netniV commented May 31, 2020

OK, can I stop you there? You really should open this as a separate issue. Feel free to make a reference to this one but yours seems to be a different, even if related, issue.

@github-actions github-actions bot locked and limited conversation to collaborators Aug 30, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Undesired behaviour resolved A fixed issue
Projects
None yet
Development

No branches or pull requests