When upgrading to 1.2.11, installer can appear to stop at 42%

[misterjmw]

On two different CentOS 7 systems the installer hangs at 42%.
The upgrade page keeps refreshing, but no progress is made.

The seems to be caused by importing template packages (which I can find no other way to import).

From the Forums: https://forums.cacti.net/viewtopic.php?f=2&t=60847

[2020-04-06 20:58:15] [ always  ] Converting Table #191 'version'
[2020-04-06 20:58:15] [ always  ] Converting Table #192 'weathermap_auth'
[2020-04-06 20:58:16] [ always  ] Converting Table #193 'weathermap_data'
[2020-04-06 20:58:16] [ always  ] Converting Table #194 'weathermap_groups'
[2020-04-06 20:58:16] [ always  ] Converting Table #195 'weathermap_maps'
[2020-04-06 20:58:16] [ always  ] Converting Table #196 'weathermap_settings'
[2020-04-06 20:58:16] [ always  ] Switched from  to /tmp/cduF1KTV7
[2020-04-06 20:58:16] [ always  ] NOTE: Using temporary file for db cache: /tmp/cduF1KTV7
[2020-04-06 20:58:16] [ always  ] Upgrading from v1.2.10 (DB 1.2.11 (DB: 1.2.10)) to v1.2.11
[2020-04-06 20:58:16] [ always  ] Found 6 templates to install
[2020-04-06 20:58:16] [ always  ] About to import Package #1 'Cacti_Stats.xml.gz'.
[2020-04-06 20:58:18] [ always  ] Import of Package #1 'Cacti_Stats.xml.gz' under Profile '1' succeeded
[2020-04-06 20:58:18] [ always  ] About to import Package #2 'Cisco_Router.xml.gz'.

Eventually, I reboot the server. The Install runs through converting the tables again;

[2020-04-06 20:58:16] [ always  ] Switched from  to /tmp/cduF1KTV7
[2020-04-06 20:58:16] [ always  ] NOTE: Using temporary file for db cache: /tmp/cduF1KTV7
[2020-04-06 20:58:16] [ always  ] Upgrading from v1.2.10 (DB 1.2.11 (DB: 1.2.10)) to v1.2.11
[2020-04-06 20:58:16] [ always  ] Found 6 templates to install
[2020-04-06 20:58:16] [ always  ] About to import Package #1 'Cacti_Stats.xml.gz'.
[2020-04-06 20:58:18] [ always  ] Import of Package #1 'Cacti_Stats.xml.gz' under Profile '1' succeeded
[2020-04-06 20:58:18] [ always  ] About to import Package #2 'Cisco_Router.xml.gz'.
-- Reboot here --
[2020-04-06 21:23:21] [ always  ] Spawning background process: /bin/php '/usr/share/cacti/install/background.php' 1586229801.2444
[2020-04-06 21:23:21] [ always  ] Setting PHP Option max_execution_time = 0
[2020-04-06 21:23:21] [ always  ] Setting PHP Option memory_limit = -1
[2020-04-06 21:23:22] [ always  ] Starting INSTALL Process for v1.2.11
[2020-04-06 21:23:22] [ always  ] Found 196 tables to convert
[2020-04-06 21:23:22] [ always  ] Converting Table #1 'aggregate_graphs'
[2020-04-06 21:23:22] [ always  ] Converting Table #2 'aggregate_graphs_graph_item'
[2020-04-06 21:23:22] [ always  ] Converting Table #3 'aggregate_graphs_items'
[2020-04-06 21:23:22] [ always  ] Converting Table #4 'aggregate_graph_templates'

This time it succeeds as the import is not performed at the end of the process.

Also; It should also be noted that the timestamps are in local time and the installation times (started/completed) are seemingly in UTC.

[2020-04-06 21:23:51] [ always  ] Converting Table #194 'weathermap_groups'
[2020-04-06 21:23:51] [ always  ] Converting Table #195 'weathermap_maps'
[2020-04-06 21:23:52] [ always  ] Converting Table #196 'weathermap_settings'
[2020-04-06 21:23:52] [ always  ] Finished INSTALL Process for v1.2.11
[2020-04-06 21:23:52] [ always  ] Installation was started at 2020-04-07 03:23:21, completed at 2020-04-07 03:23:52

[gbonny1982]

Same here, ubuntu 18.04, i tried twice with templates. Deselected them and installed ok.

[TheWitness]

I've tested again and again latest CentOS 7 on 1.2.11 both Apache and NGINX and can not reproduce this.

[netniV]

The timestamps come from PHP so are the timezone applied by the appropriate PHP.INI

[misterjmw]

The timestamps come from PHP so are the timezone applied by the appropriate PHP.INI

Obviously not terribly concerned by the times, just a bit odd that the timestamp and the referenced time didn't use the same TZ.

Both of the logs I've got seem to indicate that it hung installing template packages. I've tried the CLI and "Import" in the GUI; Is there a way to import them by hand?

[netniV]

There is the CLI program that the installer used to run, https://github.com/Cacti/cacti/blob/develop/cli/import_template.php

[F-i-f]

I ran into the same problem.
After interrupting the stuck upgrade, I was able to login, but the failed template import completely messed up my graphs. I had modified the stock templates a long time ago, and the update ended up having some graphs referencing the old template (pre-import) which I couldn't edit anymore as "Edit Graph Template" was leading me to the new imported template.
I restored from a DB backup, didn't import the templates and it worked fine.

[vperezescorcia]

I ran into the same problem.
stuck y 42% after upgrade 1.2.10 to 1.2.11 I have installed cacti over Windows server 2016 datacenter.

I rollback to 1.2.10

[ISekan]

Have the same problem on Centos7

[2020-04-14 13:36:39] [ always ] Spawning background process: /bin/php '/var/www/html/cacti/install/background.php' 1586860599.2826
[2020-04-14 13:36:39] [ always ] Setting PHP Option max_execution_time = 0
[2020-04-14 13:36:39] [ always ] Setting PHP Option memory_limit = -1
[2020-04-14 13:36:40] [ always ] Starting UPGRADE Process for v1.2.11
[2020-04-14 13:36:40] [ always ] No tables where found or selected for conversion
[2020-04-14 13:36:40] [ always ] Switched from to /tmp/cdurXZ9oZ
[2020-04-14 13:36:40] [ always ] NOTE: Using temporary file for db cache: /tmp/cdurXZ9oZ
[2020-04-14 13:36:40] [ always ] Upgrading from v1.2.8 (DB 1.2.11 (DB: 1.2.8)) to v1.2.11
[2020-04-14 13:36:40] [ always ] Found 5 templates to install
[2020-04-14 13:36:40] [ always ] About to import Package #1 'Cacti_Stats.xml.gz'.
[2020-04-14 13:36:44] [ always ] Import of Package #1 'Cacti_Stats.xml.gz' under Profile '1' succeeded
[2020-04-14 13:36:44] [ always ] About to import Package #2 'Cisco_Router.xml.gz'.

[waclaw66]

How to recover from it?
I stucked at...

2020/04/16 12:57:47 - INSTALL: always: About to import Package #4 'Local_Linux_Machine.xml.gz'.
2020/04/16 12:57:47 - INSTALL: info: Progress: 44
2020/04/16 12:57:47 - INSTALL: always: Import of Package #3 'Generic_SNMP_Device.xml.gz' under Profile '1' succeeded
2020/04/16 12:57:45 - INSTALL: always: About to import Package #3 'Generic_SNMP_Device.xml.gz'.
2020/04/16 12:57:45 - INSTALL: info: Progress: 43

the log says repeatedly...

2020/04/16 13:15:49 - INSTALL-JSON: debug: Using supplied data
2020/04/16 13:15:49 - INSTALL-JSON: notice: Start: {"Step":"97","Eula":"1"}
2020/04/16 13:15:49 - INSTALL-STEP: notice: Install Parameters: array ( 'Runtime' => 'Web', 'Step' => '97', 'Eula' => '1', )
2020/04/16 13:15:49 - INSTALL-STEP: notice: Initial: '97'
2020/04/16 13:15:49 - INSTALL-STEP: notice: Previously complete: NULL
2020/04/16 13:15:49 - INSTALL-STEP: debug: Does match: '1.2.11'
2020/04/16 13:15:49 - INSTALL-STEP: notice: After: '97'
2020/04/16 13:15:49 - INSTALL-STEP: debug: setStep(): '97'
2020/04/16 13:15:49 - INSTALL-STEP: notice: setStep(): stepError false < '97'
2020/04/16 13:15:49 - INSTALL-LANGUAGE: info: getLanguage(): en-US
2020/04/16 13:15:49 - INSTALL-LANGUAGE: debug: setLanguage(): en-US
2020/04/16 13:15:49 - INSTALL-THEME: info: getTheme(): modern
2020/04/16 13:15:49 - INSTALL-THEME: info: Checking theme: modern
2020/04/16 13:15:49 - INSTALL-THEME: debug: setTheme(): modern
2020/04/16 13:15:49 - INSTALL-STEP: debug: Error: false
2020/04/16 13:15:49 - INSTALL-STEP: debug: Done: '97'
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundTime = 1587034648.4288
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundNeeded =
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: ---------------- Check Expire ----------------
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundDateStarted = 2020-04-16 10:57:28
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundLast = 1587034648.4288
2020/04/16 13:15:49 - INSTALL-BACKGROUND: debug: backgroundExpire = 1587034249
2020/04/16 13:15:49 - INSTALL-JSON: notice:   End: {"Mode":null,"Step":"97","Errors":[],"Eula":"1","Prev":{"Text":"Previous","Step":96,"Enabled":true,"Visible":false},"Next":{"Text":"Next","Step":98,"Enabled":true,"Visible":false},"Test":{"Text":"Test Connection","Step":-
4,"Enabled":true,"Visible":false},"Theme":"modern","StepData":{"Current":"44","Total":100}}

[ParkaMark]

I can confirm the same problem (hanging) on CentOS 8 under Chrome. After many re-attempts, I then switched to using Firefox and the install did actually complete (no hang), but then I was not able to login on the cacti login screen. The login failure is not the same as just a wrong username/invalid password (which it would print in red text if this was the case) - instead, the login just doesn't work, the page refreshes/resets with no red text. And clearing cookies doesn't resolve this problem.

After spending hours on this, I've reverted back to 1.2.10.

Incidentally, during my cookie resetting etc between versions, I did clock that version 1.2.11 does not appear to set the Cacti cookie, but this is present on 1.2.10. At least I think that's what I saw - I don't have the time or patience to troubleshoot this any further.

[misterjmw]

@ParkaMark I have also experienced the 'no login' issue. I have found that I can bypass it by logging in (which redirects back to the login page) then going to a cacti link in my browser (e.g. graph), from there I'm let in. Certainly not a 'real' fix.

[TheWitness]

Unfortunately, I have been unable to reproduce this on either CentOS 8 or CentOS 7 either using Apache or Nginx. Could it be that you guys have some apache configuration setup that is not the default? Are you using https and have you inspected the cookie domain or the cookie settings? You say that there is not Cacti cookie, well that means it has to be environment related.

So, I asked these questions elsewhere:

1. What version of PHP
2. What version of Apache
3. Any customization's to Apache configuration
4. http or https self signed or legit.
5. Cookie domains enabled?
6. Good dns that matches the cookie domain?

[netniV]

If you have copied the sources over and replaced your existing include/cacti.conf file, you will have the cacti_cookie_domain setting present incorrectly and should have been commented out. If you comment that out, it should then work OK. The reason is that the cookie isn't set properly after the upgrade so it is effectively wiped out. If the domain had been set to your websites domain, this would not have been an issue for logging in I believe.

We do still have the outstanding package issue when hitting 42% (local machine package).

[ParkaMark]

@netniV Ok, success. That cacti_cookie_domain setting looks to be absent in the old configuration in version 1.2.10 (I did a quick grep). When I upgrade, I never copy across old configuration files, I re-configure the new ones, and I was not aware of the existence of this new setting. I set cacti_cookie_domain to be my actual domain and everything has worked perfectly, upgraded to 1.2.11 with no issues, using Chrome.

So my advice to others is to make sure cacti_cookie_domain is set correctly - at the moment, that setting seems to be un-commented and set to cacti.net by default, which probably breaks things and might explain this upgrade problem others are experiencing.

[TheWitness]

Cool! Yup that's my fault. Too much late night coding and not enough Zinfandel. Wife is trying to curb my both my consumption during this COVID19 pandemic and well, nope, she's always telling me "go to your office". I guess that's good for Cacti anyway.

[arno-st]

Sorry to be rude, but where is this cacti.conf is coming from ?
Are you talking about config.php ?

[ParkaMark]

@arno-st Yes, I pretty sure @netniV meant config.php, not cacti.conf - config.php is the file that requires modification and where the cacti_cookie_domain setting is located (in version 1.2.11). I'm fairly sure, as has already been said, you can comment this setting out and the code will just use what ever virtual host you're running on, but I've chosen to configure it directly, matching my website's virtual host.

[misterjmw]

While this issue has now addressed the login issue/cookie_domain bug, did the 42%/Package import hang get addressed?

[arno-st]

The login issue yes, the action to do after the process hang at 42% no!

[netniV]

@arno-st Yes, I pretty sure @netniV meant config.php, not cacti.conf

I did, my apologies. Have my head deep in a massive migration using lots of conf files for service configuration so I guess that bleed through unfortunately.

[TheWitness]

I'll take credit for the package issue too. It'll be better in 1.2.12

[hiddenicon]

I have the same issue. 1.2.10 going to 1.2.11. Stuck on Cisco_Rotuer.xml.gz import package. I'm assuming these packages haven't changed for a while, so since I'm working on 1.2.10, I can skip all of the package imports on the 1.2.11 upgrade?

I'm hitting issue#3373 currently with 1.2.10 so would like to upgrade to 1.2.11 soon as I can.

[hiddenicon]

I think I spoke too soon. I came back and the login prompt was on the screen. On version 1.2.11 now. I was able to log in. I will test more. It may have just taken forever to do that import? I wasn't watching it, but it was at least 30 mins.

[TheWitness]

Yea, there is a reason. Best not to re-import the packages unless you want all your graphs updated to the new (or in some cases) old format.

[TheWitness]

This should be fixed now.

[armend-i]

I had the same issue, upgrade hanged on 42% on Centos 7 and no matter how long I waited it did not finish its installation so I had to stop this process by closing the window, since I have many objects in my cacti I hoped I did not loose them, and it looks that it finished its process, at list it works and the version has been upgraded, I'm just wondering if it has left any problems that can occur at some later stages?
Is there a way how I could tshoot its database or cacti itself?

[netniV]

The installer doesn't actually do that much work. Most of it is sanity checks and then the few processes it does in the background (not limited to but most):

• Convert tables if necessary
• Upgrade the database if required
• Import packages
• Setup automatino
• Let you loose

If it was at the import package stage, if you haven't seen any issues by now, I think you should be OK. You could reimport all the packages by hand, but the likelihood is that they were the same anyway, so it wouldn't have changed them.

You should ALWAYS take a database backup before the upgrade though. If you do this via a VM snapshot or a database dump, it's up to you. But with any upgrade, there can be issues - though hopefully we have nailed down these recent ones as we dislike the disruption to you guys as much as you do.

[TheWitness]

It should have finished. You would see errors in the Cacti log otherwise. If you are uncertain, or you find a graph that does not match the template, you can always 'Sync Graphs' from the Graph Templates page to finish the Orphan cleaning a template at a time.

[mdella]

Just did a pkg upgrade via the cacti package in the EPEL. During the upgrade, it got stuck at the 42% mark. After reloading the server and trying again, it "finished" (ie, it skipped where it hung) and completed. However now ALL of my graphs (other than aggregrates) are completely messed up with single colors, missing information, etc. I'm assuming the templates installed incorrectly and now are messing everything else up with them.

Unfortunately I don't know how to go backwards with this one (there is no "uninstaller" other than pkg remove) so was wondering if there was a documented manual method of importing these templates and then resetting all the graphs to use them? Any pointers on this one?

This is an example of all my "in/out" graphs now. Dark green and blue are both missing as well as all the descriptions (ie, in, out). At this point I'm not sure what I'm looking at on any of my graphs anymore.

[bmfmancini]

ah poop that sucks since you are at 1.2.11 how about trying to go to 1.2.12 via source?
that would at least get you to the latest I know that 1.2.11 had a bunch of issues that was patched in 1.2.12

[mdella]

Been trying to keep things at a package level but if I can't figure out the current issue soon, I might have to go to an all manual method. Blah

[netniV]

all the packages are in the install/templates folder and can be imported via the gui or the cli tool import_package.php

[mdella]

Have already tried importing all 6 packages using the import_package tool. Of the 6, 5 had changes that were imported. One apparently had already been processed. However once importing all 6 packages, nothing has changed on the output of the graphs. I have verified however that the packages are now in place as processing a second time comes up with no changes (unlike the first time).

[netniV]

It is likely then you had previously customised the core templates. This reimporting has reset these customisations back to our defaults.

[mdella]

This was a vanilla install. As far as I know, no one here ever touched templates before. Anyway, at this point (see image above) all of our images that are not aggregates (those images seem ok). The importing of the newer templates did not seem to change things at all. Another thing we tried was the sync graphs option. however that one once we hit "continue" basically just sat there and did nothing. never completed and as far as I could tell from ps, etc, never really started anything either.

[mdella]

Ok, checking the logs, every time I try to do a graph sync to get things correct, I get the following error in the cacti.log file:

2020/05/31 00:41:40 - ERROR PHP ERROR: Uncaught Error: Call to undefined function api_data_source_cache_crc_update() in /usr/share/cacti/lib/utility.php:745 Stack trace: #0 /usr/share/cacti/lib/template.php(547): push_out_host('218', '4952') #1 /usr/share/cacti/lib/template.php(882): update_graph_data_source_output_type('5009', 0) #2 /usr/share/cacti/lib/template.php(622): change_graph_template('5009', '41', true) #3 /usr/share/cacti/graph_templates.php(305): retemplate_graphs('41') #4 /usr/share/cacti/graph_templates.php(49): form_actions() #5 {main} thrown in file: /usr/share/cacti/lib/utility.php on line: 745
2020/05/31 00:41:40 - CMDPHP PHP ERROR Backtrace: (CactiShutdownHandler())

Searching back far enough shows that this error was happening during the install update as well.

[netniV]

OK, can I stop you there? You really should open this as a separate issue. Feel free to make a reference to this one but yours seems to be a different, even if related, issue.