New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WORKAROUND] NC14 RC2: occ upgrade hangs on running cronjob it states #10949

Closed
eggithub opened this Issue Aug 31, 2018 · 35 comments

Comments

Projects
None yet
@eggithub

eggithub commented Aug 31, 2018

I cant give to much info right now due to lack of time but this is the command line output of the upgrade process:
sudo -u www-data php /var/www/nextcloud/occ upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …

A cronjob was running at that time, killed the cron process but occ upgrade keeps giving this output.

When im home I provide more info (mysql/php7/etc..).
Tell me what else you need apart from the template. I use my own upgrade script. perhaps that might help.

to be continued...

Cheers

@eggithub eggithub changed the title from NC14 RC2: occ upgrade hangs running cronjob to NC14 RC2: occ upgrade hangs on running cronjob it states Aug 31, 2018

@nextcloud-bot

This comment has been minimized.

Member

nextcloud-bot commented Aug 31, 2018

GitMate.io thinks possibly related issues are #6960 (occ upgrade is outdated), #10918 (Occ Upgrade forces to run once a day), #9562 (Check before running "upgrade" if cronjob is still running and wait for it to finish), #8092 (Upgrade notice should not be displayed when occ is run with --output=json), and #4726 (Upgrading Nextcloud sometimes causes Cronjob Deadlocks).

@snopy234

This comment has been minimized.

snopy234 commented Aug 31, 2018

Try to change your current version in [nextcloud path]/config/config.php to a lower version. In my case i've changed 14.0.0.17 to 14.0.0.0
Then use ./occ upgrade to upgrade your nextcloud. I hope this helps for you and sorry for my language problems. Have a nice day!
PS: Please let me know if it has worked.

@danielkesselberg

This comment has been minimized.

Contributor

danielkesselberg commented Aug 31, 2018

See #9992 for more information about this issue

@awrede

This comment has been minimized.

awrede commented Sep 1, 2018

@snopy234 I ran into the same issue. Changing version in config.php from 14.0.0.17 to 14.0.0.0 and then running ./occ upgrade made the upgrade run to completion.

@eggithub eggithub changed the title from NC14 RC2: occ upgrade hangs on running cronjob it states to [WORKAROUND] NC14 RC2: occ upgrade hangs on running cronjob it states Sep 1, 2018

@eggithub

This comment has been minimized.

eggithub commented Sep 1, 2018

disabled the nextcloud cronjob
waited for >12 hours
then I was able to update

@snopy234

This comment has been minimized.

snopy234 commented Sep 1, 2018

@awrede nice to hear that it's working :)

@hchristo

This comment has been minimized.

hchristo commented Sep 15, 2018

Try to change your current version in [nextcloud path]/config/config.php to a lower version. In my case i've changed 14.0.0.17 to 14.0.0.0
Then use ./occ upgrade to upgrade your nextcloud. I hope this helps for you and sorry for my language problems. Have a nice day!
PS: Please let me know if it has worked.

Thanks, this ben Worked, Working, yes Working... THX!

@jgrete

This comment has been minimized.

jgrete commented Sep 26, 2018

This issue is still present in 14.0.1 and the workaround is the same.
Very strange...

@snopy234

This comment has been minimized.

snopy234 commented Sep 26, 2018

Is my workaround still working? I havn't any problems since i changed the version

@jgrete

This comment has been minimized.

jgrete commented Sep 26, 2018

@snopy234 👍
Yepp! THanks a lot!

@Larx

This comment has been minimized.

Larx commented Sep 27, 2018

Workaround does not work for me::
Nextcloud or one of the apps require upgrade - only a limited number of commands are available You may use your browser or the occ upgrade command to do the upgrade Set log level to debug Updating database schema Updated database Checking for update of app bookmarks in appstore An unhandled exception has been thrown: Error: Call to undefined method GuzzleHttp\Client::request() in /srv/www/htdocs/nextcloud/lib/private/Http/Client/Client.php:161 Stack trace: #0 /srv/www/htdocs/nextcloud/lib/private/App/AppStore/Fetcher/Fetcher.php(106): OC\Http\Client\Client->get('https://apps.ne...', Array) #1 /srv/www/htdocs/nextcloud/lib/private/App/AppStore/Fetcher/AppFetcher.php(80): OC\App\AppStore\Fetcher\Fetcher->fetch('', '') #2 /srv/www/htdocs/nextcloud/lib/private/App/AppStore/Fetcher/Fetcher.php(170): OC\App\AppStore\Fetcher\AppFetcher->fetch('', '') #3 /srv/www/htdocs/nextcloud/lib/private/Installer.php(385): OC\App\AppStore\Fetcher\Fetcher->get() #4 /srv/www/htdocs/nextcloud/lib/private/Updater.php(445): OC\Installer->isUpdateAvailable('bookmarks') #5 /srv/www/htdocs/nextcloud/lib/private/Updater.php(256): OC\Updater->upgradeAppStoreApps(Array) #6 /srv/www/htdocs/nextcloud/lib/private/Updater.php(126): OC\Updater->doUpgrade('14.0.1.1', '14.0.0.1') #7 /srv/www/htdocs/nextcloud/core/Command/Upgrade.php(262): OC\Updater->upgrade() #8 /srv/www/htdocs/nextcloud/3rdparty/symfony/console/Command/Command.php(251): OC\Core\Command\Upgrade->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #9 /srv/www/htdocs/nextcloud/3rdparty/symfony/console/Application.php(946): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #10 /srv/www/htdocs/nextcloud/3rdparty/symfony/console/Application.php(248): Symfony\Component\Console\Application->doRunCommand(Object(OC\Core\Command\Upgrade), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #11 /srv/www/htdocs/nextcloud/3rdparty/symfony/console/Application.php(148): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #12 /srv/www/htdocs/nextcloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #13 /srv/www/htdocs/nextcloud/console.php(96): OC\Console\Application->run() #14 /srv/www/htdocs/nextcloud/occ(11): require_once('/srv/www/htdocs...')

Here the bookmarks app causes the trouble, but before the activity and accessibility app caused the problem. I disabled them.

How to go on?

PS: Nextcloud causes trouble at almost every update. As much as I like it, it would be great if the update process would me much more robust.

@Larx

This comment has been minimized.

Larx commented Sep 27, 2018

Disabling files_external_gdrive did the job... (wasn't there a time when nextcloud disabled all but the core apps by itself before updating?)

@jgrete

This comment has been minimized.

jgrete commented Sep 27, 2018

I hadn't activated that plugin, so the cause of your trouble was different.
Yes, I'm a little bit fed up also with these ever crahing updates. On most updates cycles I wasn't able to do a smooth update on my home installation. I could recover from all chrashes, but on my companies installation I'm much more conservative when it comes to nextcloud updates...

@cvandesande

This comment has been minimized.

cvandesande commented Oct 1, 2018

Also got this after the 14.0.1 upgrade. The workaround from @hchristo worked

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Oct 1, 2018

A cronjob was running at that time, killed the cron process but occ upgrade keeps giving this output.

Killing the cron job caused the trouble actually here, because we store a "is something running" timestamp for each cron job and it only get updated by the job itself once it finished. The cronjob also just finishes the currently running one and then updates the DB table.

Best would be the SQL query from #9992 (comment), but be aware that if the cron.php is still running somewhere that this could cause some issues.

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Oct 1, 2018

Also this would be helpful to fix the cause of this issue:

SELECT * FROM oc_jobs WHERE reserved_at <> 0;
@snopy234

This comment has been minimized.

snopy234 commented Oct 3, 2018

@cvandesande which workaround from @hchristo do you mean?

@cvandesande

This comment has been minimized.

cvandesande commented Oct 3, 2018

@snopy234 where you downgrade the version in config.php to 14.0.0.0

@michidk

This comment has been minimized.

michidk commented Oct 5, 2018

Same issue. Query (@MorrisJobke):

SELECT * FROM oc_jobs WHERE reserved_at <> 0;

oc_jobs

id

class

argument

last_run

last_checked

reserved_at

execution_duration

748

OC\Preview\BackgroundCleanupJob

null

1538766002

1538766001

1538766001

0

Also tried updating Nextcloud manually, following this guide: https://docs.nextcloud.com/server/13/admin_manual/maintenance/manual_upgrade.html - with the same result: stuck, waiting for cron to finish.

Is there an hotfix to get Nextcloud working again?

@michidk

This comment has been minimized.

michidk commented Oct 5, 2018

Fixed it by setting the results of the query SELECT * FROM oc_jobs WHERE reserved_at <> 0; at field reserved_at to 0.

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Oct 8, 2018

cc @rullzer for the background job in previews :/

@GLLM

This comment has been minimized.

GLLM commented Oct 11, 2018

Hi all, I also have this issue.

  • I got it while upgrading from 14.0.0 to 14.0.1. Solved it by changing version in config.php from 14.0.0.17 to 14.0.0.0 and then running ./occ upgrade

today, upgraded from 14.0.1 to 14.0.2 and also got it.
I'll try @michidk workaround ...

Feedback/Edit : solved by doing SELECT * FROM oc_jobs WHERE reserved_at <> 0;
and setting it to 0 on the one line that I got.

Thanks

@moretocome

This comment has been minimized.

moretocome commented Oct 12, 2018

Saw this error for the first time when upgrading from 14.0.2 to 14.0.3 with commandline updater.phar. Solved by changing version to 14.0.0.0 in config.php

@pkmkrishnakumar

This comment has been minimized.

pkmkrishnakumar commented Oct 12, 2018

@GLLM

How to do this please give in detail here (For novice user)

This error occured during 14 ---> 14.1 & now 14.1 -----> 14.2

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Oct 12, 2018

How to do this please give in detail here (For novice user)

This error occured during 14 ---> 14.1 & now 14.1 -----> 14.2

See my comment in #10949 (comment)

@GLLM

This comment has been minimized.

GLLM commented Oct 12, 2018

Hi all
Got the same issue once again from 14.0.2 > 14.0.3

Solved (again) it by looking in the oc_jobs table, found the line with field reserved_at <> 0, and set it up back to 0.

Thanks

@benneti

This comment has been minimized.

benneti commented Oct 12, 2018

Hi,
Got the same issue (for the first time) from 14.0.2 > 14.0.3,
The fix suggested by @snopy234 worked, so thank you for this.

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Oct 12, 2018

The big question is, why there are those entries. What type of cron job do you have? Do you run them in ajax mode?

@eggithub

This comment has been minimized.

eggithub commented Oct 12, 2018

I just use the Cron option for background jobs.

Could it be related to the News App? I reinstalled that before I updated the nextcloud instance. I know from experience that News takes a long time to update after installing a new version because it happens dat my feeds aren't accessible for hours. And since this app also uses cron for updates, could it be that it is holding up something. Just guessing... But can I check this Cron update behaviour somewhere in the logs?

Running this nextcloud instance on an RPI3...

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Oct 12, 2018

Could it be related to the News App? I reinstalled that before I updated the nextcloud instance. I know from experience that News takes a long time to update after installing a new version because it happens dat my feeds aren't accessible for hours. And since this app also uses cron for updates, could it be that it is holding up something. Just guessing... But can I check this Cron update behaviour somewhere in the logs?

Not really - except it is a news background job, but it seems to be many different ones in here.

@benneti

This comment has been minimized.

benneti commented Oct 12, 2018

I also use cron (provided by debian stable package) and have the news app installed

@enoch85

This comment has been minimized.

Member

enoch85 commented Oct 16, 2018

Sorry for bumping, just got this on one of the servers I maintain going from 14.0.1 -- 14.0.3.

root@cloud:/var/scripts# sudo -u www-data php /var/www/nextcloud/occ upgrade
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Turned on maintenance mode
Waiting for cron to finish (checks again in 5 seconds) …
Waiting for cron to finish (checks again in 5 seconds) …
^C
root@cloud:/var/scripts# sudo -u postgr^C
root@cloud:/var/scripts# mysql -u root 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 922369
Server version: 5.7.23-0ubuntu0.16.04.1 (Ubuntu)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SELECT * FROM oc_jobs WHERE reserved_at <> 0;
ERROR 1046 (3D000): No database selected
mysql> use nextcloud_db
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> SELECT * FROM oc_jobs WHERE reserved_at <> 0;
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
| id | class                                          | argument | last_run   | last_checked | reserved_at | execution_duration |
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
| 20 | OCA\Survey_Client\BackgroundJobs\MonthlyReport | null     | 1539676802 |   1539676802 |  1539676802 |                 13 |
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
1 row in set (0,00 sec)

mysql> SELECT * FROM oc_jobs WHERE reserved_at <> 0;
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
| id | class                                          | argument | last_run   | last_checked | reserved_at | execution_duration |
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
| 20 | OCA\Survey_Client\BackgroundJobs\MonthlyReport | null     | 1539676802 |   1539676802 |  1539676802 |                 13 |
+----+------------------------------------------------+----------+------------+--------------+-------------+--------------------+
1 row in set (0,00 sec)

mysql> UPDATE oc_jobs SET reserved_at=0
    -> ;
Query OK, 1 row affected (0,00 sec)
Rows matched: 24  Changed: 1  Warnings: 0

mysql> SELECT * FROM oc_jobs WHERE reserved_at <> 0;
Empty set (0,00 sec)

mysql> exit
Bye
root@cloud:/var/scripts# nano nextcloud_update.sh 
root@cloud:/var/scripts# nano nextcloud_update.sh 
root@cloud:/var/scripts# bash nextcloud_update.sh 
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Updating database schema
Updated database
Disabled incompatible app: bruteforcesettings
Disabled incompatible app: calendar
Disabled incompatible app: contacts
Disabled incompatible app: impersonate
Checking for update of app accessibility in appstore
Checked for update of app "accessibility" in appstore 
Checking for update of app activity in appstore
Checked for update of app "activity" in appstore 
Checking for update of app cloud_federation_api in appstore
Checked for update of app "cloud_federation_api" in appstore 
Checking for update of app comments in appstore
Checked for update of app "comments" in appstore 
Checking for update of app dav in appstore
Checked for update of app "dav" in appstore 
Checking for update of app federatedfilesharing in appstore
Checked for update of app "federatedfilesharing" in appstore 
Checking for update of app federation in appstore
Checked for update of app "federation" in appstore 
Checking for update of app files in appstore
Checked for update of app "files" in appstore 
Checking for update of app files_external in appstore
Checked for update of app "files_external" in appstore 

All good now.

@Larx

This comment has been minimized.

Larx commented Oct 16, 2018

Any idea how to get rid of this? I'm kind of sick of going into manual mode to repair NC after every update. Please please make the updates more robust!!

@GAS85

This comment has been minimized.

GAS85 commented Oct 22, 2018

Meet the same issue during update to 14.0.3

@MorrisJobke

This comment has been minimized.

Member

MorrisJobke commented Nov 2, 2018

We removed this via #12188

Will be shipped in 14.0.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment