Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to upgrade from 23/stable to 24/stable. Mysql won't start. #2597

Closed
develroo opened this issue Oct 29, 2023 · 18 comments
Closed

Unable to upgrade from 23/stable to 24/stable. Mysql won't start. #2597

develroo opened this issue Oct 29, 2023 · 18 comments
Labels

Comments

@develroo
Copy link

Describe the bug

When trying to refresh/upgrade 23/stable to 24/stable refresh is fine but nextcloud.mysql fails to start after. Reverting will go back to a working nextcloud and mysql.

root@nextcloud-home:/# snap refresh  nextcloud --channel=24/stable 
2023-10-29T12:01:41Z INFO Waiting for "snap.nextcloud.apache.service" to stop.
2023-10-29T12:01:45Z INFO Waiting for "snap.nextcloud.mysql.service" to stop.
nextcloud (24/stable) 24.0.12snap1 from Nextcloud✓ refreshed


Oct 29 11:18:45 nextcloud-home systemd[1]: Started Service for snap application nextcloud.mysql.
Oct 29 11:18:45 nextcloud-home audit[18384]: AVC apparmor="DENIED" operation="exec" profile="snap.nextcloud.mysql" name="/bin/systemctl" pid=18384 comm="mysql.server" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 29 11:18:45 nextcloud-home kernel: audit: type=1400 audit(1698578325.948:490): apparmor="DENIED" operation="exec" profile="snap.nextcloud.mysql" name="/bin/systemctl" pid=18384 comm="mysql.server" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 29 11:18:45 nextcloud-home nextcloud.mysql[18324]: Starting MySQL
Oct 29 11:18:48 nextcloud-home nextcloud.mysql[18324]: .. * The server quit without updating PID file (/tmp/pids/mysql.pid).
Oct 29 11:18:48 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 11:18:48 nextcloud-home audit[18672]: AVC apparmor="DENIED" operation="exec" profile="snap.nextcloud.mysql" name="/bin/systemctl" pid=18672 comm="mysql.server" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 29 11:18:48 nextcloud-home kernel: audit: type=1400 audit(1698578328.488:491): apparmor="DENIED" operation="exec" profile="snap.nextcloud.mysql" name="/bin/systemctl" pid=18672 comm="mysql.server" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Oct 29 11:18:48 nextcloud-home nextcloud.mysql[18634]:  * MySQL server PID file could not be found!
Oct 29 11:18:48 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Unit entered failed state.
Oct 29 11:18:48 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Failed with result 'exit-code'.
Oct 29 11:18:48 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Service hold-off time over, scheduling restart.
Oct 29 11:18:48 nextcloud-home systemd[1]: Stopped Service for snap application nextcloud.mysql.
Oct 29 11:18:48 nextcloud-home systemd[1]: Started Service for snap application nextcloud.mysql.

Expected behavior

mysql should start as normal as it works on 23/stable

OS/snapd/snap version

uname -a 
Linux nextcloud-home 4.4.0-1152-raspi2 #163-Ubuntu SMP Fri Apr 16 13:32:53 UTC 2021 armv7l armv7l armv7l GNU/Linux
cat /etc/os-release 
NAME="Ubuntu Core"
VERSION="16"
ID=ubuntu-core
PRETTY_NAME="Ubuntu Core 16"
VERSION_ID="16"
HOME_URL="http://www.snapcraft.io/"
snap info nextcloud
name:      nextcloud
summary:   Nextcloud Server - A safe home for all your data
publisher: Nextcloud✓
store-url: https://snapcraft.io/nextcloud
contact:   https://github.com/nextcloud/nextcloud-snap
license:   unset
description: |
  Access, share and protect your files, calendars, contacts, communication and
  more at home and in your enterprise.
commands:
  - nextcloud.disable-https
  - nextcloud.enable-https
  - nextcloud.export
  - nextcloud.import
  - nextcloud.manual-install
  - nextcloud.mysql-client
  - nextcloud.mysqldump
  - nextcloud.occ
services:
  nextcloud.apache:          simple, enabled, active
  nextcloud.logrotate:       simple, enabled, inactive
  nextcloud.mysql:           simple, enabled, active
  nextcloud.nextcloud-cron:  simple, enabled, active
  nextcloud.nextcloud-fixer: simple, enabled, active
  nextcloud.php-fpm:         simple, enabled, active
  nextcloud.redis-server:    simple, enabled, active
  nextcloud.renew-certs:     simple, enabled, active
snap-id:      njObIbGQEaVx1H4nyWxchk1i8opy4h54
tracking:     24/stable
refresh-date: today at 12:03 UTC
channels:
  latest/stable:    27.1.1snap1               2023-09-25 (37742) 281MB -
  latest/candidate: 27.1.2snap1               2023-10-19 (38363) 281MB -
  latest/beta:      27.1.2snap1+git9.1d25a28  2023-10-25 (38412) 281MB -
  latest/edge:      master-2023-10-19         2023-10-19 (38358) 290MB -
  27/stable:        27.1.1snap1               2023-09-25 (37742) 281MB -
  27/candidate:     ↑                                                  
  27/beta:          ↑                                                  
  27/edge:          27-2023-10-19             2023-10-19 (38360) 281MB -
  26/stable:        26.0.6snap1               2023-09-25 (37836) 258MB -
  26/candidate:     ↑                                                  
  26/beta:          26.0.7snap1+git5.6c44876  2023-10-21 (38399) 258MB -
  26/edge:          26-2023-10-19             2023-10-19 (38350) 258MB -
  25/stable:        25.0.11snap1              2023-09-25 (37839) 249MB -
  25/candidate:     ↑                                                  
  25/beta:          25.0.12snap1+git5.81f61cf 2023-10-21 (38397) 249MB -
  25/edge:          25-2023-10-19             2023-10-19 (38351) 250MB -
  24/stable:        24.0.12snap1              2023-09-07 (37383) 223MB -
  24/candidate:     ↑                                                  
  24/beta:          ↑                                                  
  24/edge:          24-2023-08-12             2023-08-12 (36832) 354MB -
  23/stable:        23.0.12snap1              2023-01-15 (33292) 240MB -
  23/candidate:     ↑                                                  
  23/beta:          ↑                                                  
  23/edge:          23-2023-01-19             2023-01-19 (33360) 240MB -
  22/stable:        22.2.10snap1              2022-07-29 (31275) 228MB -
  22/candidate:     ↑                                                  
  22/beta:          ↑                                                  
  22/edge:          ↑                                                  
  21/stable:        21.0.9snap1               2022-06-13 (29850) 223MB -
  21/candidate:     ↑                                                  
  21/beta:          ↑                                                  
  21/edge:          ↑                                                  
  20/stable:        20.0.14snap1              2021-11-25 (28696) 197MB -
  20/candidate:     ↑                                                  
  20/beta:          ↑                                                  
  20/edge:          ↑                                                  
  19/stable:        19.0.12snap1              2021-07-06 (28383) 238MB -
  19/candidate:     ↑                                                  
  19/beta:          ↑                                                  
  19/edge:          ↑                                                  
  18/stable:        18.0.12snap1              2020-12-16 (25111) 234MB -
  18/candidate:     ↑                                                  
  18/beta:          ↑                                                  
  18/edge:          ↑                                                  
  17/stable:        17.0.10snap1              2020-10-23 (23906) 220MB -
  17/candidate:     ↑                                                  
  17/beta:          ↑                                                  
  17/edge:          ↑                                                  
  16/stable:        16.0.11snap1              2020-06-08 (21461) 200MB -
  16/candidate:     ↑                                                  
  16/beta:          ↑                                                  
  16/edge:          ↑                                                  
  15/stable:        15.0.14snap1              2020-01-20 (18383) 192MB -
  15/candidate:     ↑                                                  
  15/beta:          ↑                                                  
  15/edge:          ↑                                                  
  14/stable:        14.0.10snap1              2019-05-04 (13211) 193MB -
  14/candidate:     ↑                                                  
  14/beta:          ↑                                                  
  14/edge:          ↑                                                  
  13/stable:        13.0.12snap1              2019-05-03 (13163) 184MB -
  13/candidate:     ↑                                                  
  13/beta:          ↑                                                  
  13/edge:          ↑                                                  
  12/stable:        12.0.13snap1              2019-01-07 (10633) 197MB -
  12/candidate:     ↑                                                  
  12/beta:          ↑                                                  
  12/edge:          ↑                                                  
  11/stable:        11.0.8snap1               2018-05-10  (6962) 193MB -
  11/candidate:     ↑                                                  
  11/beta:          ↑                                                  
  11/edge:          ↑                                                  
  10/stable:        10.0.2snap1               2020-06-26   (389) 140MB -
  10/candidate:     ↑                                                  
  10/beta:          ↑                                                  
  10/edge:          ↑                                                  
  9/stable:         9.0.53snap4               2020-06-26   (104) 133MB -
  9/candidate:      ↑                                                  
  9/beta:           ↑                                                  
  9/edge:           ↑                                                  
installed:          24.0.12snap1                         (37383) 223MB -

snap version
snap    2.60.4
snapd   2.60.4
series  16
kernel  4.4.0-1152-raspi2

Logs

.NB Because this is ubutu core the php-fm command it not availible to the system.

nextcloud-log.debug.txt

@scubamuc
Copy link
Member

your log

Oct 29 11:40:42 nextcloud-home nextcloud.apache[2577]: Making sure nextcloud is fully upgraded...
Oct 29 11:40:46 nextcloud-home nextcloud.apache[2577]: Nextcloud or one of the apps require upgrade - only a limited number of commands are available
Oct 29 11:40:46 nextcloud-home nextcloud.apache[2577]: You may use your browser or the occ upgrade command to do the upgrade

so one or more of the installed apps may be causing problems. its not obvious within your logs which apps are causing your issue. so you'll have to do a little research. considering you're on 23/stable you've got several upgrades to do before you reach 27/stable. that might take a while on your raspi2.

try upgrading apps in OCC. you can upgrade or disable apps in OCC. also see FAQ. afterwards try refresh again.

@develroo
Copy link
Author

develroo commented Oct 30, 2023

The apps are upgraded at least in 23/stable. And I believe the /snap/nextcloud/current/snap/hooks/pre-refresh script does that anyway.

if nextcloud_is_installed; then
	# Before attempting an update, or otherwise allowing the refresh to
	# continue, ensure the website that hosts Nextcloud apps is up. If
	# it's down, app updates won't work, and the refresh can go sideways.
	if ! curl -Is --max-time 10 https://apps.nextcloud.com > /dev/null 2>&1; then
		echo "Unable to refresh: apps.nextcloud.com seems to be down, please try again later" >&2
		exit 1
	fi

	if occ -n app:update --all; then
		# app:update downloads and extracts the updates, but now we
		# need to run database migrations, etc.
		occ -n upgrade
	fi
fi

The problem is with Mysql. It is failing to start and it is not easy to get a handle on why that is happening in 24/stable and not 23/stable

From the journalctl log it is clear it is falling over for a reason

-- Logs begin at Sun 2023-10-29 12:43:38 UTC, end at Sun 2023-10-29 13:17:36 UTC. --
Oct 29 13:16:18 nextcloud-home systemd[1]: Started Service for snap application nextcloud.mysql.
Oct 29 13:16:19 nextcloud-home nextcloud.mysql[4856]: Starting MySQL
Oct 29 13:16:21 nextcloud-home nextcloud.mysql[4856]: .. * The server quit without updating PID file (/tmp/pids/mysql.pid).
Oct 29 13:16:21 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 13:16:22 nextcloud-home nextcloud.mysql[5167]:  * MySQL server PID file could not be found!
Oct 29 13:16:22 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Unit entered failed state.
Oct 29 13:16:22 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Failed with result 'exit-code'.
Oct 29 13:16:22 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Service hold-off time over, scheduling restart.
Oct 29 13:16:22 nextcloud-home systemd[1]: Stopped Service for snap application nextcloud.mysql.
Oct 29 13:16:22 nextcloud-home systemd[1]: Started Service for snap application nextcloud.mysql.
Oct 29 13:16:22 nextcloud-home nextcloud.mysql[5213]: Starting MySQL
Oct 29 13:16:24 nextcloud-home nextcloud.mysql[5213]: .. * The server quit without updating PID file (/tmp/pids/mysql.pid).
Oct 29 13:16:24 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 13:16:25 nextcloud-home nextcloud.mysql[5522]:  * MySQL server PID file could not be found!
Oct 29 13:16:25 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Unit entered failed state.
Oct 29 13:16:25 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Failed with result 'exit-code'.
Oct 29 13:16:25 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Service hold-off time over, scheduling restart.
Oct 29 13:16:25 nextcloud-home systemd[1]: Stopped Service for snap application nextcloud.mysql.
Oct 29 13:16:25 nextcloud-home systemd[1]: Started Service for snap application nextcloud.mysql.
Oct 29 13:16:26 nextcloud-home nextcloud.mysql[5563]: Starting MySQL
Oct 29 13:16:28 nextcloud-home nextcloud.mysql[5563]: .. * The server quit without updating PID file (/tmp/pids/mysql.pid).
Oct 29 13:16:28 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 13:16:28 nextcloud-home nextcloud.mysql[5878]:  * MySQL server PID file could not be found!
Oct 29 13:16:28 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Unit entered failed state.
Oct 29 13:16:28 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Failed with result 'exit-code'.
Oct 29 13:16:29 nextcloud-home systemd[1]: snap.nextcloud.mysql.service: Service hold-off time over, scheduling restart.
Oct 29 13:16:29 nextcloud-home systemd[1]: Stopped Service for snap application nextcloud.mysql.
Oct 29 13:16:29 nextcloud-home systemd[1]: Started Service for snap application nextcloud.mysql.
... cont

What is most confusing is the actual log from /writable/system-data/var/snap/nextcloud/37383/logs.mysql_errors.log

2023-10-30T11:33:21.946918Z 0 [Warning] [MY-010139] [Server] Changed limits: max_open_files: 1024 (requested 8161)
2023-10-30T11:33:21.947014Z 0 [Warning] [MY-010142] [Server] Changed limits: table_open_cache: 431 (requested 4000)
2023-10-30T11:33:21.948307Z 0 [Warning] [MY-011037] [Server] The CYCLE timer is not available. WAIT events in the performance_schema will not be timed.
2023-10-30T11:33:21.967628Z 0 [System] [MY-010116] [Server] /snap/nextcloud/37383/bin/mysqld (mysqld 8.0.33) starting as process 21194
2023-10-30T11:33:22Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=ce1c77b346fb6a0d536a30dd1f16f246db8d074e
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
/snap/nextcloud/37383/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x29) [0x1b81b82]
/snap/nextcloud/37383/bin/mysqld(print_fatal_signal(int)+0x281) [0x107a22e]
/snap/nextcloud/37383/bin/mysqld(handle_fatal_signal+0x59) [0x107a31e]
/lib/arm-linux-gnueabihf/libc.so.6(+0x25750) [0x76a34750]
/snap/nextcloud/37383/bin/mysqld(memory::Aligned_atomic<long>::Aligned_atomic()+0x4d) [0x18e4c42]
/snap/nextcloud/37383/bin/mysqld(Delegate::Delegate(unsigned int)+0x39) [0x18e4e9a]
/snap/nextcloud/37383/bin/mysqld(delegates_init()+0x31) [0x18e4fde]
/snap/nextcloud/37383/bin/mysqld(+0xa22532) [0xec9532]
/snap/nextcloud/37383/bin/mysqld(mysqld_main(int, char**)+0x1f5f) [0xecf7f0]
/lib/arm-linux-gnueabihf/libc.so.6(__libc_start_main+0x99) [0x76a260e6]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

@scubamuc
Copy link
Member

is your raspi2 running 32Bit ubuntu core?

@develroo
Copy link
Author

I guess so. It is a RPi 3 and has been using Ubuntu core. It's never been a problem before, but I know Ubuntu core officially supports RPi3

https://ubuntu.com/download/raspberry-pi

@scubamuc
Copy link
Member

okay, so its a raspi3 and yes raspi3 can run 64Bit core. the question is, whether you're running 32Bit or 64Bit core.
If you're on 23/stable then chances are you haven't upgraded for years, thus you may be running a 32Bit core.
we'll need to know for sure.

@Pilzinsel64 do you know if 32Bit mysql is still supported? maybe you can assist?

@develroo
Copy link
Author

develroo commented Oct 30, 2023

Well, the question is how do you upgrade Ubuntu core to 64 bit if it was 32 bit ?

And is this the actual problem.

Can a developer chime in on what happened between the mysql version in 23/stable to 24/stable ?

FWIW this is the pi3 info


snap info pi3
name:      pi3
summary:   Raspberry Pi 3 support package
publisher: Canonical✓
store-url: https://snapcraft.io/pi3
contact:   snaps@canonical.com
license:   unset
description: |
  Support files for booting Raspberry Pi
type:         gadget
snap-id:      kHqvqCxMKWRDGGbxc3NZ13x00otcVry2
tracking:     latest/stable
refresh-date: 2019-09-22
channels:
  latest/stable:    16.04-0.7 2019-07-18 (32) 6MB -
  latest/candidate: 16.04-0.7 2019-07-17 (32) 6MB -
  latest/beta:      16.04-0.7 2019-07-16 (32) 6MB -
  latest/edge:      16.04-0.7 2019-07-16 (32) 6MB -
  18/stable:        –                             
  18/candidate:     –                             
  18/beta:          –                             
  18/edge:          18-1      2018-08-17 (25) 6MB -
installed:          16.04-0.7            (32) 6MB gadget

@scubamuc
Copy link
Member

Well, the question is how do you upgrade Ubuntu core to 64 bit if it was 32 bit

that is not possible.

you'll need to export your data, reinstall nextcloud snap on 64Bit core and reimport your data... but wait until your mysql issue is confirmed by one of the devs.

@develroo
Copy link
Author

To be honest, if I have to reinstall anyway, I am tempted just to put Debian and Docker on and be done with it. I thought ubuntu-core and snaps were to isolate the applications from the OS and, hence, would make maintenance easy.

For sure, I don't see what changed between 23 & 24, but something must have.

@kyrofa
Copy link
Member

kyrofa commented Oct 30, 2023

If you're on 23/stable then chances are you haven't upgraded for years, thus you may be running a 32Bit core.
we'll need to know for sure.

I'm not sure this matters, does it? We have published armhf snaps from the beginning, and still do. This is supported.

Can a developer chime in on what happened between the mysql version in 23/stable to 24/stable ?

You can actually find that out for yourself with a quick glance at the CHANGELOG. I'm assuming you're updating from 23.0.5snap1 to 24.0.7snap1, which covers two mysql updates, one to 8.0.30, and another to 8.0.31. It's possible that you're being bitten by a bug in one of those, but it's also possible you simply ran out of RAM during the upgrade process and it corrupted the database. Given your use of the pi3, that seems like the most likely scenario.

I suggest removing the failed upgrade revision and trying the refresh again, keeping a careful eye on your RAM.

I thought ubuntu-core and snaps were to isolate the applications from the OS and, hence, would make maintenance easy.

That's true, and as far as I can tell isn't invalidated by what you and @scubamuc were discussing: if you need to reinstall your OS, no level of separation between your apps and your OS will save you from needing to backup and reinstall the apps on the OS. Even HA systems like Kubernetes work that way, by migrating the apps off the OS to another one before taking the node down.

@develroo
Copy link
Author

It's possible that you're being bitten by a bug in one of those, but it's also possible you simply ran out of RAM during the upgrade process and it corrupted the database. Given your use of the pi3, that seems like the most likely scenario.

If that were the case why would reverting it back to 23/stable fix it? It is the same DB obviously so I don't think it is corrupted, but the new mysql barfs anyway.

@kyrofa
Copy link
Member

kyrofa commented Oct 30, 2023

If that were the case why would reverting it back to 23/stable fix it?

Because of the way this snap utilizes snapd features. When you upgrade from revision A to revision B, snapd makes a copy of revision A's data (which includes the database), then fires up revision B on that data. Revision B has a new mysql in it, upgrades the database, and chokes for whatever reason in the middle, corrupting it. So you revert back to revision A, which fires up on the exact same data right before the copy was made, i.e. before B garbled the database.

For completeness, note that the snap does not by default do this for Nextcloud's data directory. It's just too large to make it feasible for most installations, so there's only ever one copy of that.

@develroo
Copy link
Author

Fairy nuff, but how does that really help me to debug the issue? Mysql seems unable to start before it is accessing databases else that would be shown in the stack trace. As it is there is just the weird ? over a libc error causing a segfault, which seems to indicate the underlying system is not compatible. In which case how are the snaps isolated from the host system dependencies then ?

@scubamuc
Copy link
Member

@develroo so I've had another look at your logs, especially possible issues with 3rd party apps that have been deprecated or are not upgradeable... you have a fairly long list:

Oct 29 11:40:46 nextcloud-home nextcloud.apache[2577]: Turned on maintenance mode
Oct 29 11:40:51 nextcloud-home nextcloud.apache[2577]: Updating database schema
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updated database
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updated to 1.13.0
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updating <lookup_server_connector> ...
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updated <lookup_server_connector> to 1.11.0
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updated to 1.11.0
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updating <password_policy> ...
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updated <password_policy> to 1.13.0
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:33 nextcloud-home nextcloud.apache[2577]: Updated to 1.18.0
Oct 29 11:41:34 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:35 nextcloud-home nextcloud.apache[2577]: Updated to 23.1.2
Oct 29 11:41:35 nextcloud-home nextcloud.apache[2577]: Updating <cloud_federation_api> ...
Oct 29 11:41:35 nextcloud-home nextcloud.apache[2577]: Updated <cloud_federation_api> to 1.6.0
Oct 29 11:41:35 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Fix broken values of calendar objects
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Starting ...
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: 0/0 [>---------------------------] 0%
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Clean up old calendar subscriptions from deleted users that were not cleaned-up
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Starting ...
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: 0/0 [->--------------------------] 0%
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Updated to 1.21.0
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Updating <files_sharing> ...
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Updated <files_sharing> to 1.15.0
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Updating <files_trashbin> ...
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Updated <files_trashbin> to 1.13.0
Oct 29 11:41:36 nextcloud-home nextcloud.apache[2577]: Updating <files_versions> ...
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updated <files_versions> to 1.16.0
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updated to 1.13.0
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updated to 2.5.0
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updated to 1.13.0
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updated to 2.12.0
Oct 29 11:41:37 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updated to 2.11.2
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updated to 1.13.0
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Migrate old user config
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Starting ...
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: 0/0 [--->------------------------] 0%
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updated to 1.9.0
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updated to 1.4.0
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updated to 7.3.0
Oct 29 11:41:50 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated to 1.13.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating <files_pdfviewer> ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated <files_pdfviewer> to 2.4.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating <files_rightclick> ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated <files_rightclick> to 1.2.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating <files_videoplayer> ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated <files_videoplayer> to 1.12.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated to 2.8.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated to 1.5.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated to 1.7.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating <provisioning_api> ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated <provisioning_api> to 1.13.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated to 1.2.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated to 1.13.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated to 1.5.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated to 1.6.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating <survey_client> ...
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updated <survey_client> to 1.11.0
Oct 29 11:41:51 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:41:56 nextcloud-home nextcloud.apache[2577]: Updated to 3.4.1
Oct 29 11:41:56 nextcloud-home nextcloud.apache[2577]: Updating <twofactor_backupcodes> ...
Oct 29 11:41:56 nextcloud-home nextcloud.apache[2577]: Updated <twofactor_backupcodes> to 1.12.0
Oct 29 11:41:56 nextcloud-home nextcloud.apache[2577]: Updating <user_status> ...
Oct 29 11:42:09 nextcloud-home nextcloud.apache[2577]: Updated <user_status> to 1.3.1
Oct 29 11:42:09 nextcloud-home nextcloud.apache[2577]: Updating ...
Oct 29 11:42:09 nextcloud-home nextcloud.apache[2577]: Updated to 1.7.0
Oct 29 11:42:09 nextcloud-home nextcloud.apache[2577]: Updating <weather_status> ...
Oct 29 11:42:09 nextcloud-home nextcloud.apache[2577]: Updated <weather_status> to 1.3.0
Oct 29 11:42:37 nextcloud-home nextcloud.apache[2577]: Starting code integrity check...
Oct 29 11:47:50 nextcloud-home nextcloud.apache[2577]: Finished code integrity check
Oct 29 11:47:50 nextcloud-home nextcloud.apache[2577]: Update successful
Oct 29 11:47:50 nextcloud-home nextcloud.apache[2577]: Turned off maintenance mode

The only app I can't relate to is lookup server connector. I'm guessing you've installed this manually, since its not available in the appstore as far as I can tell. Try disabling any/all 3rd party apps before running an upgrade and see if that works for you.

@scubamuc
Copy link
Member

Ooops... I err... - lookup_server_connector: 1.15.0 this is not 3rd party and is a global install in Nextcloud!

sorry about that!

here is a working app list from my testing instance for comparisson:

Enabled:
  - activity: 2.19.0
  - announcementcenter: 6.7.0
  - bruteforcesettings: 2.7.0
  - calendar: 4.6.0
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.5.0
  - contactsinteraction: 1.8.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - external: 5.2.1
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_external: 1.19.0
  - files_fulltextsearch: 27.0.1
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - fulltextsearch: 27.0.3
  - groupfolders: 15.3.2
  - guests: 2.5.1
  - impersonate: 1.14.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - mail: 3.5.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.9.1
  - notifications: 2.15.0
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - quota_warning: 1.18.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - richdocuments: 8.2.3
  - richdocumentscode: 23.5.604
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - spreed: 17.1.4
  - support: 1.10.0
  - survey_client: 1.15.0
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - admin_audit: 1.17.0
  - encryption: 2.15.0
  - twofactor_totp: 9.0.0
  - user_ldap: 1.17.0

@develroo
Copy link
Author

Well, I can try and disable all plugins I guess. I'll let you know.

@develroo
Copy link
Author

So FWIW I disabled as many apps as I could

nextcloud.occ app:list|awk -F - '{print $2}'|awk -F : '{print $1}' | head -n -7 | while read line ; do nextcloud.occ app:disable "$line" ; done 
No such app enabled: 
accessibility 1.9.0 disabled
activity 2.15.0 disabled
circles 23.1.2 disabled
cloud_federation_api can't be disabled.
comments 1.13.0 disabled
contactsinteraction 1.4.0 disabled
dashboard 7.3.0 disabled
dav can't be disabled.
federatedfilesharing can't be disabled.
federation 1.13.0 disabled
files can't be disabled.
files_pdfviewer 2.4.0 disabled
files_rightclick 1.2.0 disabled
files_sharing 1.15.0 disabled
files_trashbin 1.13.0 disabled
files_versions 1.16.0 disabled
files_videoplayer 1.12.0 disabled
firstrunwizard 2.12.0 disabled
logreader 2.8.0 disabled
lookup_server_connector can't be disabled.
notifications 2.11.2 disabled
oauth2 can't be disabled.
password_policy 1.13.0 disabled
photos 1.5.0 disabled
privacy 1.7.0 disabled
provisioning_api can't be disabled.
recommendations 1.2.0 disabled
serverinfo 1.13.0 disabled
settings can't be disabled.
sharebymail 1.13.0 disabled
support 1.6.0 disabled
survey_client 1.11.0 disabled
systemtags 1.13.0 disabled
text 3.4.1 disabled
twofactor_backupcodes can't be disabled.
user_status 1.3.1 disabled
viewer can't be disabled.
weather_status 1.3.0 disabled
workflowengine can't be disabled.

and re-ran the refresh.

snap refresh  nextcloud --channel=24/stable 
2023-12-22T13:37:13Z INFO Waiting for "snap.nextcloud.mysql.service" to stop.
2023-12-22T13:37:20Z INFO Waiting for "snap.nextcloud.php-fpm.service" to stop.
2023-12-22T13:37:21Z INFO Waiting for "snap.nextcloud.apache.service" to stop.
nextcloud (24/stable) 24.0.12snap1 from Nextcloud✓ refreshed

And now the mysql snap seems to start ok

systemctl status snap.nextcloud.mysql.service
● snap.nextcloud.mysql.service - Service for snap application nextcloud.mysql
   Loaded: loaded (/etc/systemd/system/snap.nextcloud.mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2023-12-22 13:44:35 UTC; 2s ago
  Process: 4553 ExecStop=/usr/bin/snap run --command=stop nextcloud.mysql (code=exited, status=0/SUCCESS)
 Main PID: 4598 (start_mysql)
   CGroup: /system.slice/snap.nextcloud.mysql.service
           ├─4598 /bin/sh -e /snap/nextcloud/37383/bin/start_mysql
           ├─4641 /bin/sh /snap/nextcloud/37383/support-files/mysql.server start
           └─4832 sleep 1

Dec 22 13:44:35 nextcloud-home systemd[1]: Started Service for snap application nextcloud.mysql.
Dec 22 13:44:35 nextcloud-home nextcloud.mysql[4598]: Starting MySQL

However, there is no web interface and you cannot configure it from the command line any more.

nextcloud.occ status
Waiting for PHP... ^C

This is because the php service is still waiting to connect to mysql

systemctl status snap.nextcloud.php-fpm.service
● snap.nextcloud.php-fpm.service - Service for snap application nextcloud.php-fpm
   Loaded: loaded (/etc/systemd/system/snap.nextcloud.php-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2023-12-22 13:38:40 UTC; 7min ago
 Main PID: 1727 (start-php-fpm)
   CGroup: /system.slice/snap.nextcloud.php-fpm.service
           ├─ 1727 /bin/sh /snap/nextcloud/37383/bin/start-php-fpm
           └─16768 sleep 1

Dec 22 13:38:40 nextcloud-home systemd[1]: Started Service for snap application nextcloud.php-fpm.
Dec 22 13:38:42 nextcloud-home nextcloud.php-fpm[1727]: Waiting for MySQL...

So still broken, alas.

@scubamuc
Copy link
Member

@develroo well I guess it was worth a try?

Were you able to revert to a working system?

Here is a similar issue with more or less the same result, but stuck on a later version of the snap.

I'm afraid I'm all out of ideas on how to solve this.

Copy link

This issue is stale because it has been without activity for 60 days. It will be closed after 7 more days of inactivity.

@github-actions github-actions bot added the Stale label Feb 21, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants