Issues with v28 - 'sudo nextcloud.occ upgrade;' fails on very large databases #2758
Comments
|
And as a note - thank you again for the amazing product. HPF works great recently and I enjoy your snap 99% of time upgrades perfectly every time. As I have a VM snapshot now, i'm happy to experiment. I can do whatever you like on my prod environment backup as it is a VM snapshot I can restore and test and break and rewind. Hopefully an ideal situation to help. |
|
Further note: |
|
I'm also on Ubuntu LTS 22.04 (but x86/64) and the upgrade to 28 went without issue so we can eliminate something global here: |
|
Thank you for confirming Interesting is I have attempted it a couple of times now (thanks to VM snapshot) and it fails every time so it's not a fluke I might try disabling all apps next (like nextcloud mail etc) and see if works better |
|
@LMRW thanks for your request and your heads up
with your help we'll surely reach 100% 👍 run the debugging script and post here
while I'd agree, lets see if we can help @LMRW solve his issue.
yip, that's exactly what @user8446 is suggesting but not ALL only 3rd party apps |
|
So With the error screens you were getting earlier in the week, MySQL monitor not working, and the specific MySQL error this seems to point in that direction. |
|
I will do a deep dive tomorrow and post the debug script but I wanted to immediately mention this incase relevant I read this: #2734 This week I (not via snap) installed cmake on this VM for the first time. Could that be related? |
could it? |
4 instances (x86/64) upgraded without issues for me too. had some issues with "News" app not being up to date, but activated it anyway without further issues. |
|
The upgrade to 28 worked well for me with Ubuntu 22.04 LTS (x86_64), but I had to restart the cron I also use the News app, version 24.0.0 is not supportet for NC28. You can activate it on NC28, then the sitebar layout is brocken, you have to adjust the custom.css there. Templates are available on Github / News, or you can manually install the unstable version 25.0.0. I would like to take this opportunity to thank the Nextcloud Snap team for their hard work in bringing NC28 to Snap. I read in your issue how much effort went into it this time for this Version. Great! |
confirmed this:
would you mind creating a new issue to track this? |
|
|
done.... |
|
When I searched for the problem about News App, I first found the workaround for custom.css nextcloud/news#2610 and I try it, only later did I see the nextcloud/news#2585 for install the unstable 25.0.0 What still made me suspicious was the question of whether the unstable channel updates automatically or how do you go from unstable back to stable. But if the new version is so cool, maybe I should try it 😉 |
|
yes I agree, I'm thinking along the same lines. how to get back to stable?
we'll have to watch nextcloud/news#2585
yeah it is quite spiffy, we could watch this together. dunno 'bout you, but I rely heavily on my news app |
|
I also have nextcloud news installed but haven't seen any issues with that specifically as nothing is booting at all. I still plan for a deep dive later today. Will report all I find. Thank you everyone |
|
@LMRW, if all else fails, you can revert to last working good like @adrianvg did here: #2759 (comment) |
|
Thank you. I read previously we cannot skip major versions? So I cannot just skip 28 and go 27->29 when released? The comment in #2759 (comment) seems to indicate thats possible? |
yeah, I feel the same, the news app is one of my most important ones. I have now installed the unstable version. Then a message appeared: "Missing optional index "news_feeds_deleted_at_index" in the table "news_feeds.", I guess the new version came with new indexes. So I ran Later I noticed that the cron job on Nextcloud-Sanp had failed again, the failure started exactly at the time when I installed the News app unstable app. |
that is correct, but last known good was 28.05 if I'm not mistaken? but this stifles me also:
@Pilzinsel64 cloud you make a suggestion? |
|
@scubamuc That's ok, as 28.0.6snap1 is the latest one and active. But why 27.1.9snap1 actually is in stable might be just a mistake. However, as long as it's disabled it's all fine. Shouldn't happen again for the next update. Also, yes, last known good was 28.0.5. but for me 28.0.6 is also nicely working (as manual install and as snap). You can also not skip an major version from side of nextcloud. It will lizery break your instance if you skip necessary migration steps.
I can't imagine that it has an negative side effect. We moved to make snap, yes, but only relevant for building the snap. |
@LMRW that's not possible. What @adrianvg did was revert to the
@scubamuc no, 28.0.5 was never released to the
@Pilzinsel64 it's not a mistake: it is the exact version that was in |
|
I don't know if this will help, but my own server wasn't working, so I watched the logs, and saw that Circles was throwing errors. So I disabled it and everything worked perfectly. |
|
I disabled all non-default apps. I stopped nextcloud
I run the below: Response was: I started Nextcloud up... still v27,. still working.
Now I run a second time... this time I do not stop it first, his time it worked to upgrade the snap. Now I run The response was I have no non-default apps and it is not working. One clue: This is a large Nextcloud instance with 5TB of external data. The mysql databse is over 6GB of file cache. Could it be so big the upgrade is timing out? |
|
@scubamuc I run the debug script - its a bit too much with some personal information for me to paste the whole output publicly. Is there anything in particular to share? Some points of interest below though: I saw this repeatedly:
This as well: Some redis errors: Also showed multiple times
|
|
Im thinking now maybe my mysql database is abnormally large? Its about 14GB uncompressed and a zipped dump about 5GB. But its always been about this size and previous nextcloud snap updates worked. I use external_files to connect to a very large storage drive so its all file cache data |
|
OK, GOOD NEWS I now have NC 28 working! 😃 It was indeed my database was too large which was the issue. I started with a 14GB nextcloud MySQL database which could not be upgraded by nextcloud snap as part of the v28 upgrade process. For whatever reason, a database that large fails during The reason my database was so large, is a database can corrupt and grow exponentially when using the files_external app with large external locations, and when this occurs, nextcloud snap cannot seemlessly upgrade a user without error as it cannot handle databases that large. files_external has a known issue where fast moving folders can cause trumendously large databases of old caches (like a video editors server editing lots of online media, with rendering files and caches etc -- they generate, move, delete fast and files_external caches them all and never removes them even when the files are gone) So to fix this issue for myself, essentially I managed to shrink my database down from 14gb to 500mb perform a snap upgrade... The Steps I took
(And before I run below, I followed https://github.com/nextcloud-snap/nextcloud-snap/wiki/Backup-and-Restore to perform a snap backup of my working v27)
But now NC is working, I can next re-add my files_external shares in nextcloud web admin inside nextcloud 28... once they are all re-added in the web admin, I ran the below to scan them and double check everything is still good condition:
I now check my database size once more, and it is only 500mb, which is shocking but true, as it is now a cache of the EXACT same set of files and went from 14GB down by approx 13.5GB.
Then I run the below commands That cleared out the massive undo_001 and undo_002 files, getting my SSD space free! Now I finally was not wasting space and had NC28 snap. KNOWN ISSUES WITH THIS SOLUTION:
Otherwise web admin was complaining about the very high error count.
RESULTS IN SUMMARY Nextcloud has a bug or design flaw where the database grows over years and can never be shrinked back down. Essentially, if you create, then move or delete a file, it is not removed from the database by Nextcloud. So fast moving folders with a lot of activity (like video editors) can cause insanely large databases. I cleared it right down by removing everything to nothing by local files only at 300mb and when I rescanned my externals fresh again, it was only 500mb. At the 300mb point in my process, nextcloud snap v28 successfully installed. But I assume at the 500mb point it would have been fine too. NOTES:
https://www.reddit.com/r/NextCloud/comments/du62gw/nextcloud_and_external_storage/?rdt=44521 My steps above though resolved this issue for me.
And key note: 5) I will leave this issue open please as the issue now is "users with very large databases cannot upgrade"? We now at least know the issue for me was that. -- And thank you everyone above in comments, and those in the URLs I linked, and of course the amazing nextcloud and nextcloud snap devs. It all really helped me get NC 28 up and running. I hope above was helpful to someone. |
|
TLDR: |
LMRW
changed the title
job well done and thanks for your detailed description. this thread will be an information goldmine for large database issues.
... personal information is minimal and generally not a leak-issue and the debug script absolutely necessary for supporting issues here. luckily you were able to triage the issue yourself. 👍
definitely so! most of the database scanning and cleaning jobs mentioned in your first script are usually default maintenance jobs which Nextcloud handles automatically.
for interest sake, do you think this could be a "timing" issue? for a similar setup, no where near the size of your external files, external files are being scanned, requiring good network speeds to external media and volumes... |
|
My external mounts are SFTP. They are not too slow because when I run file scan manually, they scan fine (about 30 minutes), and also Nextcloud snap has upgraded many times perfectly fine before with these mounted. So if the file scan taking too long is the issue -- this was not an issue in any prior nextcloud snap upgrade. So my hunch is either v28 is different somehow, or, it's coincidence and I just now went over the threshold in size where it breaks (for example: maybe I was 13gb database before and that was fine but now it's 14gb and isn't). If that's the case, I think perhaps the upgrade script is timing out? When I run it, it runs for a couple of minutes and then fails. It's about the same amount of time every time I tried. It's not tested but I wouldn't be surprised if it was for example a 5 minute limit? Also to clarify: in one sense I'm hesitant to call this a nextcloud snap issue because really databases shouldn't be that large and it's a nextcloud files_external issue... but, we should suppose, having a 14gb database is not impossible! Someone could have a lot of valid file caches, large teams, a tonne of nextcloud talk conversations etc. It could be possible make 14gb naturally without the files_external bug, so perhaps it should still be a solved issue? |
agree 100% |
|
As far as I know there isn't a timer in background. When But yes, I agree, there may be two "bugs". One upstream for |
|
That sounds like a good theory Hope we can all track down together Thank you again everyone I really enjoy the snap version of nextcloud. |
An update on the above from my original comment files_external has a DECADE old bug https://help.nextcloud.com/t/still-issue-cant-get-app-storage-app-files-external-user-not-logged-in/170160/16 I'm wondering if its that which is causing the error screens. Above is absolutely not snap related or even this issue related - but incase anyone found my original comment quoted above, here is the answer for that too. I added some new SFTP shares and removed some old ones around time these errors started. |
|
@LMRW, yeah, been watching this too. There seems to be no fix in the pipeline yet. Hopefully this is fixed in 29. My SFTP shares seem to be working okay, but there have been warnings in the logs. not at the moment though, seems to be stable. just to compare, are your SFTP connections local or external? mine are mostly local. |
|
You can continue the discussion, but this can be closed, as it is already solved. Thanks to everyone involved! |
Didn't realise it was to be EOLed quite that soon. Thanks for the heads up @scubamuc! |
|
@LMRW, so here's how I got my logs cleaned up after logs were being spammed by "user not logged in". since local SFTP connects per SSH credentials to the local server (NAS in my case), I deleted the global credentials which had my Nextcloud user credentials saved. After entering SSH/SFTP user credentials there, log spamming stopped for me. I'm guessing that if the global credentials are empty, then the credentials per SFTP connection will be used and logs spamming will stop. |
|
That's very interesting. I used to use credentials per connection (local sftp). I very recently swapped to using global. Can I ask how you deleted the global ones please? |
|
@pachulo agreed this thread now has moved in a different direction but I do think an issue remains... very large databases cannot be upgraded. The question is
I would counter if anyone has legitimate large data they won't be able to upgrade nextcloud snap atm. But maybe it's not a realistic real world issue? |
get settings: delete setting:
|
Amazing thank you I set them to both empty and pressed save in UI yesterday before I knew this command Checking today I see So looks like I already removed them Thank you very kindly for your tip here to do it via CLI and also tracking down the cause. This thread has a wealth of information now. Maybe I will when I have time make new issues on files_external to share with their team Thank you |
|
@steinger see here nextcloud/news#2585 (comment) |
|
@scubamuc yes, I saw it and updated it. |
My cron also crashed a few times -- I also use the new news app. I upgraded to v29 today. I'm also unsure why. But I think we need a new issue. |
|
Please check/create issues at the desired app then. |
|
@steinger see here: #2793 (comment) there is a new version of |
Hi There
Nextcloud or one of the apps require upgrade - only a limited number of commands are availablenextcloud snap Repair error: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away&Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] Connection refusedLuckily I run within a VM. So I have restored a snapshot and locked to channel 27 for now.
System:
Ubuntu LTS 22.04 Arm64
Note:
Prior to this for last few days I was occasionally seeing "System Internal Error" Nextcloud branded error screens when changing page/url. A refresh would always resolve the issue and work second time. I never saw these before this week and they have been a couple of times a day now. Unsure how related. My snaps auto refresh daily.
The text was updated successfully, but these errors were encountered: