Instance breaks while updating from 4.1.0 to 4.1.1 #322
Comments
|
I tried restoring from a backup, but the server is still in the same state even after the backup. Probably going to require some manual digging, would certainly appreciate any ideas on this one. |
|
Here's the result from journalctl
|
|
Update: I tried deleting the row that as complaining about a "duplicate" entry, but still no luck. I tried upgrading again, and it just fell down in a different spot at the database. Here's the new log - https://paste.yunohost.org/raw/fokenatome This is where it looks like things start falling apart...
I'll post this over on the PeerTube github as well, since I'm not sure if the root-cause of the problem lies with PeerTube, yunohost, or somewhere in-between. |
|
I deleted a few more db records that the log was complaining about, but this has never been an issue before (I last ran an update a few weeks ago, and certainly had the "problem" tags on the last upgrade). I really think there might be some issue with the upgrade script for yunohost, and am wondering if the root-cause is somewhere in this error message:
It really seems like the script is not being patient enough while something is happening in the background. The above error message always happens after this point:
I just ran another update, it's complaining about the database again. I'm going to try removing the offending entry (it's a tag this time) but I really doubt this is the root-cause, since a) I've updated many times in the past with some of these tags present and b) no matter how many database entries I remove, it keeps complaining while waiting to do something. Will x-post this to PeerTube Github as well, just in case... |
|
Don't know if @yalh76 or anyone else might know if there might be some issue with the new upgrade script? I literally didn't change anything between my update to 4.1.0 and this attempt to upgrade to 4.1.1. Only thing I did was try to push the upgrade (and possibly whatever system updates yunohost said were available). |
|
You seem to have an issue while installing the That's weird, I take a look. |
|
OK, you seem to have installed a plugin named |
|
Yup, here is the output @fflorent
|
|
That's it! There has been a recent change in the This line in what you pasted is interesting:
The error says: So I guess updating this plugin should do the trick? Is your instance running at this moment or is it broken? In the first case, you should be able to update the plugin through the web admin interface (https://tilvids.com/admin/plugins/list-installed?pluginType=1) |
|
Nope, sadly this has taken down the entire site for the last three days. :( Is there a way to remove it from CLI? That plugin was added over a year ago because a user wanted to be able to watch PeerTube via Chromecast. It definitely does not need to be installed, especially if it's the cause of the site being down. :) |
|
I guess you can attempt to update the plugin through npm (if cd /var/www/peertube
sudo -u peertube PATH="/opt/node_n/n/versions/node/16/bin:$PATH" NODE_CONFIG_DIR="/var/www/peertube/config" NODE_ENV=production npm run plugin:install -- --npm-name peertube-plugin-tv-streamingIf there is an error, please share the logs. Otherwise, attempt a restart: systemctl restart peertubeOr a backup restoration. |
|
I'll try updating it, but is there a way to just remove it from the command line? We definitely don't need it. |
|
Quite the same command, except you invoke cd /var/www/peertube
sudo -u peertube PATH="/opt/node_n/n/versions/node/16/bin:$PATH" NODE_CONFIG_DIR="/var/www/peertube/config" NODE_ENV=production npm run plugin:uninstall -- --npm-name peertube-plugin-tv-streaming |
|
Tried and failed to uninstall:
Looks like it can't find that plugin maybe? |
No, you have an issue with yarn not being installed. I guess the peertube installation is in an unknown state. Before going further, do you have a backup of your PeerTube instance with all its videos in it? |
|
I believe I have a snapshot that would work, yes, though obviously would love to avoid putting that to the test... |
|
Alright. That should be somewhat OK. Optional: If you have enough disk space for duplicating your videos, you can attempt a full backup: If you attempted a backup restoration using the Then try this: cd /home/yunohost.app/peertube/storage/plugins
sudo -u peertube PATH="/opt/node_n/n/versions/node/16/bin:$PATH" NODE_CONFIG_DIR="/var/www/peertube/config" NODE_ENV=production npm uninstall --save peertube-plugin-tv-streamingAnd try restoring the backup ( EDIT: and if after that your instance is running, go to https://tilvids.com/admin/plugins/list-installed?pluginType=1 and uninstall the plugin. You may have an error but the uninstallation should still be effective after a manual page reload. PS: You may be interested in borg for your backups, it's quite powerful. You have a client and a server for Yunohost |
|
Here's the result of that:
When I tell it to restore the backup, it outputs:
Should I uninstall the peertube app via Yunohost first? |
|
Have you made the optional backup? |
|
Yeah, I have a downloaded local copy of that peertube-pre-upgrade1 |
|
My question was for this:
Because I fear of removing the data directory in Still, that must not be required if I am correct, the removal by default keeps the data directory: https://github.com/YunoHost-Apps/peertube_ynh/blob/master/scripts/remove#L89-L94 If you are OK with that assumption, you can attempt this: sudo yunohost app remove peertube
sudo yunohost backup restore peertube-pre-upgrade1 |
|
Unfortunately don't have space to make a full backup on the server like that. I have removed the app before though (two days ago, trying to fix this) and it appears that removing the PeerTube app via Yunohost does not touch the video data. I will give it a shot, because at this point, I'm not sure what else to try... |
|
Tried restoring, there were some errors, but looks like it restored all the data. However, site was still 502'ing. I tried a restart, no luck. I'm going to try running the update again, I'll post the (inevitable) error log when it's done. |
|
What logs do you have this far? |
|
Waiting on the upgrade attempt right now, I'll post what I see from that when it's done. Let me know if there are any other logs you think would help. |
|
Upgrade failed. Here's some of the output:
Let me know what else would be useful to diagnose. |
|
Don't you have more information above? (especially about the index creation) |
|
Well wait, so now this is interesting. It looks like the upgrade succeeded...
But there were still a bunch of errors with database entries, but looks like it at least went through this time. But then when I go to the site (after a restart of the server), I'm still getting a 502 error. So...maybe it's a different thing we have to figure out now? |
|
Can you please share error logs? You can use the Yunohost UI: https://<YUNOHOST.tld>/yunohost/admin/#/services/peertube Or using |
|
Output of journalctl -u peertube:
|
|
OK Get your database password: yunohost app setting peertube db_pwdRun this command and when prompted enter the password you got above: And run this SQL query in order to see the names in the tag table and the duplicates ( select name, count(name) from tag group by name order by count(name); |
|
Hmm, I'm not a SQL expert at all, but when I run the command
I get a HUGE list with lots of counts that say "1". Should I be doing something else to see the duplicates? |
|
OK. Try running this query instead: select name, count(name) from tag group by name having count(name) > 1; |
|
There look to be 158 items that have a count > 1 (all are 2). First one listed is "freesoftware" (which is what the update listed as failing on). Let me know if you need more. |
|
I guess you indeed have duplicated entries in your Try this: delete from tag t where t.id NOT IN (select MIN(id) from tag group by name); |
|
Ok, it came back with having deleted "160", which appears to be what we had above (plus two more?). Next? Restart the server? |
|
Sure, try a restart.
28 mars 2022 23:02:55 TILvids ***@***.***>:
… Ok, it came back with having deleted "160", which appears to be what we had above (plus two more?). Next? Restart the server?
—
Reply to this email directly, view it on GitHub[#322 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AAC2X6MJEXRQHX3PUOMAGUDVCIM7ZANCNFSM5RXRYJXA].
You are receiving this because you were mentioned.[Image de pistage][https://github.com/notifications/beacon/AAC2X6MPK4ULP52PD2N665TVCIM7ZA5CNFSM5RXRYJXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOIBYOIFQ.gif]
|
|
@fflorent Restart and...looks like it's working! I have no idea what the root-cause of this was that caused everything; only two factors I can see that fixed it were:
Might be worth digging into for any of the yunohost PeerTube folks, and happy to give any info if it's useful. Super appreciate the effort @fflorent hit me up with a DM on one of our social channels to let me know how I can buy you a beverage of your choice! :) |
After running the update from 4.1.0 to 4.1.1 the instance breaks. This is the log file that was generated.
https://paste.yunohost.org/raw/lecorivudu
Let me know what else I can get to dig into it. My best guess so far is that while trying to run through the database during the update, the job wasn't completed and the installer timed-out, leaving it in a broken state. No idea where to start on fixing it. Maybe @yalh76 has thoughts?
The text was updated successfully, but these errors were encountered: