Post upgrade-to-3.5.1 strangeness - some pre-upgrade content missing/abridged #18303
Replies: 1 comment 9 replies
-
|
It seems like something could have gone wrong when migrating from PostgreSQL 9.6 to PostgreSQL 14… I can't explain the missing posts and empty notifications otherwise…
I don't understand what could have caused this, but it's related to Elasticsearch.
This is the instance trying to connect to Elasticsearch while it is not running.
Yeah, something seems to have gone wrong when migrating postgres versions. Was there any error? It's possible that the dump was for some reason incomplete… From the symptoms you are describing, it does not sounds like the issues can be attributed to missing indices, but rather missing data… |
Beta Was this translation helpful? Give feedback.
-
|
Update - I have got my local version of the Mastodon site working, and I can confirm that the historical posts have been retained if I restore and migrate the data from the ''unvacuumed'' database dump. Now I need to experiment with importing the subsequent data (post-upgrade) to see if I can get a harmonious combination of the two. 🤞 |
Beta Was this translation helpful? Give feedback.
-
|
A further update - I've now managed to try both suggested ways of merging the pre- and post-upgrade database dumps. I've tried importing the post-upgrade database dump (missing the historical posts) over the migrated (pg 9.6-14) pre-upgrade database. It was imported without error, but it seemed to entirely overwrite the pre-upgrade data, and looked identical to the current live (post-upgrade) site. I also tried installing the pre-upgrade dump over the top of the post-upgrade databased... and it had lots of errors related to problems with db constraints and existing relations. The end result again seemed to mirror the post-upgrade live site. The only difference in both cases was that the "home" timeline was empty, only showing "Your home timeline is empty! Follow more people to fill it up. See some suggestions"... So the question is this: is there any way to export the posts from the pre-upgrade (but db-migrated 9.6->14) site to the post upgrade site, e.g. via the per-user post export? I found that selecting the 'export' archive on my test site whirred away for quite while saying it was 'compiling your archive..' my export archive... but then I never saw the option to download the archive (I don't think it ever finished - just trying it again now to see for sure). If I was to get one and archive of my 'posts and uploaded media', is there a way to import the the archive into the live site and use it to recover the past posts? |
Beta Was this translation helpful? Give feedback.
-
|
Ah - it's likely that the data export is failing because I haven't transferred the media files to my local machine... I'll transfer them now (will take a while)... |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I'll try to reply to your questions:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your response. Darn. That's a bit frustrating. I'll see if I can identify errors related to the archive generation. At this stage, my only option seems to be creating a parallel public instance of Mastodon (on a different URL) holding the old data, so that it's available for those users affected. They'll then be able to log in via their old details and (if we can get it to work) compile an archive of their historical data. That'd give them an option if/when Mastodon supports importing them... Is there currently any plan for that? |
Beta Was this translation helpful? Give feedback.
-
Just did an upgrade from 3.3.0 to 3.5.1 (including PostgreSQL 9.6->14 upgrade) on a Docker Compose deployed Mastodon (using pre-built containers). It was running Elasticsearch, and when I restarted it the first time after the upgrade, it all seemed to run ok, but I realised that Elasticsearch was still on version 6.7 (missed applying the 7.10.3 update to the docker-compose.yml image for es) wasn't there - got an error in the Admin saying Mastodon 3.5.1 needed Elasticsearch 7.x...
At the same time, I noticed that pre-upgrade notifications were strangely abridged - they showed in the notifications timeline, but only the name of the related user and the action (e.g. boosted, favourited, followed, etc.). None of the relevant toot contents appears as would normally be shown. Also, no older (pre-upgrade) local toots were shown in the local server stream. New local toots, all mentions, and all other notifications seemed to work as expected...
Noting this, I upgraded es to 7.10.3, pulled it, and removed & restarted the containers. The same notification, mention, and local toot behaviour has persisted. I decided to turn off Elatsicsearch (I don't run it on a separate Mastodon I run which I also recently upgraded from 3.3.0 -> 3.5.1 by the same method, and it does not exhibit this set of toot problems).
I note occasional warnings in the sidekiq logs, including
NameError: uninitialized constant AccountsIndex::AccountandWARN: Faraday::ConnectionFailed: getaddrinfo: Temporary failure in name resolution (es:9200)where I presume es:9200 is a residual reference to the Elasticsearch instance... this was also happening before I disabled es.Also, a few users (I've verified this) have found that Tusky crashes when looking at notifications as soon as the user reaches the end of any post-upgrade notifications (i.e. as soon as it hits/retrieves any pre-upgrade notifications that don't contain the toot content)...
Anyone have any suggestions on what the problem might be? Thanks in advance for any advice.
Additional info: it appears that some data might have been lost in the data migration process. I did a db vacuum (I have both a pre-and-post vacuum db dump) in pg 9.6 before importing the data into pg 14. I then applied the db:migrations which seemed to complete without issue. I note now, though, that my db directory in the container went from 12GB in pg 9.6 to 4.5GB in pg 14, so looks like actual data was lost. Could it merely be indices that could be recreated? I'm not sure if the data reduction could be accounted for entirely due to lost indices...
I'm now looking at redoing the data migration (on a local test version of Mastodon) to see if I can find where the data was lost, and to see if there's a way I can somehow safely update the existing live database (which seems to be working fine, but is not showing all data pre-upgrade) to include any data that was lost.
Beta Was this translation helpful? Give feedback.
All reactions