Upgraded from 4.0.6 to 4.0.7 deleted all bookmarks

[ghost]

Describe the bug
My script upgraded the Nextcloud Bookmarks from 4.0.6 to 4.0.7 recently, and the bookmarks are completely gone.
Recent shows nothing, and the entries in the database are empty either.
#1305 again?

To Reproduce

1. Execute

$ occ app:update bookmarks

2. Open the browser and check the bookmarks.
3. Nothing but the folders remain.
4. Connect to the nextcloud database.

$ mysql -u user -p nextcloud

MariaDB [nextcloud]> select * from oc_bookmarks;
Empty set (0.002 sec)

Expected behavior
The bookmarks are not touched during the updates.

Screenshots

Desktop (please complete the following information):

• OS: openSUSE 15.2
• Browser: Firefox
• Version: 78.6.1esr (x64)

Server (please complete the following information):

• OS: postmarketOS / Alpine Linux Edge
• HTTP server: nginx 1.18.0
• Database: MariaDB 10.5.8
• PHP version: php v7.4.14
• Nextcloud version: 20.0.4
• Bookmarks app version: 4.0.6 -> 4.0.7 -> 4.0.8
• Activated Nextcloud Apps:

accessibility: 1.6.0
audioplayer: 3.0.0
bookmarks: 4.0.8
bruteforcesettings: 2.0.1
calendar: 2.1.3
checksum: 1.0.1
cloud_federation_api: 1.3.0
comments: 1.10.0
contacts: 3.4.3
contactsinteraction: 1.1.0
dashboard: 7.0.0
dav: 1.16.2
federatedfilesharing: 1.10.2
files: 1.15.0
files_external: 1.11.1
files_pdfviewer: 2.0.1
files_rightclick: 0.17.0
files_sharing: 1.12.1
files_versions: 1.13.0
files_videoplayer: 1.9.0
firstrunwizard: 2.9.0
logreader: 2.5.0
lookup_server_connector: 1.8.0
news: 15.1.1
nextcloud_announcements: 1.9.0
notes: 4.0.2
notifications: 2.8.0
oauth2: 1.8.0
password_policy: 1.10.1
passwords: 2020.12.2
photos: 1.2.1
privacy: 1.4.0
provisioning_api: 1.10.0
recommendations: 0.8.0
serverinfo: 1.10.0
settings: 1.2.0
survey_client: 1.8.0
systemtags: 1.10.0
tasks: 0.13.6
text: 3.1.0
theming: 1.11.0
twofactor_backupcodes: 1.9.0
updatenotification: 1.10.0
user_status: 1.0.1
viewer: 1.4.0
weather_status: 1.0.0
workflowengine: 2.2.0

• Nextcloud configuration:

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "20.0.4.0",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "default_language": "ja_JP",
        "default_locale": "ja_JP",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "maintenance": false,
        "theme": "",
        "loglevel": 2
    }
}

• Nextcloud external user backend: none

Additional context
Add any other context about the problem here.

Web server error log

(No error occurred.)

Nextcloud log (nextcloud/data/nextcloud.log)

(No Bookmarks entry found.)

Browser log

(I don't know how to get it. I will paste it in need.)

[jim-perkins]

yep mine are missing as well. the folders are there but no bookmarks. Just saw an update for bookmarks to 4.0.8 but that did not fix it. thanks. and nothing in the database table bookmarks ??

[sreutter]

same problem here

[Kourai]

same here ! :(

[j-ed]

I was able to restore my bookmarks by reimporting all entries from a valid database backup to the oc_bookmarks table. All categories were still available but the sorting into sub folders was missing. Although moving the bookmarks into the related sub folders caused several error messages being displayed in the gui, they were successfully moved to the folders. So at the end I was able to recover my data.

[sreutter]

so the urls are still there in the db? can you please give a more detailed instruction?

[Kourai]

I was able to restore some of my bookmarks thanks to firefox and Floccus.
On firefox, use "show all bookmarks" > Import and Bakcup > Restore.
This gives you back your bookmarks on firefox !
Then, go to Floccus setting for your(s) nextcloud-folder(s), and in "dangerous action", click on Trigger sync from scratch and start syncing.

Now all my firefox bookmarks are back (I updated to 4.0.8 in between), but I still lost some old bookmarks that weren't sync on firefox (or any other browser)

[sreutter]

hmm. I used bookmarks without sync. I have some old db backup, I'll try what I can do with it. It looks like in my db, there are all bookmarks gone.

[jim-perkins]

yea my bookmark entries in the database were empty also. I wasnt using an external sync. Nextcloud was my sync :-) .

Halfway glanced at nextcloud/updater-ocredactedxxx/backups/nextcloud-20.0.5.2-1611693483 but nothing saying database jumped out at me.

Luckily my instance had a zfs snapshot . Within the old snapshot I exported bookmarks from nextcloud. then on the current instance I deleted bookmarks ( because the folder structure was intact ) , and then imported bookmarks from within nextcloud. Never directly touched either database.

Also posted comments at: https://www.reddit.com/r/NextCloud/comments/l5w061/updating_bookmarks_andor_nextcloud_erased_all/

[j-ed]

so the urls are still there in the db? can you please give a more detailed instruction?

No, the table is indeed empty. You need to have a valid backup to re-import the bookmark entries.

[nursoda]

This issue reminds me of a similar one that struggles with character encoding corrections/regression (in the calendar app). Was anything in that direction changes in Bookmarks?

[ghost]

so the urls are still there in the db? can you please give a more detailed instruction?

@sreutter Take a look.

MariaDB [nextcloud]> select * from oc_bookmarks;
Empty set (0.001 sec)

[marcelklehr]

I'm so sorry guys. The update deleted all bookmarks. I'm pulling my hair out over this again. 😱 😡

v4.0.8 fixed the underlying bug but didn't recover your bookmarks.

If you're lucky and have a backup you can follow this guide: #1305 (comment)

[cbwoog]

I do have the same. It seems to me that the update to 4.0.8 deleted all bookmarks.
Only the empty folders are left.
New bookmarks can be created.
NC 20.0.6
truenas 12

[marcelklehr]

@cbwoog You likely updated to v4.0.7 first which had the bug.

[marcelklehr]

If I've learned anything from these incidents it's that I need collaborators but at least people that review code changes for mistakes, like it's done in the server repository.

[sreutter]

unluckily, I'm not a progammer, but I could help in testing the app before the release?

[teemue]

Because of this bug I tried removing and reinstalling bookmarks. However, installation gives this error:

Error: An exception occurred while executing 'SELECT p1.id FROM oc_bookmarks_shared_folders p1 LEFT JOIN oc_bookmarks_shared_folders p2 ON p1.folder_id = p2.folder_id AND p1.user_id = p2.user_id WHERE p2.id < p1.id':

SQLSTATE[42S02]: Base table or view not found: 1146 Table 'nextcloud.oc_bookmarks_shared_folders' doesn't exist`

What should I do ?

[marcelklehr]

@teemue When dropping the bookmarks tables you also need to delete all entries related to the bookmarks app from oc_migrations. This will cause all migrations to be rerun.

for the record: Re-installation is not necessary for fixing this. If you've been affected and want your bookmarks back, follow: https://github.com/nextcloud/bookmarks/wiki/Recovering-from-DB-Backup if you have other means of getting your bookmarks back do that. Update to v4.0.8 should fix the software issue that caused the removal. Automated integrity checks for Updates and Repair steps have landed and together with reviews from @stijink (:tada:) will make sure this hopefully doesn't happen again in the future.

[teemue]

@teemue When dropping the bookmarks tables you also need to delete all entries related to the bookmarks app from oc_migrations. This will cause all migrations to be rerun.

for the record: Re-installation is not necessary for fixing this. If you've been affected and want your bookmarks back, follow: #1305 (comment) if you have other means of getting your bookmarks back do that. Update to v4.0.8 should fix the software issue that caused the removal. Automated integrity checks for Updates and Repair steps have landed and together with reviews from @stijink (🎉) will make sure this hopefully doesn't happen again in the future.

Could you provide me with an information on how to delete those entries? Thanks in advance :)

[marcelklehr]

Oh, sure. Usually DELETE from oc_migrations WHERE app = 'bookmarks'; should work :)

[marcelklehr]

In addition to the above, I've just landed a failsafe feature in floccus that will cause it to refuse execution of a sync run if more than 50% of local bookmarks are to be deleted. This should be an additional safeguard against any possible misconfigurations or software bugs. Take care!

[fossxplorer]

I had a big impact instance with thousands of users who lost their bookmarks.
Was using version 3.3.3 and went 4.0.6 where the loss occured.
Trying to follow #1305 did not solve it for us. And recovering back to 3.3.3 with old data and then trying to upgrade to 4.1.0 also deletes it all. Question is whether i deleted the bookmark entries in oc_migration or not.
Now i have restored once again into 3.3.3 and i deleted oc_migration prior to that.
What is a safe step forward now?

[marcelklehr]

@fossxplorer I'm sorry for your troubles. I'm available at https://gitter.im/nextcloud-bookmarks/community for support