to slow, takes to much cpu

[akidburn]

version and config

Everything current (Firefox, floccus, nextcloud, chrome)
Nextcloud 13 installed on odroid c2 ( arm 1.5bghz quad-core, 2 gb ram) using nginx and php7-fpm
Nextcloud bruteforce protection disabled (had some 1500 entries there before I noticed)

About 3700 bookmarks previously synced by xmarks across 3 machines with Firefox and chrome each, so 6 clients.

current situation

First sync to upload bookmarks is relatively fast, If the tables are empty. Next download is not too bad on the server, but if you have a few hundred tabs open and a few extensions, you get a rate of one bookmarks per second added to Firefox. (Waaaay to slow)

But the real problem is, if you want to update the collection by adding new bookmarks. For one, each machine has to be fully synced from nextcloud before you start adding new bookmarks or you can delete everything again. But the actual problem with nextcloud on small raspberry-like hosts is the missing CPU power compared to desktop PC's. If I try to add a single bookmark to one client and sync it to server, I can't sync anything for at least 2 hours. There are at least 10 threads of php7-fpm running comparing the bookmarks, and that's just bad.

[marcelklehr]

Hey @akidburn

Thank you for reporting this. So far I'm aware of performance problems with Firefox (which is due to implementation details in the browser), but not with Chrome.

version and config: Everything current

Generally, this doesn't help much, I'm afraid, as I don't know what your perception of 'current' is and if it is even correct. I need hard facts :)

For one, each machine has to be fully synced from nextcloud before you start adding new bookmarks or you can delete everything again.

Could you elaborate what you mean by this? It should be possible to add bookmarks in a stale local state and even while syncing.

But the actual problem with nextcloud on small raspberry-like hosts is the missing CPU power compared to desktop PC's. If I try to add a single bookmark to one client and sync it to server, I can't sync anything for at least 2 hours. There are at least 10 threads of php7-fpm running comparing the bookmarks, and that's just bad.

Do you sync any data on that machine other than bookmarks, with a comparable amount of data? I would be surprised if you saw better performance there. I'm afraid, this will hardly be improvable with the nextcloud bookmarks app as a backend, work is being done to develop a WebDAV backend, which may be more performant on low-grade server hardware.

Still, it's surprising to me that a single bookmark addition causes so much load on your server, as the server doesn't do any kind of comparison. It simply handles the CRUD operations necessary for the sync to work.

[akidburn]

Hi @marcelklehr

Thanks for the reply. In general, about Chrome, maybe I am biased compared to Xmarks, so this is mainly about Firefox and the Serverload.

Versions

Clients

Windos 10 Pro Version 1709
Google Chrome 66.0.3359.139 (mostly fine, speedwise)
Mozilla Firefox 59.0.3 (64-Bit)
Floccus 2.1.0 (both FF and Chrome)

Server

odroid-c2 3.14.79-117
Amlogic ARM® Cortex®-A53(ARMv8) 1.5Ghz quad core CPU
2Gbyte DDR3 SDRAM
Ubuntu 16.04
Nextcloud 13.0.2
Bookmarks App 0.11.0
nginx/1.13.12
php Version: 7.2.4
mariadb Version: 10.0.34

Speed Issue

As it is, i watched htop on the odroid-c2 during these issues.
there were up to 20 php-fpm processes open, all working, and the mariadb was using only 20 percent of a single core. i dont know what the php processes were doing, but there sure as hell was more going on than just CRUD into the db. also, whenever there was a sync error, meaning bookmark already existed, the cpu was dropping to zero immediately. could it be that the bookmark comparisons are done by php and not by sql?

akidburn

[marcelklehr]

As I maintain the server-side bookmarks app as well, I'm fairly sure about what's going on. Let me give you a rough overview:

Floccus synchronization has 4 phases:

1. Create new bookmarks on the server (this is the only part where the server tries to match incoming URLs with existing ones, so as not to create duplicates). When syncing for the first time, this will take the longest, even though floccus sends multiple requests at once.
2. Delete removed bookmarks from the server.
3. Get all bookmarks from the server and compare them to the local ones, deciding whether to
   a) update the matching local one
   b) update the server-side one
   c) create a new local bookmark in case there is none that matches
4. Delete local bookmarks that no longer exist on the server.

Floccus makes a maximum of 10 requests in parallel to mitigate network latency. If the server is not that fast this may put it under stress, but floccus will wait for the running requests to finish before starting new ones. Downloading all bookmarks is a problem, but can't be avoided currently, as the bookmarks app doesn't have folders that could be used to make a tree comparison, which might terminate early.

[akidburn]

just tried to sync bookmarks into a chrome directory. no server load, though chrome nearly fries my old notebook with constant 100 % load, and now...

Cannot read property 'hash' of undefined Cannot read property 'hash' of undefined Cannot read property 'hash' of undefined Cannot read property 'hash' of undefined Cannot read property 'hash' of undefined Cannot read property 'hash' of undefined Cannot read property 'hash' of undefined Cannot read property 'hash' of undefined Cannot read property 'hash' of undefined Cannot read property 'hash' of undefined

could the reason be if the bookmarks have no name, only an url?

EDIT:

another error:

Removing a bookmark on the server failed. remoteId=11451 Removing a bookmark on the server failed. remoteId=11057 Removing a bookmark on the server failed. remoteId=11056 Removing a bookmark on the server failed. remoteId=11043 Removing a bookmark on the server failed. remoteId=11054 Removing a bookmark on the server failed. remoteId=11053 Removing a bookmark on the server failed. remoteId=11450 Removing a bookmark on the server failed. remoteId=11045 Removing a bookmark on the server failed. remoteId=11058 Removing a bookmark on the server failed. remoteId=11055

how to reproduce

1. sync a folder with subfolders between two chrome clients
2. delete all bookmarks from the synced folder on one client and wait until the bookmarks are gone on nc
3. check client 2, if necessary click sync and watch errors

[marcelklehr]

"Removing a bookmark on the server failed." Is an upstream bug with nextcloud bookmarks, which will be resolved with the next release.

"Cannot read property 'hash' of undefined" is a new one. Does it persist?

[akidburn]

No it doesnt persist. May be a chrome issue during high cpu loads.
Floccus 2.2.2 works now, though if any two machines are synchronizing at the same time, errors occur.

[marcelklehr]

Which kind of errors?

[akidburn]

Either the "Duplicate Bookmark" error or the "Removing a bookmark on the server failed" error, though in watching the oc_bookmark table i found, that just attempting to add a single new bookmark sometimes creates duplicates in the table, that dont produce errors until you force sync.

The Duplicate Error also leads directly or indirectly to the Removing error, if more then one client is in a failure state. As in, if you remove the duplicate client side, at least one client is going to display the Removing Error.

Though i found a little sql script that removes the duplicates from the tables, could be integrated in the bookmarks app as a cronjob if the limitation of no duplicates isnt going away.

DELETE a FROM oc_bookmarks a, oc_bookmarks b
WHERE a.id < b.id
AND a.url <=> b.url;

DELETE c FROM oc_bookmarks_tags c
LEFT JOIN oc_bookmarks f ON f.id = c.bookmark_id
WHERE f.id IS NULL;

[marcelklehr]

Try v2.2.6 which solves a bug which I could verify causes some if not all of these errors.

[marcelklehr]

All issues resolved?! 🎉

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.