Large addressbook makes browser slow/unresponsive #428

djermann · 2017-11-03T13:34:32Z

A customer has a large address book with more than 13'000 contacts. When opening the contacts app it takes more than 3 minutes to load the contacts and chromium shows the page-is-unresponsive message several times. When the contacts are finally loaded, every further action takes at least a half a minute to be performed. The whole tab the uses more than 700 MB of RAM. There should be a better memory handling as the actual setting makes the contacts app nearly unusable with many contacts.

Steps to reproduce

Have more than 13'000 contacts in one address book.
Open contacts app.

Expected behaviour

Contacts get loaded fast. Only the first contacts should load and as you scroll down, the rest should get downloaded.

Actual behaviour

Contacts load very slow, and browser nearly crashes.

Server configuration

Operating system:
Debian 8.9
Web server:
Nginx
Database:
Mysql
PHP version:
5.6
Nextcloud version: (see Nextcloud admin page)
12.0.2
Contacts version: (see Nextcloud apps page)
1.5.3
Updated from an older Nextcloud or fresh install:
Fresh install
Signing status:
No errors have been found.
List of activated apps:

Enabled:
  - activity: 2.5.2
  - comments: 1.2.0
  - contacts: 1.5.3
  - dav: 1.3.0
  - federatedfilesharing: 1.2.0
  - federation: 1.2.0
  - files: 1.7.2
  - files_external: 1.3.0
  - files_pdfviewer: 1.1.1
  - files_sharing: 1.4.0
  - files_texteditor: 2.4.1
  - files_trashbin: 1.2.0
  - files_versions: 1.5.0
  - files_videoplayer: 1.1.0
  - firstrunwizard: 2.1
  - gallery: 17.0.0
  - logreader: 2.0.0
  - lookup_server_connector: 1.0.0
  - nextcloud_announcements: 1.1
  - notifications: 2.0.0
  - oauth2: 1.0.5
  - password_policy: 1.2.2
  - provisioning_api: 1.2.0
  - serverinfo: 1.2.0
  - sharebymail: 1.2.0
  - survey_client: 1.0.0
  - systemtags: 1.2.0
  - theming: 1.3.0
  - twofactor_backupcodes: 1.1.1
  - updatenotification: 1.2.0
  - user_external: 0.4
  - workflowengine: 1.2.0
Disabled:
  - admin_audit
  - encryption
  - user_ldap

Nextcloud configuration:

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbtype": "mysql",
        "version": "12.0.2.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "localhost",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "ocsfx00acnxt",
        "loglevel": 0
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
no
Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

Browser:

Chromium Version 57.0.2987.98

Operating system:

Debian 8.9

CardDAV-clients:

Logs

Web server error log

no errors

Nextcloud log (data/nextcloud.log)

nothing interesting

Browser log

no messages in console

The text was updated successfully, but these errors were encountered:

irgendwie · 2017-11-03T14:33:34Z

@djermann The contacts app by now handles large amounts of contacts in a bad way, there is no question about it. The problem for me is, that I have no real idea on how to improve this situation. Maybe you can share the use case of such a high number of contacts and how we should improve the handling? E.g.

How to choose which contacts to initially load?
Completely filter out specific contacts?
What information is valuable of each contact?
etc.

djermann · 2017-11-03T16:05:40Z

I can think of different possible approaches to solve this.

When a user opens the contacts app, it only shows the first 10-20 contacts in alphabetical order. If there was a way to detect which contacts are visible in the webinterface, then the contacts app would only download these plus maybe 100 more. Later the user maybe decides to scroll down, so the already downloaded 100 more contacts would be displayed, while in the mean time it already downloads another 100 contacts.
When opening the contacts app, it could just download the first say 100 contacts and when the user scrolls down to the 100. contact, a button appears that says something like 'load more contacts'. Then it would download another 100 contacts.
One could also split the contacts in pages of 100.
One could split the contacts into pages of the first letter. So all contacts whose last name starts with an A would appear on the page named 'A'. Although, this behavior would only make sense when having more than a certain amount of contacts (e.g. 200).

The same behavior should also apply for the results of a search.

j-ed · 2017-11-08T18:01:46Z

Let me point your attention to the CardDAV plugin for Roundcube (https://github.com/blind-coder/rcmcarddav), which is a CardDAV client, programmed in PHP, which works perfectly well with my Nextcloud server. I must admit, that I only have 600 contacts in my address book, but it might give you an idea how it could look like.
Right after a CardDAV address book has been set-up an initial address synchronization takes place, which will not be necessary for Nexctloud because you can directly work on the database. The address book is presented in a three column view, the first column shows the address book and the groups, the second column shows the contacts and the third column shows the address details if an address is selected. So it more or less uses the same layout as the contacts app in Nextcloud. This is how it looks like:

It seems that the contact groups, the first and last names, and the email address, are loaded at once (or maybe in chunks) and the address details are only loaded when a contact is selected. (Loading ... is displayed for a fraction of a second). This reduces the amount of data which need to be kept in memory and makes usage really fast.

skjnldsv · 2017-11-08T18:05:13Z

@j-ed aside from the chunk loading, we're actually loading our info like that :)

j-ed · 2017-11-08T18:50:33Z

@skjnldsv Ok, that's new to me. I wonder where the huge performance differences are coming from if I compare both applications 😕

CamZie · 2017-11-10T11:06:30Z

We have also encountered a problem after upgrading the contacts app to 2.0.1. On the Firefox browser version 52 on Linux, large contacts (in our case 398 and 4908) are not showing and only the spinning wheel appears. I had a look on the log files, as well on the browser console but I did not find anything related to this.

I tested this on different browsers and it worked for the browsers below. I have 4908 contacts in my Nextcloud installation but it still takes more than 5 minutes to load on different browsers, which makes the app not usable for me or any users who has many contacts.

Chrome (version 57.0.2987.98 on Linux)
Vivaldi (version 1.11.917.43 on Linux)
Firefox (version 42 and version 56 on Windows)
Internet Explorer (version 11 on Windows)

skjnldsv · 2018-02-28T15:42:37Z

Hello everyone! Is this still a big issue? Or are our latest optimisation good enough? :)

skjnldsv · 2018-09-28T18:12:01Z

Latest 3.0.0 should be fast enough :)

fwolfst · 2019-03-26T11:25:52Z

The app/contacts page still puts "too much" stress on my browser (FF 66), which warns me about slow script and asks me whether I'd prefer to shut it up. That is with NC 15.0.5 and Contacts 3.0.5, ~ 8k Contacts in a adress book shared with me. It is certainly not technically unusable and synchronisation with Thunderbird/Cardbook is pretty fast; but the Web-Ui takes a minute to load and is laggy afterwards, with a warning about a slow script more or less every time interacting with it.

skjnldsv · 2019-03-26T13:18:46Z

but the Web-Ui takes a minute to load and is laggy afterwards, with a warning about a slow script more or less every time interacting with it.

Which make sense. Browsers cannot handle that many things at once.
We did some pretty intense optimisations, but it still is what it is: a javascript application.
We will never be as fast and reactive as a desktop app as cardbook or something else and we'll never be :(

Though 8k users seems pretty crazy. I'm wondering what is the bottleneck here, the loading request? The display of all of those users? 🤔

fwolfst · 2019-03-26T13:36:12Z

We will never be as fast and reactive as a desktop app as cardbook or something else and we'll never be :(

Well - yes and no. One could put the load on the server, like in "traditional" web-applications and only render what is needed, paginate etc. But I assume that the architecture wants to have most of the accessible data already at client side to do the work there (in my oppinion that is the "crazy" part, not the amout of contacts). By the way, you wrote "Users", but its really about contacts. Although there is ~100 at least potential Users involved too.

And I don't want to blame. Thanks for the optimiziations! I guess the the choice of using a heavy (?) JavaScript-Client-Framework also has had good consequences.

skjnldsv · 2019-03-26T13:41:56Z

By the way, you wrote "Users", but its really about contacts. Although there is ~100 at least potential Users involved too.

I indeed wanted to write contacts :)

And I don't want to blame. Thanks for the optimiziations! I guess the the choice of using a heavy (?) JavaScript-Client-Framework also has had good consequences.

Don't worry, I was not taking it like that :)
It's not really a choice here. Nextcloud is just a php environment. You're talking about server-side rendering, but that would require a whole rework of the architecture we're working on. There are some options our there, yes, but it would complicate things far more than it would help us being able to properly maintain such applications 😉

I will try to check if there are some other optimisations to be done here :)

fwolfst · 2019-03-26T14:06:53Z

I'm wondering what is the bottleneck here, the loading request? The display of all of those users? thinking

Just from looking at the "Network" tab of Dev Tools in Firefox: Loading the 5MB data of the larger adressbook takes a while, but then comes a long period of no network activity until a PROPFIND to a specific .vcf path is done (I guess that is the currently selected/first contact).

skjnldsv · 2019-03-26T15:10:47Z

but then comes a long period of no network activity until a PROPFIND to a specific .vcf path is done (I guess that is the currently selected/first contact).

how long (roughly)? :)

fwolfst · 2019-03-26T15:27:32Z

A minute. My machine is not top notch, but imho sufficiently powered (5gb ram, 4x 2.6ghz core).

…

On Tue, Mar 26, 2019 at 4:10 PM John Molakvoæ ***@***.***> wrote: but then comes a long period of no network activity until a PROPFIND to a specific .vcf path is done (I guess that is the currently selected/first contact). how long (roughly)? :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#428 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANY7WoYmndkx9c17JpD3KP7mgpxslu8ks5vajh5gaJpZM4QRF9p> .

skjnldsv · 2019-03-26T16:34:10Z

@fwolfst please check out #1018
It should improve perfs a lot 🤞

fwolfst · 2019-04-09T08:47:21Z

Sorry that I could not contribute more. I updated to 3.1.0 and performance is way better now. Thanks a lot!

skjnldsv · 2019-04-09T09:06:41Z

@fwolfst do not be sorry!!
Thanks for helping me! :)

djermann changed the title ~~Large Addressbook makes browser slow/unresponsive~~ Large addressbook makes browser slow/unresponsive Nov 3, 2017

irgendwie added the needs info Not enough information provided label Nov 3, 2017

skjnldsv closed this as completed Sep 28, 2018

skjnldsv added this to the 3.0.0 milestone Sep 28, 2018

ChristophWurst mentioned this issue Aug 22, 2022

Unable to load large address books due to amount of data transfer #2896

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Large addressbook makes browser slow/unresponsive #428

Large addressbook makes browser slow/unresponsive #428

djermann commented Nov 3, 2017 •

edited by jospoortvliet

irgendwie commented Nov 3, 2017

djermann commented Nov 3, 2017

j-ed commented Nov 8, 2017

skjnldsv commented Nov 8, 2017

j-ed commented Nov 8, 2017

CamZie commented Nov 10, 2017

skjnldsv commented Feb 28, 2018

skjnldsv commented Sep 28, 2018

fwolfst commented Mar 26, 2019

skjnldsv commented Mar 26, 2019

fwolfst commented Mar 26, 2019 •

edited

skjnldsv commented Mar 26, 2019

fwolfst commented Mar 26, 2019

skjnldsv commented Mar 26, 2019

fwolfst commented Mar 26, 2019 via email

skjnldsv commented Mar 26, 2019

fwolfst commented Apr 9, 2019

skjnldsv commented Apr 9, 2019

Large addressbook makes browser slow/unresponsive #428

Large addressbook makes browser slow/unresponsive #428

Comments

djermann commented Nov 3, 2017 • edited by jospoortvliet

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

Browser log

irgendwie commented Nov 3, 2017

djermann commented Nov 3, 2017

j-ed commented Nov 8, 2017

skjnldsv commented Nov 8, 2017

j-ed commented Nov 8, 2017

CamZie commented Nov 10, 2017

skjnldsv commented Feb 28, 2018

skjnldsv commented Sep 28, 2018

fwolfst commented Mar 26, 2019

skjnldsv commented Mar 26, 2019

fwolfst commented Mar 26, 2019 • edited

skjnldsv commented Mar 26, 2019

fwolfst commented Mar 26, 2019

skjnldsv commented Mar 26, 2019

fwolfst commented Mar 26, 2019 via email

skjnldsv commented Mar 26, 2019

fwolfst commented Apr 9, 2019

skjnldsv commented Apr 9, 2019

djermann commented Nov 3, 2017 •

edited by jospoortvliet

fwolfst commented Mar 26, 2019 •

edited