fetch UIDs and DATEs of all messages and do the pagination client side

[ChristophWurst]

Inspired by @DeepDiver1975's PR #747 I debugged the message fetching code and found out none of the IMAP servers I use have SORT capabilities. This means horde will do the sort client-side. While that's alright, looking at horde_imap.log I observed that way too much information is queried for sorting. It appears the full envelope data is fetched for sorting by date, which is useless and performance heavy.

This PR is a experiment where the sorting is done client-side by fetching message UIDs and DATEs, slicing the result and fetching the full info of that slice only.

Benchmark

I compared fetching the unified inbox with three accounts (two on my hoster, one is gmail).

Current master: ~2050ms
This branch: ~1550ms

Considering that ~500ms are needed for an OC request without querying the a IMAP server (tested with PageController::index), the speedup on my setup is about 32%. It will differ on other systems, depending on the server configuration and the mailbox size.

If a server supports SORT, this strategy could be slower than what horde is doing, but I don't have any server available to test this with. It is slower, see comment below

[mention-bot]

By analyzing the blame information on this pull request, we identified @DeepDiver1975, @thomasi and @zinks- to be potential reviewers

[ChristophWurst]

@owncloud/mail let me know what you think. Would be cool if some of you do the comparison too to see how that strategy behaves on different systems ;-)

[ChristophWurst]

Another benchmark: Created a folder and put 5000 messages into it on an IMAP server without SORT capability:
master: ~6350ms
this PR's branch: ~1650ms

[ChristophWurst]

Here's a simple script to create a bunch of messages https://github.com/ChristophWurst/horde-playground/blob/master/imap_spammer.php

[ChristophWurst]

Another benchmark using the IMAP test docker that support SORT, also with 5000 test messages:
master: ~500ms 😮
this PR's branch: ~1400ms

[ChristophWurst]

SORT capability can be checked with

$supported = $client->capability->query('SORT');

We could use that to select the most qualified algorithm

[kklem0]

I noticed the same thing on my Gmail account awhile ago, +1 for this.
When the server supports SORT we should always use it directly IMO.

[ChristophWurst]

@Gomez @jancborchardt this is ready for some testing!

[jancborchardt]

👍 still works on my SORT servers. :)