mu4e initial search slow...

[hakanserce]

Expected or desired behavior

Normally, searches in mu4e is almost instantaneous for me. I would like this speed in the initial search (or jump to folder) right after I start mu4e, or right after I update email.

Actual behavior

When I start mu4e and immediately try to jump to inbox (or do a search), this takes 7-10 seconds for me. Subsequent jumps/searches are instantaneous.

Same slowness happens when I update email. After the update finishes, header view shows "Searching" for 7-10 seconds. Then, again, subsequent jumps searches are instantaneous.

Steps to reproduce

Make sure you have email address autocomplete enabled ((setq mu4e-compose-complete-addresses t)).

Initial search: Kill all mu4e buffers. Open mu4e and immediately jump to inbox (or bookmark, or search). Search will take a long time. Try jumping/searching again after the initial one, it will be much faster.

Or similarly, update: Update email (C-S-U or whatever shortcut you have), watch header view showing "Searching" for a long time after showing the message list.

Versions of mu, mu4e/emacs, operating system etc.

mu version: 1.0 (installed via Homebrew) and 1.1.0 (git head)
mu4e: bundled with 1.0, and 1.1.0 correspondingly
OS: MacOS Sierra 10.12

Any other detail

I noticed that if I wait for a while after I opened mu4e before doing my first jump/search, the slowness is not there. Digging a little further I noticed that the messages are displayed only after the message:
"Contacts received: 22702".

I tried to replace mu4e~fill-contacts function with an empty one, to see if that is the reason, but this didn't help.

Then I have seen that this function is called from mu4e~request-contacts. And replacing this function with an empty function fixed the slowness issue. Reading through the code, I noticed that the disabling email address completion effectively does the same thing. So I went back to the original code, and simply added config to disable it.

(setq mu4e-compose-complete-addresses nil)

This also fixed the slowness. But with this config, I also lost the email autocomplete feature.

[rhgg2]

I have exactly the same issue with 1.1.0 (HEAD b4cc67d). Installed from homebrew on MacOS Sierra 10.12.6.

I updated from 0.9.19c (HEAD 22e11fc) and initial indexing became significantly slower (from 1s to 7-8s). I have around 80,000 messages and 10,000 contacts.

If I (setq mu4e-compose-complete-addresses nil) then initial indexing is instantaneous.

[rhgg2]

If I update mail from mu4e-main rather than mu4e-headers, then there is no delay.

If I update from mu4e-headers, press q while it is stuck on "Searching..." and then re-enter inbox view, then there is no further delay.

If I (setq debug-on-quit t) and C-g while mu4e is stuck on "Searching..." then it is always stuck inside the C function redisplay_internal

[djcb]

The slow first query is apparently due to us waiting the contacts to appear.

Not sure why this takes so long on MacOS; on my Linux machine, M-: (mu4e~request-contacts) is done in less than a sec (with 8465 contacts).

[marcbowes]

I'm on MacOS too. Running the command on the CLI returns 22k contacts really fast. The same thing takes 7s for me from within Emacs (this time determined by setting mu4e-debug to t and then looking at the timestamps in *mu4e-log*:

2018-08-23 16:48:44 -> "cmd:contacts personal:false after:1262332800"
2018-08-23 16:48:44 * Received 4096 byte(s)
.........
.........
2018-08-23 16:48:51 * Received 3915 byte(s)
2018-08-23 16:48:51 <- (:contacts ((:mail "....... etc.

[marcbowes]

This is still causing me problems. Every time I encounter a new contact, I get 1568741 bytes from the mu contacts command. This freezes my Emacs. Because I have background updates enabled, this happens in an "unsolicited" fashion and is pretty frustrating. The workaround I have is to comment out the "update contacts" functionality and, periodically, fully reload the contacts. While this is fine, it'd be nice to have this fixed.

I guess first prize is to not send 1.5 megabytes if only 1 contact is new. What are the semantics of "after"? Is that a monotonic non-repeating timestamp? i.e. can mu4e get a timestamp back from each update, remember that, and pass it back the next time?

Another point of view is why is 1.5 mb so slow to parse? Is this just some silly code somewhere or inherent to parsing sexps..? (I have no idea, haven't profiled.)

For another brain dead idea, I was thinking of offloading this to another thread (Emacs has these now!). I have plenty of CPU, I just don't want my Emacs to lock up.

I'm happy to contribute a fix, but I'd like some guidance before pursuing a path you (@djcb) are not happy with.

[marcbowes]

Recent updates to Emacs (I think - I recently updated to 26.2) have improved this - time is now between 0.2s and 0.8s for the same payload size. But the freeze is still noticeable.

I realized threads don't help at all because they're currently cooperative.

[djcb]

We're sending all contacts because it's easy, and seems fast enough when tested (i.e, , don't even notice it). In those case, easy of implementation wins.

Now, if apparently it is very slow in some environments / with a lot of contacts, perhaps we should reconsider, and only send the delta.

[marcbowes]

That'd be great. What I have right now is:

1. I commented out the call to mu4e~request-contacts-maybe
2. I use an idle timer to run it

This is actually an OK experience since I seldom need to compose to a newly discovered contact - if I do, I'm usually replying to an email list or something where the reply action does the job of filling in the address.

The only gap then is integration with ldap (or something) so that I can email people I've never emailed before. But that's something I can sort out for myself.

[djcb]

The (development) git branch has something for this... only the changed contacts are sent.

[marcbowes]

Sorry for the late reply..

Thanks for this. I've pulled the changes and will let you know if I run into anything.

[marcbowes]

I've noticed a couple of things over the last few weeks.

1. Generally, things are much slower. I have no idea why, but updating mail and rendering searches is significantly slower.
2. Many contacts are missing. When I use cfind on the command line I can find them. If I setq the variables/updated-at timestamp, mu4e will do a full refresh but still doesn't find them.

I may have screwed something up and haven't dug into what is going on, but I just wanted to let you know that I'm still trying this out.

[djcb]

For 2), mu4e-compose-complete-only-after was updated to 2014-01-01, that might explain.

As for 1), yeah, would need something a bit more precise. There are no changes in mu4e 1.3 that would affect search speed or updating mail.

[ryantate]

The (development) git branch has something for this... only the changed contacts are sent.

Has this been merged in to a release yet, @djcb, or do I still need to pull from development branch?

I need this as apparently I have 137,427 "contacts" :-\

At least now I know why my initial bookmark visit is always blank for many minutes.

(Anything else I should be doing with that many contacts? lol)

[djcb]

This is only in the git version (1.3.x) right now.

[djcb]

There are some things you can do: use only 'personal' contacts, and set mu4e-compose-complete-only-after. See the manual for the former; there's a section on Limiting the number of addresses.