IMAP messages retrieved out-of-order

[patricktokeeffe]

I used another IMAP client (Thunderbird, fwiw) to copy messages into my inbox from another account. To the extent I can verify, none of the date/time metadata changed on those messages but K-9 Mail retrieved these old copied messages as if they were the newest messages.

Before I copied those messages, my K-9 Mail inbox was correctly populated with at least several dozen recent emails. After the copy, it only contained old copied messages and messages which arrived after I performed the copy. Hitting Load up to 25 more loaded 25 more old messages, not the 25 most recent inbox messages.

I tried using 'Clear local messages' on the Inbox folder but the old-message-first behavior remained.

The only solution I found was to keep loading messages until all the old copied ones were present in my K-9 Mail inbox. That's obviously a poor solution since it means I have to haul around hundreds of old emails.

Environment

K-9 Mail version: 5.010

Android version: 6.0.1

Account type: Exchange via IMAP

[philipwhiuk]

Developer notes - some hypothesising:

I think the reason for this is that we show and display messages based on UID not received time. The copied messages will have newer UIDs but older received times. UID is an IMAP property which is fairly transparent in terms of message contents.

I think we should still fetch the messages UIDs during the poll, but also keep a mapping of UID to receipt time and then show by receipt time.

There are other issues with relying on UIDs exclusively - e.g. #1074 - which should be considered when doing this work.

My general instinct is that this will be a fairly chunky re-work of some of the store behaviour though.

[cketti]

See e.g. https://groups.google.com/d/msg/k-9-mail/Vbt8LmEECqY/IPIiKbcjT7oJ

[uschindler]

I think k9Mail should simply call the IMAP SORT command using the correct sort. Very old IMAP servers don't support sorting while fetching mails, but all new ones do it:

https://datatracker.ietf.org/doc/html/rfc5256

K9Mail only executes simple FETCH commands with UID, which no modern mail program does anymore if the SORT extension in IMAP is available. Other programs like Thunderbird or Outlook all use the SORT command, if available.

If you manually select a sort order in the menu of K9, this order only gets applied to local messages and is not executed on server.

[penguinpee]

I think that's the correct approach. K-9's current behavior of fetching the newest (highest) UIDs bit me today. An unexpected filter behavior (possibly a bug) in Thunderbird emptied my INBOX into another folder. I moved stuff back, but Thunderbird wasn't quite done moving stuff. So, that happened out of order.

UIDs are per folder. So, when moving stuff around, UIDs are re-assigned.

Way to recovery:

1. Move everything out of INBOX into a temporary folder
2. Sort it by reverse date
3. Move it back into INBOX

Or wait for the next 20 messages to arrive...