k9mail silently stopped to sync mails with IMAP, while other clients work.

[mmueller-kaffeeschluerfercom]

I am facing two problems here: The first is that the IMAP synchronization does not work anymore, and the second is that it fails silently, i.e. there is no notification about the error. This is bad because I rely on k9mail to be notified when I receive mails or when there is an error.

Expected behavior

IMAP mail should sync according to the time schedule I set in the settings. If there is a problem, a notification should appear.
Normally, when sync is fired, the status changes to: "Poll Account:Inbox 0/200" and then the 0 is counting up the numbers of new mails in the Inbox.

Actual behavior

The sync is fired periodically like it should. The status hangs a few seconds (around 10 s) on: "Poll Account:Inbox 0/200" an then returns to "Next poll In ... minutes". No new mails are shown. No notification is shown. All other clients show newly arrived mails.
I tried "Clear messages" and "Recreate data". After that, the Inbox was empty. Then, the next sync attempt downloaded 4 old messages and then failed again (silently). Every now and then, a single new mail comes through, but very few do...
The problem began sometime on october 24. Before this, I never had this problem (using k9mail for some years now).
If I switch to the folder view, under the "Inbox" title, there is a grey line saying "NumberFormatException: For".

I then switched on logging and the last entry is an error:

[10-28 13:53:17.798 8020:8040 E/ImapSync] synchronizeMailbox java.lang.NumberFormatException: For input string: "NIL" at java.lang.Integer.parseInt(Integer.java:521) at java.lang.Integer.parseInt(Integer.java:556) at com.fsck.k9.mail.store.imap.ImapList.getNumber(ImapList.java:48) at com.fsck.k9.mail.store.imap.ImapFolder.parseBodyStructure(ImapFolder.java:1028) at com.fsck.k9.mail.store.imap.ImapFolder.parseBodyStructure(ImapFolder.java:981) at com.fsck.k9.mail.store.imap.ImapFolder.handleFetchResponse(ImapFolder.java:885) at com.fsck.k9.mail.store.imap.ImapFolder.fetch(ImapFolder.java:737) at com.fsck.k9.controller.imap.ImapSync.downloadLargeMessages(ImapSync.java:701) at com.fsck.k9.controller.imap.ImapSync.downloadMessages(ImapSync.java:471) at com.fsck.k9.controller.imap.ImapSync.synchronizeMailboxSynchronous(ImapSync.java:258) at com.fsck.k9.controller.imap.ImapSync.sync(ImapSync.java:63) at com.fsck.k9.controller.imap.ImapMessageStore.sync(ImapMessageStore.java:25) at com.fsck.k9.controller.MessagingController.synchronizeMailboxSynchronous(MessagingController.java:742) at com.fsck.k9.controller.MessagingController$9.run(MessagingController.java:726) at com.fsck.k9.controller.MessagingController.runInBackground(MessagingController.java:206) at com.fsck.k9.controller.MessagingController.access$000(MessagingController.java:122) at com.fsck.k9.controller.MessagingController$1.run(MessagingController.java:175) at java.lang.Thread.run(Thread.java:761)

So apparently, k9mail is detecting an error. But why is there no notification about the error? I missed some important mails thinking that no mails arrived... I don't know if I have to file a separate issue for the missing notification...

And could there be a workaround for the problem? Or a fix? As the other clients (Thunderbird, Apple Mail) work, I assume it's a problem of k9mail.

Just FYI, I'm using the davmail Exchange gateway davmail.sourceforge.net but this should not be the problem as all the other clients work like a charm and k9mail did work as well until october 24.

Steps to reproduce

1. Configure an IMAP account in k9mail.
2. Hit "Check mail".
3. No (or not all) new mails are synced.
4. No notification is shown.

Environment

K-9 Mail version: 5.600

Android version: 7.1.2

Account type: IMAP (davmail)

[mmueller-kaffeeschluerfercom]

I have tested master branch. Problem is still there.

I'm not sure why, but some messages generate an ImapList with the String "NIL" on index 6, where the body size should be stored, according to ImapFolder.java:977. Maybe because of messages with empty body? I don't know.

I have have changed ImapList.java to return 0 in case of "NIL" and the problem of not syncing mails seems to be resolved:

@@ -47,6 +47,9 @@ class ImapList extends ArrayList<Object> {
     }
 
     public int getNumber(int index) {
+        if (index==6 && getString(index).equals("NIL"))
+            return 0;
+
         return Integer.parseInt(getString(index));
     }

I'm not sure if I'm introducing a new problem with that. Is this the right location to solve this problem?

And then, for the second problem (not showing a notification): The error is catched at ImapSync.java:233. How can I fire a notification from there?

[cketti]

RFC 3501 defines the body size (body-fld-octets) as number, with number = 1*DIGIT. So using NIL instead of a series of digits is not standard-conforming.

Now it could be the server sending an invalid response. It could also be a bug in K-9 Mail when parsing the BODYSTRUCTURE response. The easiest way to tell is if you record a debug log that includes the server response.

For error reporting there's #1058.

[mmueller-kaffeeschluerfercom]

I tried to make a debug log but I can't find out how to include the server response. I checked "Log sensitive information" but the server response is sill not in the log. Nevertheless, while debugging in android studio I believe I narrowed down the problem to one server response. I read out the inputStream buffer inside the readResponse in ImapResponseParser.java. The answer is:

* 1378 FETCH (UID 29357 BODYSTRUCTURE ("TEXT" "PLAIN" NIL NIL NIL "7BIT" NIL 0))

So it really seems to be the server sending NIL, isn't it? But then, I wonder why the other email clients (thunderbird and macos mail) accept the NIL if it is not standard...

And I'm thinking about a better way to react on a malformed server response. Now it's just silently stopping to sync. For now, I can see three scenarios:

• Add my modification NIL ==> 0 (but handles only this specific malformation)
• Skip mails generating malformed responses and notify about server problems
• Stop syncing like now but show an error notification

What do you think?

[fbessou]

I also received an email which made synchronization fail. From what I understand, the second body part of the body structure is invalid:

2021-01-10 15:03:08.771 22990-23006/? V/ImapConnection: conn162279409<<<#null# [38, FETCH, [UID, 97136, BODYSTRUCTURE, [[TEXT, PLAIN, [CHARSET, utf-8], NIL, NIL, QUOTED-PRINTABLE, 2038, 41, NIL, NIL, NIL, NIL], [NIL, RELATED, [BOUNDARY, _=_swift_v4_1604479656_95b3ddbef36d89c6fac1d1baa1fd1f92_=_], NIL, NIL, NIL], ALTERNATIVE, [BOUNDARY, _=_swift_v4_1604479656_c82b80e4e05e0795b1edf3b11cdb39c2_=_], NIL, NIL, NIL]]]
2021-01-10 15:03:08.771 22990-23006/? V/ImapFolder: Stored uid '97136' for msgSeq 38 into map
2021-01-10 15:03:08.772 22990-23006/? E/ImapSync: synchronizeMailbox
    java.lang.IndexOutOfBoundsException: Index: 6, Size: 6
        at java.util.ArrayList.get(ArrayList.java:437)
        at com.fsck.k9.mail.store.imap.ImapList.getString(ImapList.java:38)
        at com.fsck.k9.mail.store.imap.ImapList.getNumber(ImapList.java:68)
        at com.fsck.k9.mail.store.imap.ImapFolder.parseBodyStructure(ImapFolder.kt:897)
        at com.fsck.k9.mail.store.imap.ImapFolder.parseBodyStructure(ImapFolder.kt:856)
        at com.fsck.k9.mail.store.imap.ImapFolder.handleFetchResponse(ImapFolder.kt:774)
        at com.fsck.k9.mail.store.imap.ImapFolder.fetch(ImapFolder.kt:622)
        at com.fsck.k9.backend.imap.ImapSync.downloadLargeMessages(ImapSync.kt:577)
        at com.fsck.k9.backend.imap.ImapSync.downloadMessages(ImapSync.kt:380)
        at com.fsck.k9.backend.imap.ImapSync.synchronizeMailboxSynchronous(ImapSync.kt:189)
        at com.fsck.k9.backend.imap.ImapSync.sync(ImapSync.kt:32)
        at com.fsck.k9.backend.imap.ImapBackend.sync(ImapBackend.kt:50)
        at com.fsck.k9.controller.MessagingController.syncFolder(MessagingController.java:657)
        at com.fsck.k9.controller.MessagingController.synchronizeMailboxSynchronous(MessagingController.java:614)
        at com.fsck.k9.controller.MessagingController.lambda$synchronizeMailbox$3$MessagingController(MessagingController.java:599)
        at com.fsck.k9.controller.-$$Lambda$MessagingController$Xx9NETpzWcoXEDzz-o_rzXywOEM.run(Unknown Source:8)
        at com.fsck.k9.controller.MessagingController.runInBackground(MessagingController.java:216)
        at com.fsck.k9.controller.MessagingController.access$000(MessagingController.java:108)
        at com.fsck.k9.controller.MessagingController$1.run(MessagingController.java:157)
        at java.lang.Thread.run(Thread.java:764)

In thunderbird, the message is fetched but is not rendered.

I suppose that the mail structure is broken leading to the imap server not producing a correct bodystructure, or something like that.
A solution could be to skip mails with broken descriptions and let the user know about that.