Improve onboarding by scanning all folders from time to time

[hpk42]

Especially at the beginning, users e.g. of Web.de but also other providers face issues when contact requests or messages are sorted to spam. Some providers also have special "priority" folders by default. It thus happens that Messages don't arrive at just-started Delta Chat users and they abandon usage. This is a known problem and there were discussions between various folks. Yesterday i sat together with @dignifiedquire who just had a failure to onboard a friend because of folders. After a lot of to/and-fro and discussing various scenarios this is where we landed as practical next steps with this plan and considerations:

• before or after INBOX idle trigger a new "scan all folders for messages". It does a "list folders" and then goes through all folders with select-statements, checking if "next-uid" was changed since checked last time. This might be batchable but in any case should not consume a lot of traffic. We might debounce this scan activity to happen at most every N minutes

• if next-uid changed for a folder, we "prefetch" and "fetch" DC-messages as as needed ("dc-messages" are not just those with "Chat-Version" headers, but can also be regular emails)

• all folders, their last uidvalidity, next-seen etc. are kept in a separate "imap-sync" sqlite table. Maybe this can be used to streamline some of the "Sent" folder and "DeltaChat" folder detection code we already have.

• if we discover DC-messages in folders that have the "/Spam" flag (maybe excluding ContactRequests) we automatically move them to INBOX/DeltaChat folder to help provider-spam-systems to regard this contact/mail as non-spam

• for now, we do not change any user visible option, but introduce this "scan all" automatically and on top of what exists. The DeltaChat folder-watching does not perform scan-all-folders (maybe with the exception to trigger scan-all also with DeltaChat if INBOX is not watched)

• maybe for the first 50 messages or so, we could reduce the IDLE-timeout (currently 23 minutes or so) to faster detect messages sent to non-inbox folders. However, on Android and iOS, we would likely trigger scan-all when the app moves to foreground, and so it might not be neccessary to reduce the current idle-timeout at least for them. We can leave this "faster discovery" question for the end, after we move to real-life testing.

Later on, after the above works, we can consider heuristics on which folders to perform IDLE on, and remove the Watch-folder options (inbox, deltachat, sent). We tried to find a safe scheme for already doing it but failed to fine one, too many unknowns, also some questions regarding multi-device (you might have different settings with each of it, one moves, the other doesn't etc.) so we postponed this in favor of the above incremental suggestion.

[gerryfrancis]

I would like to understand why some providers mark contact requests as SPAM. It should be possible to reach out to the mail providers and find it out.

[hpk42]

On Mon, Nov 02, 2020 at 01:50 -0800, gerryfrancis wrote: I would like to understand why some providers mark contact requests as SPAM. It should be possible to reach out to the mail providers and find it out.

Spam-filtering is often not happening very transparently, and it's not easy to find out about what a particular provider does or why, let alone ask them for change. That being said, we sometimes change DC if it helps avoid being marked as spam. As we want to provide a reliable onboarding experience, where sending and receiving messages "just" works, i don't see a way around refining our folder-scanning algorithms. Very likely, when we automatically move messages from spam to inbox folder the provider will learn (i think GMail does) from it.

[Hocuri]

Most of these points are done, for what's left see #2140