abusehelper.core.mail: Generalized mail parsing

[ghost]

This pull request adds the subpackage abusehelper.core.mail, which provides the capability to write a generic handler for a mail feed. The trick is that the same handlers can then used parse mails fetched over IMAPv4 or read from a Maildir directory. The handlers can also be tested from the command line and in automated unit tests.

The included README.md aims to document the functionality while giving a short tutorial how to use the code.

The goal is to offer a replacement the current mail feed related code such as abusehelper.core.imapbot and abusehelper.core.shadowservermail. This pull request doesn't remove any old functionality though, and should therefore be fully backwards compatible.

Tasks

• Add a base Handler class
• Add a Message class w/ the ability to stream the payload without needing to read everything to memory at once.
• Add a Mailbox runner
  • Add --concurrency option for handling multiple mails at once
• Add a IMAPv4 runner
• Add a command line tester
• Add a way to write tests for handlers
• Implement a ShadowServer mail handler
• Update README.md

[ghost]

The current abusehelper.core.shadowservermail sets the IMAP filter to (BODY "dl.shadowserver.org" UNSEEN) by default. It would be handy if the IMAP runner, when launched with a ShadowServer handler shadowserver.Handler with

$ python -m abusehelper.core.mail.imapbot user@xmpp.example lobbyroom shadowserver.Handler ...

would know magically to apply a suitable IMAP filter. Currently the IMAP runner can be explicitly configured with a filter, but this doesn't happen automatically based on the handler.

Should the handler even be bothered with such things, though? Also the separation of concerns would be clearer is the Handler could just concentrate on parsing mails that get thrown at it, while the runners can concentrate on fetching the mails.

Currently the Maildir runner just assumes that all mails in a mailbox should be processed. Maybe the IMAP runner could do the same.

[execgit]

I always specify filters manually for imap bots, so fine by me.

[ghost]

Removed the backwards compatibility layer task, as it should not be a blocker for merging this pull request. This pull request doesn't touch abusehelper.core.imapbot and abusehelper.core.shadowservermail at all and they can be deprecated separately if needed.