Dovecot as main storage
When I posted the "Alpha" (postgres based) version, it was a good conversation on Reddit. I was thinking about IMAP support after that. I didn't want to write my own IMAP server, so I started to look closer to Dovecot. After first investigation I had an idea that probably I can use Dovecot as main storage, because I didn't want to duplicate emails over Dovecot and Postgres anyway. At summer 2017 I had a chance to evaluate the idea, after some period of deep work I got a first Dovecot based prototype and confidence that I can build a great webmail based on it.
The hardest things to deal with:
- unusual storage (limited IMAP instead of well known and powerful SQL)
- IMAP folders instead of tags
- I needed proper threads with ability to link them together
But at the same time it gives great features:
- already a part of mail stack (often used in home-grown setup)
- simple immutable storage, designed exactly for emails
- a bunch of email related plugins and integrations
- master-master replication
After trying few different approaches for hardest things I got pretty flexible storage.
The current storage
IMAP keywords are used for system tags (like
#trash) and user-defined tags.
Just four IMAP folders for:
- all original messages (no separate folder for Inbox, Spam or Trash)
- all parsed messages for effective web representation
- useful metadata (settings, linked threads, cache, etc.)
- cleaned messages from
#trash(autoexpunge in 30 days)
The most sensitive data is a folder with original messages and settings, parsed messages and metadata can be regenerated. Dovecot ACL is used to deny mailbox operations, expunging messages, etc. I use Dovecot's own high-performance mailbox format single-dbox, so emails are never changed even during applying IMAP flags on them.
As for now, I don't rely on Gmail as storage anymore, so the Dovecot mailboxes are only source of truth, so I should be really careful with them. I use two servers from two different datacenters with master-master replication enabled, so far it has been working pretty well.
From time to time, I want to check Inbox on mobile, so it has limited "general" IMAP support using Dovecot virtual mailboxes and filtering by
#trash tags. I use it with FairEmail for Android for fast notifications. Mailur also has a basic mobile view.
As I have only one IMAP folder with all messages, it works pretty well with Sieve scripts using IMAP FILTER=SIEVE, because I don't need to move messages around and just mark them with a proper flag. Also it's easy to run them through all messages (available from UI).
I use Dovecot Lucene for full text search indexing and it works pretty well for me.
I like the current storage and I think it should be pretty stable from that point. It has good flexibility in storing different metadata. Metadata can change over time with new features, but as I said before metadata can be easily regenerated, so it shouldn't be a problem.
I think I'm pretty close to my initial goal to move away from Gmail. Even I still use Gmail as a transport (importing over IMAP and sending over SMTP), but I already can remove emails from Google and I'll do this soon.