Acknowledgements: Maxim Khitrov
WARNING: DOES NOT WORK WITH GMAIL IMAPD SINCE IT DOES NOT IMPLEMENT THE STANDARD RFC5256
imap2json converts an imap URL of a folder to a HTML conversation view (Organised by thread)
- (Dovecot) IMAP server for http://tools.ietf.org/html/rfc5256
- Golang for sanitising and serialising IMAP response to JSON
- Web interface to render the JSON, between familiar index and expanded conversation views
Example use case:
Ruth and Kai have an email exchange arranging a movie. A friend Jamie wants to catch up on the conversation and perhaps join. Kai emails Jamie the link to the start of the conversation.
The URL points to the Web email archive produced imap2json of the entire conversation. Now Jamie knows all the details, in a familiar Web conversation view, that is mails of the same topic are grouped together, of the email exchange. There is no need for Kai or Ruth to use the clumsy "Forward all" mechanism on their email client.
- go get github.com/kaihendry/imap2json
- $GOPATH/bin/imap2json imap://imap.dabase.com
This will generate mail.json from our test repo imap://imap.dabase.com
For a simple Web interface (json2html) for mail.json see serve our html directory from your Webserver.
Why use Dovecot?
This saves us from having to code this tricky part ourselves by tracking Message-ID: and References: / In-Reply-To:, which can be rather complex.
IMAP also gives us a uniform interface at an IMAP URL, e.g.
imap://$USER[@]imap.dabase.com[/folder] to generate an archive from.
Raw dovecot output
printf "1 select inbox\n2 UID THREAD references us-ascii all\n3 fetch 1:* envelope\n4 logout\n" | /usr/lib/dovecot/imap
Or using netcat with our read only test IMAP service, so you don't need to setup Dovecot:
nc imap.dabase.com imap << EOF 1 authenticate anonymous foobar 2 select inbox 3 UID THREAD references UTF-8 all 4 fetch 1:* envelope 5 logout EOF
THREAD (1 2 3 4)(5)
We can see messages 1,2,3,4 are a single conversation about a "Movie".
5 is a separate conversation about a Dentist appointment.
-- 1 \-- 2 |-- 3 \-- 4 -- 5
Mutt is an excellent MUA and reference point.
mutt -f Maildir
Mutt conjures a great threaded conversation view as a reference independent of Dovecot's THREAD rfc5256 support. So it would be interesting to eventually compare the threading structure.
Using mutt to debug IMAP commands
mutt needs to be compiled with
$ mutt -v | grep -i debug Configure options: '--prefix=/usr' '--sysconfdir=/etc' '--enable-imap' '--enable-debug' '--enable-smtp' '--enable-hcache' '--with-curses=/usr' '--with-regex' '--with-gss=/usr' '--with-ssl=/usr' '--with-sasl' '--with-idn' 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -D_FORTIFY_SOURCE=2' 'LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,--hash-style=gnu' +DEBUG
We need a mutt configuration for testing:
$ cat mutt-imap2json set spoolfile=imap://email@example.com set folder=imap://firstname.lastname@example.org set sort=threads set sort_aux=reverse-last-date-received
Now run mutt in debug mode with the above configuration:
mutt -d2 -F mutt-imap2json
.muttdebug0 for a complete log
grep '4>' .muttdebug0 # show the IMAP commands issued by mutt
Going to use Golang to interface with Dovecot IMAP using http://godoc.org/code.google.com/p/go-imap
http://golang.org/pkg/net/mail/ can help parse/sanitise addresses, text bodies and convert Dates to Epoch.
json.MarshalIndent the sanitised email metadata & body into JSON.
Hoping to statically generate a large JSON file per mailbox. This will be compressed on delivery to the Web interface.
JSON to Web interface
The index & expanded views are probably most important.
Conversation URL structure
Gmail uses 16 character alphanumeric string as a conversation id, e.g. https://mail.google.com/mail/u/0/#inbox/1424243f629686c1
Fastmail uses two 16 character alphanumeric strings with a dash in between, e.g. https://www.fastmail.fm/mail/Inbox/5cf2a643c23b82dc-f65945369u632?u=148c411e
The sha1sum of the first message (\r\n line endings) of a conversation could be used as the id of the conversation.
$ unix2dos 'Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S' unix2dos: converting file Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S to DOS format ... $ sha1sum 'Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S' 5048d370149a7a5d25dc17869cb1404cf747b6bb Maildir/cur/1386050966.M540929P13587.sg.webconverger.com,S=315,W=328:2,S
So the "movie" conversation would be identified by
http://imap2json.dabase.com/#5048d370149a7a5d25dc17869cb1404cf747b6bb or rather some short, easily sharable version like: http://imap2json.dabase.com/#5048d