Search and preview for MailDir files on MacOS
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
mutt-launcher-app/Contents
muttlight.xcodeproj
src
.gitignore
LICENSE
LICENSE_BOOST
LICENSE_BSD
design.md
muttlight.duti
readme.md

readme.md

Muttlight

Muttlight Logo

Muttlight is a MacOS application that improves search and preview for email files stored in MailDir format. Specifically, it allows the associated file extensions to be specified and registered via a graphical user interface, provides a Spotlight importer to extract meta data, integrates with Quick Look to provide previews and thumbnails, and allows files to be opened directly in Mutt.

Muttlight Launcher was created with Sveinbjörn Þórðarson's Platypus application. It responds to ‘open’ requests on MailDir messages by running a simple (naive) shell script to launch an Iterm2 session, or failing that, a native Terminal session, with Mutt open on the selected message.

Documentation is available elsewhere.

Pull requests are welcome.

Building Muttlight

  1. Install prerequisites
brew install gettext
brew install ncurses
  1. Clone (or link) the Mutt source code into a src/mutt subdirectory
wget -qO- ftp://ftp.mutt.org/pub/mutt/mutt-1.8.3.tar.gz \
  | tar xvz -C ./src && ln -s mutt-1.8.3 src/mutt
  1. Build Mutt
(cd src/mutt && ./configure && make)
  1. Build Muttlight
cd src
make

Install Muttlight simply by copying the Muttlight.app bundle (directory) to $HOME/Applications.

Background

Problem

Mutt is a text-based email client that works as well in a terminal on MacOS as on any other system. It provides an efficient, configurable, programmable, and keyboard-only interface, seamless integration of powerful texts editors like Vim, and a non-vendor-specific and easy-to-synchronize storage format, namely individual MIME files in MailDirs.

Unfortunately, mail messages stored in MailDir format are not well integrated into the MacOS search (Spotlight) and preview (Quick Look) features. Mails appear in Spotlight results, since they are indexed, but their encoded filenames are all but unreadable and their contents are not displayed. Renaming these files to have the .eml extension causes them to be properly indexed, displayed, and previewed by exploiting plugins in the native Mail application. But, this violates the MailDir format and renders the files unreadable by Mutt and similar applications.

Two possible solutions suggest themselves.

  1. Introduce a version of the MailDir specification that requires .eml extensions and modify Mutt and similar applications accordingly.

  2. Somehow get the Mail application to also index these MailDir files without renaming them.

The first solution is possible since Mutt is open-source software, but it either complicates system configuration, since Mutt must be installed with patches, or requires upstream acceptance of the patches. It would, furthermore, be necessary to patch mail delivery agents like fetchmail and postfix. But, is it reasonably for a relatively minor issue on a particular operating system to spread to such long-standing and widely-used pieces of software?

The second solution would be ideal, but unfortunately I was not able to make it work. I tried registering file extensions in the finder, with the duti application, and using the UTTypeConformsTo feature of MacOS applications. The Mail application is closed source and so it is not possible to investigate it directly or to patch it. If you know how to do this or you work for Apple and can influence the application, please contact me! I would be very pleased to make Muttlight redundant by finding a better solution to the problem described above.

Solution

Muttlight represents a third solution: an application that registers itself as the owner of MailDir files and leverages Mutt's source code to improve their integration into the MacOS search and preview features.

A filename in the MailDir format ends with a suffix of the form :2,DFPRST, where the six final letters are flags that are either present or absent (giving 64 different combinations). Since file extensions on MacOS begin with a ., it is also necessary to consider the characters to the left of the suffix. They are sometimes mbox, but more often they are (part of) the host name on which the file was originally created (to ensure distinct names across multiple systems). Files in the new subdirectory are named without a :2,DFPRST suffix but usually include a host name. Obviously, the host name extensions will vary across installations. Muttlight provides a GUI that searches the system for MailDir files, summarizes the extensions that are being used, and allows users to select those which should be treated. Note that the DoveCot version of the MailDir scheme cannot be handled since it embeds file size information in the extension.

Once the MailDir filename ‘extensions’ have been registered to the Muttlight file type (org.tbrk.muttlight.email), the system will rely on Muttlight to extract metadata and provide preview images. Both features are provided by plugins that exploit the existing Mutt source code to parse and display mail messages.

Metadata is extracted from the mail header and any plain text, html, or enriched parts. Attachment names are indexed, but not their contents. The various parts are decoded before being indexed. This should improve the quality of search results, particularly for emails in character sets other than ASCII. I do not know whether the Mail application indexes mail attachments but this seems difficult to achieve using the public Spotlight interfaces (since it would be necessary to recursively call other importer plugins).

The Quick Look plugin exploits (...hacks around...) the Mutt pager to give previews that, while not as elegant as the native Mail ones, closely resemble the text-based display in the Mutt client. This style may even be preferable to some users. Rudimentary parsing is applied to colorize the header fields, attachment status lines, and quoted replies.

Altogether, these features, together with the Mutt Launcher component, allow a rapid configuration and natural integration of MailDir contents into the MacOS user interface.

Other Solutions

There are, of course, other ways to index and search mail messages in MailDir format. For instance, I already use mairix on both MacOS and Linux, and it works well once you get the hang of it. It has the advantage of working specifically with mail files and of showing the results directly within Mutt.

The advantage of Spotlight is that the indexes are kept up-to-date more dynamically and they span all files and file types on a system. Muttlight allows more convenient browsing of mail messages when they come up amongst other results (pdfs, text files, source code, etcetera).

Debugging

Run mdimport -L to see the list of installed Spotlight plugins. The list should include .../muttlight.app/Contents/Library/Spotlight/muttlight.mdimporter.

Run mdls on a MailDir file to check that the plugin is working correctly. The kMDItemContentType field should be org.tbrk.muttlight.email and the other fields (kMDItemAuthors, kMDItemDisplayName, etcetera) should be valid. To log the importing process and to see whether the plugin is being called, run mdimport -d 4 <filename>.

Run qlmanage -m | egrep --color '.*muttlight.*|$ to see the list of installed Quick Look plugins (and to highlight the muttlight entry). The list should include org.tbrk.muttlight.email -> .../muttlight.app/Contents/Library/QuickLook/muttlight.qlgenerator. It may be necessary to reset the plugin manager by running qlmanage -r (and qlmanage -r cache).

Open a MailDir directory in Finder to check that thumbnails are being generated correctly, press ⌘-Y to preview a file. The thumbnails and previews should show the messages more or less as they appear in mutt.

The Quick Look plugin can be debugged by running qlmanage -d 4 -p <filename>. This will show detailed logging. There may be a delay before the preview is displayed. Add -o . to the command line to dump the generated files to disk.

Run mdfind <keywords> to list indexed files that contain the given keywords. Entering the same keywords into the Spotlight search box (⌘-Space) should display the same list. When muttlight is functioning correctly, any mail files in the search results are named according to their subject (and not their MailDir filename) and their contents should be previewed as in the mutt pager. It will probably be necessary to reindex your MailDir directories before the search results are named correctly (the mds and mds_stores processes become active during reindexing).

The following command deletes and regenerates the Spotlight index on the root volume (regeneration may take some time).

sudo mdutil -E /

Run the following command and search for muttlight to see registered file extensions and plugin “claims”.

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister -dump | less

Reset the entire Launch Services database:

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -kill -r -domain local -domain system -domain user