Muttlight design notes
There are three subdirectories:
cur are named using six flags:
P: passed (resent/forwarded/bounced)
Flags must be stored in ASCII order.
2, indicates version 2 (production) of the standard.
There are thus 64 possible combinations (per hostname).
(* Generate all possible combinations in OCaml. *) let rec generate' xs = match xs with |  ->  | x::xs -> let xs' = generate' xs in x :: List.map (fun s -> x ^ s) xs' @ xs' let generate xs = "" :: generate' xs let print_query s = Printf.printf "kMDItemFSName=\"*:2,%s\"||" s (* let print_query = print_endline *) let _ = List.iter print_query (generate ["D"; "F"; "P"; "R"; "S"; "T"]) let _ = print_endline "false"
ocamlc -o gen gen.ml), then run
tmp) are named without the version indicator (
DoveCot uses an extension of the basic MailDir scheme. Filenames are of the form:
S= field gives the file size, and the
W= field gives the
"rfc822" size. This extended scheme cannot be supported by our tool.
TODO: Provide a simple script to (safely) rename such files.
It may be easy to adapt the application to work with MH.
kMDItemFSName: basename in file system
kMDItemDisplayName: name used in result list
kMDItemFSCreationDate: creation time
List all indexed mbox files:
See attributes of a file:
A spotlight importer is a small plug-in bundle. Only one importer is allowed per uniform type identifier (UTI). Specify supported UTIs in the importers Info.plist file.
Xcode includes a Spotlight project template that provides the required CFPlugin support, as well as templates for the required schema file.
Include inside app bundle inside the
MyApp.app/Contents/Library/Spotlight/SpotlightImporter.mdimporter +--/Contents +--Info.plist +--MacOS/SpotlightImporter +--Resources/schema.strings (custom attribute names) +--Resources/schema.xml (treatment of attributes/custom metadata)
- Show installed Spotlight plugins:
- Testing importer:
/usr/bin/mdimport -d2 test.myCustomDocument
- Explicitly register application:
lsregister -f MyApp.app
Results of running
mdls on a Mail draft:
_kMDItemOwnerUserID = 501 com_apple_mail_dateLastViewed = 2017-07-09 10:01:35 +0000 com_apple_mail_dateReceived = 2017-07-09 10:01:35 +0000 com_apple_mail_flagged = 0 com_apple_mail_messageID = "GIHOFmmmhBFl" com_apple_mail_priority = 3 com_apple_mail_read = 1 com_apple_mail_repliedTo = 0 kMDItemAccountIdentifier = "6E2A5D1B-7F22-426A-8D88-0963A36A3C08" kMDItemAuthorEmailAddresses = ( "firstname.lastname@example.org" ) kMDItemAuthors = ( "email@example.com" ) kMDItemContentCreationDate = 2017-07-09 10:01:35 +0000 kMDItemContentModificationDate = 2017-07-09 10:01:35 +0000 kMDItemContentType = "com.apple.mail.emlx" kMDItemContentTypeTree = ( "com.apple.mail.emlx", "public.data", "public.item", "public.email-message", "public.message" ) kMDItemDateAdded = 2017-07-01 17:47:10 +0000 kMDItemDisplayName = "Test email message 2" kMDItemFSContentChangeDate = 2017-07-09 10:01:35 +0000 kMDItemFSCreationDate = 2017-07-01 17:47:10 +0000 kMDItemFSCreatorCode = "" kMDItemFSFinderFlags = 0 kMDItemFSHasCustomIcon = (null) kMDItemFSInvisible = 0 kMDItemFSIsExtensionHidden = 0 kMDItemFSIsStationery = (null) kMDItemFSLabel = 0 kMDItemFSName = "143923.emlx" kMDItemFSNodeCount = (null) kMDItemFSOwnerGroupID = 20 kMDItemFSOwnerUserID = 501 kMDItemFSSize = 1286 kMDItemFSTypeCode = "" kMDItemIdentifier = "<8698CBE2-4ABF-464B-8F61-B11D93EF07AE@tbrk.org>" kMDItemIsApplicationManaged = 1 kMDItemIsExistingThread = 0 kMDItemIsLikelyJunk = 0 kMDItemKind = "Mail Message" kMDItemLastUsedDate = 2017-07-09 10:01:35 +0000 kMDItemLogicalSize = 1286 kMDItemMailboxes = ( "mailbox.drafts" ) kMDItemPhysicalSize = 4096 kMDItemRecipientEmailAddresses = ( "firstname.lastname@example.org", "email@example.com", "firstname.lastname@example.org", ) kMDItemRecipients = ( "Timothy Bourke", "Jean Do", "Andre\U0301 Who" ) kMDItemSubject = "Test email message 2" kMDItemUseCount = 4 kMDItemUsedDates = ( "2017-06-30 22:00:00 +0000", "2017-07-08 22:00:00 +0000" )
Fill in information:
com_apple_mail_dateReceived = from header Date: field. com_apple_mail_flagged = from filename (F) com_apple_mail_read = from filename (S) com_apple_mail_repliedTo = from filename (R) kMDItemMailboxes = ( "mailbox.drafts") ? kMDItemKind = "Mail Message" kMDItemIdentifier = from header Message-Id: field kMDItemDisplayName = from header Subject: field kMDItemSubject = from header Subject: field kMDItemContentCreationDate = from header Date: field. kMDItemAuthorEmailAddresses = ( from header From: field ) Array of CFStrings kMDItemAuthors = ( from header From: field ) Array of CFStrings kMDItemRecipientEmailAddresses = ( from header To: field ) kMDItemRecipients = ( from header To: field ) kMDItemContentType = "org.tbrk.muttlight.email" kMDItemContentTypeTree = ( UTI hierarchy of file "com.apple.mail.emlx", "public.data", "public.item", "public.email-message", "public.message" ) ?
The size of the Spotlight index for a given volume can be determined by
du -h -d 1 /.Spotlight-V100.
Spotlight still indexes files when no plugin is registered for them—that is,
the files appear in the search results for keywords that they contain. It
seems just to treat their contents as plain text. When a plugin is
registered, this does not happen automatically; it is necessary to return
kMDItemTextContent field. There are at least four possible ways to do
Recurse through the message (MIME) parts, including text directly, turning HTML into text, and possibly calling other filters to treat binary attachments. This is the ideal solution, but demands non-negligible programming and debugging time.
Use the mutt pager to turn a message into plain text, as is already done for the Quick Look plugin. This approach leverages the existing mutt code and attachments can be treated using the mailcap mechanism. The disadvantage is that many types of attachements (.pdf, .xls, .doc, etcetera) are not normally displayed as plain text by mutt. Unfortunately, this approach is not easy to implement without duplicating much of the mutt source code, since Spotlight importers execute in a sandbox which seems to prohibit them from creating temporary files and directories. The calls to
mkdtempin the mutt pager thus fail and the plugin crashes.
Simply slurp the whole file into
kMDItemTextContent. This approach is easy to implement and may be no worse than before installation of the plugin. Besides not properly treating file attachments, it does not decode RFC2045 text nor strip tags from HTML. It may cause spotlight to generate a larger index file than necessary.
Use a hybrid of 1 and 3: recurse through the parts, using mutt functions to decode RFC2045 text and
NSAttributedStringor custom code to decode HTML, and ignoring other attachments. This seems to be a reasonable compromise between development effort and results. It has been implemented in the current version.
There are several existing open-source "Quick Look Plugins":
It does not seem possible to somehow reuse the existing Mail Quick Look feature.
Put Quick Look plugins in
/Library/Quick Look and activate by resetting
Quick Look with
Quick Look can be invoked with ⌘-Y or
qlmanage -p file.
The following text summarizes the Apple Quick Look Programming docs.
Quick Look displays
- thumbnail: static image depicting a document (multiple at once).
- preview: a larger representation of a document (one at a time).
- Quick Look Consumer (client): wants to display a thumbnail or preview.
- Quick Look Producer (daemon): satisfies requests for thumbnails and previews using Quick Look Generators.
Producers and consumers communicate over one or more Mach ports. Allows crash-recovery of daemons and their termination when idle.
The producer consists of one or more Quick Look daemons (
multiple Quick Look generators.
The Generator interface is based on
CFPlugIn and is specified in ANSI C.
Clients have access to the public function
QLThumbnailImageCreate and to
the Quick Look preview panel (
The Generator must convert document data into one of the QuickLook native
types (plain text, rtf, html, pdf, jpg, png, tiff, etc.).
QLGenerator.h file specifies the programmatic interface for
generators. The API is broken into three categories:
GeneratePreviewForURLcallbacks (and callbacks for cancelling generation).
- Functions for creating graphics contexts to generate thumbnails and previews.
- Functions for returning more information about a given request.
Requests specify distinct options (dictionary of hints for generation) and properties (supplemental data).
QLPreviewRequestSetDataRepresentation with a contentTypeUTI of
kUTTypeHTML to render the text-based preview with the Web Kit. Use the
properties dictionary to specify attachements in the HTML (images, sounds,
If the generator and frameworks that it uses are thread-safe, then set the
in the generator's
Info.plist file. To handle cancellations, a generator
should either implement the two callback functions (difficult and not
recommended), or poll
A Generator bundle must have the
.qlgenerator extension and be in the
filesystem at one of the following locations (in order).
Debugging a generator, use either
/usr/bin/qlmanage -t /path/to/document # generate thumbnail /usr/bin/qlmanage -p /path/to/document # generate preview /usr/bin/qlmanage -m # print report from daemon defaults write -g QLEnableLogging YES # turn on logging
Run with particular generator:
qlmanage -c org.tbrk.muttlight.email -g ~/Library/Developer/Xcode/DerivedData/muttlight-quicklook-ezvizvcnspmnffbobvbwamlodciz/Build/Products/Debug/muttlight-quicklook.qlgenerator -d4 -p test5.mbox\:2\,S qlmanage -d4 -p test5.mbox\:2\,S | egrep --color '.*tbrk.*|$'
qlmanage is not displaying previews, it may be necessary to kill the
Parsing mail headers (DONE)
Better (easier and less potential for bugs) to just integrate the Mutt code
Yes. This works well.
Interpreting and Formatting Mail Messages
mutt_parse_mime_message, we need a
HEADER and a
mx.c:mx_open_mailbox() to create a
mh.c:maildir_parse_dir() (it calls
mutt_new_header()), which afterwards calls
which returns a
Or just use
copy.c:mutt_copy_message() to render the message as plain text
in a file? It calls
handler.c:mutt_body_handler which has a switch over
Opening a given message in Mutt
mutt -f ~/tmp/testeml -e 'push <limit>~i<186B6575EF414D42849814076AE9B22B0100401380@EU-DCC-MBX01.dsone.3ds.com>\n<limit>all\n<display-message>'
Uniform Type Identifiers
UTI: a string that identifies a document type.
E.g., "public.jpeg" supersedes "JPEG" OSType, ".jpg" and ".jpeg" extensions, and the mime type "image/jpeg".
Use reverse-DNS format, e.g., "com.apple.quicktime-movie".
They are defined in an inheritance hierarchy.
Identifier tags indicate alternate methods of type identification, such as filename extensions, MIME types, or NSPasteboard types. You use these tags to assign specific extensions, MIME types, and so on, as being equivalent types in a UTI declaration.
Mac apps can declare new UTIs for their own proprietary formats. You declare
new UTIs inside a bundle’s information property list.
info.plist of an application or spotlight bundle.
Ours would be declared as an imported UTI.
Do we need to declare our own UTI? Or can we simply import the Apple UTI
com.apple.mail.email and simply declare new filename-extensions?
No. This will not work, because we want our Spotlight Importer to be called and
not Apple's. We should thus declare our own UTI (
to.yp.cr.maildir.mail?) and declare that it derives from
the standard Apple one.
<key>UTExportedTypeDeclarations</key> <array> <dict> <key>UTTypeConformsTo</key> <array> <string>public.data</string> <string>public.email-message</string> </array> <key>UTTypeDescription</key> <string>Email Message</string> <key>UTTypeIconFile</key> <string>document.icns</string> <key>UTTypeIdentifier</key> <string>com.apple.mail.email</string> <key>UTTypeTagSpecification</key> <dict> <key>public.filename-extension</key> <string>eml</string> <key>public.mime-type</key> <string>message/rfc822</string> </dict> </dict> ... </array>
public.message, base type for messages (email, IM, etc.))
public.email-message, e-mail message, conforms to
There are API functions to convert other type identifiers (OSType, MIME, etc) to and from UTIs.
Dump the launch services database:
Register the UTIs and file extensions declared in an app's
Build a preferences pane modelled after the
Write it in Swift? No: stick with a mix of C and Objective C.
Show the icon (mutts sniffing around in folders), and a brief description.
Have two tabs:
"File extensions": table view of checkboxes populated from the plist file and an mdfind search.
"About": show author names and the license details.
Need to learn basic Mac GUI concepts:
- [http://www.apress.com/us/book/9781430245421](Learn Cocoa on the Mac)
- Preferences pane
- Rows with checkboxes
- Building from the commandline
Exploiting Apple plugins
MacOS already includes the following applications and plugins
/Applications/Mail.app /System/Library/Spotlight/Mail.mdimporter /System/Library/QuickLook/Mail.qlgenerator
Why not try to exploit them directly to minimize programming and give better results?
It seems that the following steps would be necessary.
/Applications/Mail.app/Contents/Info.plist, under the
UTExportedTypeDeclarationskey, wrap the existing
<array>and add the required extensions. To reregister,
touchthe application binary (
Contents/MacOS/Mail) and run
lsregisteron the bundle directory.
/System/Library/QuickLook/Mail.qlgenerator, as above, but edit the
I have not tested these steps, since they require write permission on the
system files (and
sudo is not enough). The Muttlight GUI could easily be
adapted to perform the required modifications (but using it in this way
would also required ‘violating’ MacOS' standard security settings).
There is reason to expect the QuickLook plugin to work, as evidenced by running the following command on a suitably renamed mail file.
qlmanage -d 4 -c com.apple.mail.email -g /System/Library/QuickLook/Mail.qlgenerator -p test.hostname\:2,S
But what about Spotlight indexing? The following command correctly imports
data from the message headers, but the
kMDItemTextContent field is not
populated. This means that the payload contents will not be indexed (maybe
the programmers were worried about Spotlight indexing costs?).
mdimport -d 4 -g /System/Library/Spotlight/Mail.mdimporter ./test.hostname\:2\,S
The same can be observed with a filename like
test.eml, showing that it is
not simply a problem of UTI registration. In fact, googling for ‘spotlight
eml files’ unearths a bunch of people trying to work around this limitation.
One approach would be to modify the Muttlight app to update the Mail
application and its QuickLook plugin, and to remove the Muttlight QuickLook
plugin but to keep the Muttlight spotlight plugin (edited to accept the
com.apple.email.email UTI). The only obstacle is the System Integrity
Protection (SIP) of MacOS. Suggestions on
include disabling it (requiring a reboot), or making copies of the required
applications elsewhere and editing the copies (may not work for Mail.app
since it includes a
Another approach would be to copy
/System/Library/QuickLook/Mail.qlgenerator into the Muttlight application
and to edit its
Info.plist file to accept the
UTI. This would seem to be an ideal solution: no need to work around SIP, it
exploits the Muttlight interface and Spotlight plugins, and use the Mail
plugin to generate high-quality previews. Unfortunately, I have not yet been
able to make it work. it's possible that the Mail plugin switches internally
on the UTI, in which case it won't accept
find and directly modify the relevant comparison operand in the binary?