Copyright 2015 by Elliot Wolk
This project is free software, released under the GPLv3

smtp-cli Copyright Michal Ludvig 2003-2014
smtp-cli is free software, released under the GPLv3
  small changes to smtp-cli Copyright 2015 Elliot Wolk

qmlcompletionbox Copyright 2013 Vyacheslav Blinov
qmlcompletionbox is free software, released under the GPLv2
qmlcompletionbox is based on SuggestionBox.qml from liquid browser

liquid Copyright 2011 Jocelyn Turcotte
liquid is free software, released under the GPLv2

Simple IMAP client with a QT gui and a CLI
Minimal dependencies, currently minimal features

icons:
  Blue Magic
    by RevZAP
    https://www.iconfinder.com/iconsets/blue-magic
  WooCons #1
    by Janik Baumgartner
    http://www.woothemes.com/2010/08/woocons1
  Knob Buttons Toolbar icons
    by iTweek
    http://itweek.deviantart.com/art/Knob-Buttons-Toolbar-icons-73463960
  Diagram
    by Double-J designs
    https://www.iconfinder.com/iconsets/diagram

FILES
  src/email-gui.py
    simple python QT gui (pyside+QML)
      essentially a wrapper around email.pl
    should work anywhere QT/python/perl work with little effort
    explicitly supported platforms:
      Sailfish OS
      Debian GNU/Linux
    QML is located in /opt/email-gui/

  src/email.pl
    simple command line IMAP client
    caches all headers, and caches unread bodies
    e.g.:
      > echo "configure $HOME/.secrets first"
        configure /home/wolke/.secrets and run 'email.pl'
      >
      > email.pl
      GMail: logging in
      fetching all message ids
      fetched 15906 ids
      caching headers for 15906 messages
      caching bodies for 1 messages
      >
      > email.pl --summary
      GMail 2015-01-20 11:10:01 "Naked Girls Reading NYC" <ngrnyc@gmail.com>
        Join us tomorrow night for SCIENCE!
      >
      > email.pl --print
      ACCOUNT: GMail
      UID: 35870
      DATE: Tue, 20 Jan 2015 11:10:01 -0500
      FROM: "Naked Girls Reading NYC" <ngrnyc@gmail.com>
      SUBJECT: Join us tomorrow night for SCIENCE!
      BODY:
        This November, the Naked Girls give it all away
        as Naked Girls Reading presents SPOILER ALERT!
        Join the hit nude literary salon for an evening...
      >
      > email.pl --unread-line && echo
      G1
      > email.pl --body G 35870 | grep -o '<div>' | wc -l
      0
      > email.pl --body-html G 35870 | grep -o '<div>' | wc -l
      12

  src/email-search.pl
    builds and queries a sqlite database for header information like subject/to/from
    e.g.:
      > email-search.pl --updatedb G inbox all
      updatedb: all:19128 cached:19128 uncached:0 adding:0
      no UIDs to add
      > email-search.pl --search G 'bananas'
      23654
      23741
      23777

  src/smtp-cli
    SMTP command line client, written by Michal Ludvig
    https://github.com/mludvig/smtp-cli

  src/bash-complete.pl
    helper script that generates bash completion words

  bashcompletion/qtemail
    bash completion script that invokes bash-complete.pl

  ~/.secrets
    Account config goes here. See "Usage" below.


Dependencies:
  perl
  perl modules:
    Mail::IMAPClient
    IO::Socket::SSL
    MIME::Parser
    Date::Parse
    Date::Format
    MIME::Lite
  python
  pyside
  qt
  qtquick 1.1
  sqlite


Usage:
===== email.pl =====

  Simple IMAP client. {--smtp command is a convenience wrapper around smtp-cli}
  Configuration is in ~/.secrets
    Each config entry is one line of the format:
      email.GLOBAL_OPTION_KEY = <value>
      or
      email.ACCOUNT_NAME.ACCOUNT_CONFIG_KEY = <value>

    Account names can be any word characters (alphanumeric plus underscore)
    Lines that do not begin with "email." are ignored.

    ACCOUNT_NAME:    the word following "email." in ~/.secrets

    FOLDER_NAME:     "inbox", "sent" or one of the names from "folders"

    UID:             an IMAP UID {UIDVALIDITY is assumed to never change}

    GLOBAL_OPTION_KEY:
      update_cmd  [OPT] command to run after all updates
      encrypt_cmd [OPT] command to encrypt passwords on disk
      decrypt_cmd [OPT] command to decrypt saved passwords

    ACCOUNT_CONFIG_KEY:
      user             [REQ] IMAP username, usually the full email address
      password         [REQ] password, stored with optional encrypt_cmd
      server           [REQ] IMAP server, e.g.: "imap.gmail.com"
      port             [REQ] IMAP server port
      smtp_server      [OPT] SMTP server, e.g.: "smtp.gmail.com"
      smtp_port        [OPT] SMTP server port
      ssl              [OPT] set to false to forcibly disable security
      inbox            [OPT] primary IMAP folder name (default="INBOX")
      sent             [OPT] IMAP folder name to use for sent mail, e.g.:"Sent"
      folders          [OPT] extra IMAP folders to fetch (sep=":")
                       the FOLDER_NAME used as the dir on the filesystem
                       has non-alphanumeric substrings replaced with _s
                       and all leading and trailing _s removed
                       e.g.:
                         email.Z.folders = junk:[GMail]/Drafts:_12_/ponies
                           =>  ["junk", "gmail_drafts", "12_ponies"]
      count_include    [OPT] FOLDER_NAMEs for counts (default="inbox", sep=":")
                       list of FOLDER_NAMEs for account-wide unread/total counts
                       this controls what gets written to the global unread file,
                         and what is returned by --accounts
                       note this is the FOLDER_NAME, and not the IMAP folder
                       e.g.:
                         email.Z.sent = [GMail]/Sent Mail
                         email.Z.folders = [GMail]/Spam:[GMail]/Drafts
                         email.Z.count_include = inbox:gmail_spam:sent
                           => included: INBOX, [GMail]/Spam, [GMail]/Sent Mail
                              excluded: [GMail]/Drafts
      skip             [OPT] set to true to skip during --update
      body_cache_mode  [OPT] one of [unread|all|none] (default="unread")
                       controls which bodies get cached during --update
                         (note: only caches the first MAX_BODIES_TO_CACHE=100)
                       unread: cache unread bodies (up to 100)
                       all:    cache all bodies (up to 100)
                       none:   do not cache bodies during --update
      prefer_html      [OPT] prefer html over plaintext (default="false")
      new_unread_cmd   [OPT] custom alert command
      update_interval  [OPT] GUI: seconds between account updates
      refresh_interval [OPT] GUI: seconds between account refresh
      filters          [OPT] GUI: list of filter-buttons, e.g.:"s1=%a% s2=%b%"
                       each filter is separated by a space, and takes the form:
                         <FILTER_NAME>=%<FILTER_STRING>%
                       FILTER_NAME:   the text of the button in the GUI
                       FILTER_STRING: query for /opt/qtemail/bin/email-search.pl
                       e.g.:
                         email.Z.filters = mary=%from~"mary sue"% ok=%body!~viagra%
                           => ["mary", "ok"]


  /opt/qtemail/bin/email.pl -h|--help
    show this message

  /opt/qtemail/bin/email.pl [--update] [--folder=FOLDER_NAME_FILTER] [ACCOUNT_NAME ACCOUNT_NAME ...]
    -for each account specified {or all non-skipped accounts if none are specified}:
      -login to IMAP server, or create file ~/.cache/email/ACCOUNT_NAME/error
      -for each FOLDER_NAME {or just FOLDER_NAME_FILTER if specified}:
        -fetch and write all message UIDs to
          ~/.cache/email/ACCOUNT_NAME/FOLDER_NAME/all
        -fetch and cache all message headers in
          ~/.cache/email/ACCOUNT_NAME/FOLDER_NAME/headers/UID
        -fetch and cache bodies according to body_cache_mode config
            all    => every header that was cached gets its body cached
            unread => every unread message gets its body cached
            none   => no bodies are cached
          ~/.cache/email/ACCOUNT_NAME/FOLDER_NAME/bodies/UID
        -fetch all unread messages and write their UIDs to
          ~/.cache/email/ACCOUNT_NAME/FOLDER_NAME/unread
        -write all message UIDs that are now in unread and were not before
          ~/.cache/email/ACCOUNT_NAME/FOLDER_NAME/new-unread
        -run /opt/qtemail/bin/email-search.pl --updatedb ACCOUNT_NAME FOLDER_NAME 100
    -update global unread counts file ~/.cache/email/unread-counts
      count the unread emails for each account in the folders in count_include
      the default is just to include the counts for "inbox"

      write the unread counts, one line per account, to ~/.cache/email/unread-counts
      e.g.: 3:AOL
            6:GMAIL
            0:WORK_GMAIL

  /opt/qtemail/bin/email.pl --smtp ACCOUNT_NAME SUBJECT BODY TO [ARG ARG ..]
    simple wrapper around smtp-cli. {you can add extra recipients with --to}
    calls:
      /opt/qtemail/bin/smtp-cli \
        --server=<smtp_server> --port=<smtp_port> \
        --user=<user> --pass=<password> \
        --from=<user> \
        --subject=SUBJECT --body-plain=BODY \
        --to=TO \
        ARG ARG ..

  /opt/qtemail/bin/email.pl --mark-read [--folder=FOLDER_NAME] ACCOUNT_NAME UID [UID UID ...]
    login and mark the indicated message(s) as read

  /opt/qtemail/bin/email.pl --mark-unread [--folder=FOLDER_NAME] ACCOUNT_NAME UID [UID UID ...]
    login mark the indicated message(s) as unread

  /opt/qtemail/bin/email.pl --delete [--folder=FOLDER_NAME] ACCOUNT_NAME UID [UID UID ...]
    delete the indicated messages (from IMAP server AND local cache)

  /opt/qtemail/bin/email.pl --delete-local [--folder=FOLDER_NAME] ACCOUNT_NAME UID [UID UID ...]
    delete the indicated messages from the local cache ONLY
    (does not delete from IMAP server)

  /opt/qtemail/bin/email.pl --move [--folder=FOLDER_NAME] ACCOUNT_NAME DEST_FOLDER_NAME UID [UID UID ...]
    move the indicated messages on the IMAP server from FOLDER_NAME to DEST_FOLDER_NAME
    this deletes the indicated messages from the local cache.
    this does NOT, however, download the newly moved messages in DEST_FOLDER_NAME;
      you need to update DEST_FOLDER_NAME to fetch them from the IMAP server

  /opt/qtemail/bin/email.pl --accounts
    format and print information about each account
    "ACCOUNT_NAME:<timestamp>:<relative_time>:<update_interval>s:<unread_count>/<total_count>:<error>"

  /opt/qtemail/bin/email.pl --folders ACCOUNT_NAME
    format and print information about each folder for the given account
    "FOLDER_NAME:<unread_count>/<total_count>"

  /opt/qtemail/bin/email.pl --header [--folder=FOLDER_NAME] ACCOUNT_NAME UID [UID UID ...]
    format and print the header of the indicated message(s)
    prints each of [Date Subject From To CC BCC]
      one per line, formatted "UID.FIELD: VALUE"

  /opt/qtemail/bin/email.pl --body [--no-download] [-0] [--folder=FOLDER_NAME] ACCOUNT_NAME UID [UID UID ...]
    download, format and print the body of the indicated message(s)
    if -0 is specified, print a NUL character after each body instead of a newline
    if body is cached, skip download
    if body is not cached and --no-download is specified, use empty string for body
      instead of downloading the body
    if message has a plaintext and HTML component, only one is returned
    if prefer_html is true, HTML is returned, otherwise, plaintext

  /opt/qtemail/bin/email.pl --body-plain [--no-download] [-0] [--folder=FOLDER_NAME] ACCOUNT_NAME UID [UID UID ...]
    same as --body, but override prefer_html=false,
      and attempt to convert the result to plaintext if it appears to be HTML
      (uses /usr/bin/html2text if available, or just strips out the tags)

  /opt/qtemail/bin/email.pl --body-html [--no-download] [-0] [--folder=FOLDER_NAME] ACCOUNT_NAME UID [UID UID ...]
    same as --body, but override prefer_html=true

  /opt/qtemail/bin/email.pl --attachments [--folder=FOLDER_NAME] ACCOUNT_NAME DEST_DIR UID [UID UID ...]
    download the body of the indicated message(s) and save any attachments to DEST_DIR
    if body is cached, skip download

  /opt/qtemail/bin/email.pl --cache-all-bodies ACCOUNT_NAME FOLDER_NAME
    attempt to download the body of all uncached bodies

  /opt/qtemail/bin/email.pl --print [--folder=FOLDER_NAME] [ACCOUNT_NAME ACCOUNT_NAME ...]
    format and print cached unread message headers and bodies
    fetches bodies like "/opt/qtemail/bin/email.pl --body-plain --no-download",
      similarly converting HTML to plaintext
    formats whitespace in bodies, compressing multiple empty lines to a max of 2,
      and prepending every line with 2 spaces

  /opt/qtemail/bin/email.pl --summary [--folder=FOLDER_NAME] [ACCOUNT_NAME ACCOUNT_NAME ...]
    format and print cached unread message headers

  /opt/qtemail/bin/email.pl --status-line [ACCOUNT_NAME ACCOUNT_NAME ...]
    {cached in ~/.cache/email/status-line when ~/.cache/email/unread-counts or error files change}
    does not fetch anything, merely reads ~/.cache/email/unread-counts
    format and print ~/.cache/email/unread-counts
    the string is a space-separated list of the first character of
      each account name followed by the integer count
    no newline character is printed
    if the count is zero for a given account, it is omitted
    if accounts are specified, all but those are omitted
    e.g.: A3 G6

  /opt/qtemail/bin/email.pl --status-short [ACCOUNT_NAME ACCOUNT_NAME ...]
    {cached in ~/.cache/email/status-short when ~/.cache/email/unread-counts or error files change}
    does not fetch anything, merely reads ~/.cache/email/unread-counts
    format and print ~/.cache/email/unread-counts
    if accounts are specified, all but those are omitted
    omits accounts with unread-count of 0

    the string is two lines, each always containing exactly three characters
    no line can be longer than 3, and if it is shorter, it is left-padded with spaces
    each line ends in a newline character

    if any account has error, prints:
      "ERR", "<total>"
    if any account has more then 99 emails, prints
      "big", "<total>"
    if more than two accounts have a positive unread-count, prints:
      "all", "<total>"
    if exactly two accounts have a positive unread-count, prints:
      "<acc><count>", "<acc><count>"
    if exactly one account has a positive unread-count, prints:
      "<acc><count>", ""
    otherwise, prints:
      "", ""

    <total> = total of all unread counts if less than 1000, or '!!!' otherwise
    <acc> = first character of a given account name
    <count> = unread count for the indicated account

  /opt/qtemail/bin/email.pl --has-error [ACCOUNT_NAME ACCOUNT_NAME ...]
    checks if ~/.cache/email/ACCOUNT_NAME/error exists
    print "yes" and exit with zero exit code if it does
    otherwise, print "no" and exit with non-zero exit code

  /opt/qtemail/bin/email.pl --has-new-unread [ACCOUNT_NAME ACCOUNT_NAME ...]
    checks for any NEW unread emails, in any account
      {UIDs in ~/.cache/email/ACCOUNT_NAME/new-unread}
    if accounts are specified, all but those are ignored
    print "yes" and exit with zero exit code if there are new unread emails
    otherwise, print "no" and exit with non-zero exit code

  /opt/qtemail/bin/email.pl --has-unread [ACCOUNT_NAME ACCOUNT_NAME ...]
    checks for any unread emails, in any account
      {UIDs in ~/.cache/email/ACCOUNT_NAME/unread}
    if accounts are specified, all but those are ignored
    print "yes" and exit with zero exit code if there are unread emails
    otherwise, print "no" and exit with non-zero exit code

  /opt/qtemail/bin/email.pl --read-config ACCOUNT_NAME
    reads ~/.secrets
    for each line of the form "email.ACCOUNT_NAME.KEY\s*=\s*VAL"
      print KEY=VAL

  /opt/qtemail/bin/email.pl --write-config ACCOUNT_NAME KEY=VAL [KEY=VAL KEY=VAL]
    modifies ~/.secrets
    for each KEY/VAL pair:
      removes any line that matches "email.ACCOUNT_NAME.KEY\s*="
      adds a line at the end "email.ACCOUNT_NAME.KEY = VAL"

  /opt/qtemail/bin/email.pl --read-options
    reads ~/.secrets
    for each line of the form "email.KEY\s*=\s*VAL"
      print KEY=VAL

  /opt/qtemail/bin/email.pl --write-options KEY=VAL [KEY=VAL KEY=VAL]
    reads ~/.secrets
    for each line of the form "email.KEY\s*=\s*VAL"
      print KEY=VAL

  /opt/qtemail/bin/email.pl --read-config-schema
    print the allowed keys and descriptions for account config entries
    formatted, one per line, like this:
    <KEY_NAME>=<DESC>
      KEY_NAME: one of: user password server port smtp_server smtp_port ssl inbox sent folders count_include skip body_cache_mode prefer_html new_unread_cmd update_interval refresh_interval filters
      DESC:     text description

  /opt/qtemail/bin/email.pl --read-options-schema
    print the allowed keys and descriptions for global option entries
    formatted, one per line, like this:
    <KEY_NAME>=<DESC>
      KEY_NAME: one of: update_cmd encrypt_cmd decrypt_cmd
      DESC:     text description

===== email-search.pl =====

  /opt/qtemail/bin/email-search.pl --updatedb ACCOUNT_NAME FOLDER_NAME LIMIT
    create sqlite database if it doesnt exist
    updates database incrementally

    LIMIT
      maximum number of headers to update at once
      can be 'all' or a positive integer

  /opt/qtemail/bin/email-search.pl --format WORD [WORD WORD..]
    parse and format QUERY="WORD WORD WORD" for testing

  /opt/qtemail/bin/email-search.pl --search [OPTIONS] ACCOUNT_NAME WORD [WORD WORD..]
    print UIDs of emails matching QUERY="WORD WORD WORD .."

    OPTIONS:
      --folder=FOLDER_NAME
        use FOLDER_NAME instead of "inbox"
      --minuid=MIN_UID
        ignore all UIDs below MIN_UID
      --maxuid=MAX_UID
        ignore all UIDs above MAX_UID
      --limit=UID_LIMIT
        ignore all except the last UID_LIMIT UIDs

    SEARCH FORMAT:
      -all words separated by spaces must match one of subject/date/from/to/cc/bcc
        apple banana
        => emails where subject/from/to/cc/bcc/date matches both 'apple' AND 'banana'
      -specify an individual field of subject/date/from/to/cc/bcc with a '~'
        from~mary
        => emails from 'mary'
      -negate a header field query with '!~' instead of '~'
        from!~mary
        => emails from everyone EXCEPT 'mary'
      -specify that the body must match with a '~'
        body~bus
        (can be abbreviated with just 'b', e.g.: "b~bus")
        => emails where the cached body matches 'bus'
      -negate a body query with '!~' instead of '~'
        body!~bus
        => emails where the cached body does NOT match 'bus'
      -specify disjunction with '++'
        from~mary ++ from~john ++ from~sue
        => emails from 'mary' PLUS emails from 'john' PLUS emails from 'sue'
      -group space or ++ separated words with parentheses
        (from~mary a) ++ (from~john b)
        => emails from 'mary' that match 'a' PLUS emails from 'john' that match 'b'
      -parentheses can nest arbitrarily deep
        (a ++ (b (c ++ d)))
        => emails that match 'a', PLUS emails that match 'b' AND match 'c' or 'd'
      -special characters can be escaped with backslash
        subject\~fish\ table
        => emails where subject/from/to/cc/bcc/date matches 'subject~fish table'
      -doublequoted strings are treated as words
        "this is a (single ++ w~ord)"
        => emails where subject/from/to/cc/bcc/date matches 'this is a (single ++ w~ord)'

    GRAMMAR:
      QUERY = <LIST_AND>
            | <LIST_OR>
            | <HEADER_QUERY>
            | <NEGATED_HEADER_QUERY>
            | <SIMPLE_HEADER_QUERY>
            | <BODY_QUERY>
            | <NEGATED_BODY_QUERY>
            | (<QUERY>)
        return emails that match this QUERY
      LIST_AND = <QUERY> <QUERY>
        return only emails that match both QUERYs
      LIST_OR = <QUERY> ++ <QUERY>
        return emails that match either QUERY or both
      HEADER_QUERY = <HEADER_FIELD>~<PATTERN>
        return emails where the indicated header field matches the pattern
      NEGATED_HEADER_QUERY = <HEADER_FIELD>!~<PATTERN>
        return emails where the indicated header field does NOT match the pattern
      SIMPLE_HEADER_QUERY = <PATTERN>
        return emails with at least one header field that matches the pattern
      BODY_QUERY = body~<PATTERN>
        return emails where the body matches the pattern
      NEGATED_BODY_QUERY = body!~<PATTERN>
        return emails where the body does NOT match the pattern
      HEADER_FIELD = subject | from | to | cc | bcc | date | body
        restricts the fields that PATTERN can match
      PATTERN = <string> | <string>"<string>"<string>
        can be any string, supports doublequote quoting and backslash escaping

    EXAMPLES:
      =====
      from~mary
        [from] LIKE mary
      =====
      mary smith
        ALL(
           |--[to cc bcc from subject] LIKE mary
           |--[to cc bcc from subject] LIKE smith
           )
      =====
      ((a b) ++ (c d) ++ (e f))
        ANY(
           |--ALL(
           |     |--[to cc bcc from subject] LIKE a
           |     |--[to cc bcc from subject] LIKE b
           |     )
           |--ALL(
           |     |--[to cc bcc from subject] LIKE c
           |     |--[to cc bcc from subject] LIKE d
           |     )
           |--ALL(
           |     |--[to cc bcc from subject] LIKE e
           |     |--[to cc bcc from subject] LIKE f
           |     )
           )
      =====
      subject~b ++ "subject~b"
        ANY(
           |--[subject] LIKE b
           |--[to cc bcc from subject] LIKE subject~b
           )
      =====
      from!~bob ++ subject!~math
        ANY(
           |--[from] NOT LIKE bob
           |--[subject] NOT LIKE math
           )
      =====
      "a ++ b"
        [to cc bcc from subject] LIKE a ++ b
      =====

===== email-gui.py =====

  /opt/qtemail/bin/email-gui.py [OPTS]

  OPTS:
    --page=[account|header|config|send|folder|body]
      start on the indicated page
    --account=ACCOUNT_NAME
      default the account to ACCOUNT_NAME {only useful with --page}
    --folder=FOLDER_NAME
      default the folder to ACCOUNT_NAME {only useful with --page}
    --uid=UID
      default the message to UID {only useful with --page}