downloading limited number of headers from IMAP/other box, based on search

[grault]

What would you like to be able to do?

In the time of Webmail clients like Google/Fastmail/..., mailboxes grow huge, with tens of thousands of emails in people's Inbox, because

1. they only get the most recent "one page" when they enter
2. they'll most probably search for something right after that anyway

I use Fastmail, both via Webmail and IMAP (sometimes with mutt, sometimes with bare openssl client_s, etc). I'm doing a the todo-list Inbox paradigm, but after being done, I just drop emails to Archive and search later if something is needed.

I don't like mutt for the latter activity (searching in Archive), because when I open the mailbox, it could take minutes to fetch the headers.

So my idea is --

To have a mailbox syntax for starting with a search, something like imaps://server/Inbox?search%20all%20on%2013-Jul-2018 (which is the IMAP syntax here, but the date could be "today") and then maintain an "active search", and when it changes, perform another IMAP search, like "search all subject foo" and fetch only the headers returned by the IMAP server.

What needs to change in NeoMutt?

I guess this could be some kind of extension.

Have you tried any alternative methods?

Not yet.. Let me know if this could already be done somehow.

Additional Info

I'm a developer myself and interested enough to give this feature some love and implement it. If you find it interesting.

[flatcap]

That's a great idea! Like Notmuch's virtual folders.

A quick look through the IMAP RFC shows lots of searching commands.

Because the server's doing the work, there would be a few limitations:

• Searches would happen on a single folder
  (We could merge the results from several folders)
• The searches used fixed strings, not regular expressions
• The server wouldn't decode the message (I think)
  So, encoded text might not work, e.g. quoted-printable

That said, even simple "time windows" would be very useful to power users with too much mail.

[flatcap]

Here's some info to get you started:

• General newbie info - https://neomutt.org/dev/newbie-tutorial
• How to build NeoMutt - https://neomutt.org/dev/build
• How to debug a curses app - https://neomutt.org/dev/debug

Inside NeoMutt, an email looks like:

• https://neomutt.org/code/email.html

There's also a global Context. This represents the 'current' mailbox.
Backends can store mailbox-specific data in its data member.

When NeoMutt opens a mailbox, it calls mx_mbox_open() which delegates to imap_mbox_open()
In Notmuch, this is where the query is decoded, using functions from email/url.c

Later NeoMutt will connect to the server and issue commands:

• LIST or LSUB to get mailboxes
• SELECT a mailbox

There's a lot to know to get started, so ask lots of questions.

[grault]

Thanks a lot for the pointers! I read through these documents, cloned the repo, built the master branch (to play with and to read). I'm using vim/ctags for now, it seems to be enough to navigate the parts I'm interested in. I'll spend some time now with reading and in the debugger.

I'll be back.

Possibly with some questions. :)

[poetaman]

@flatcap This could be the need of the hour, it takes a lot of time for neomutt to start. I have 150k+ emails in gmail, and it does take quite a bit of time (45 seconds) each time for neomutt to launch (it says "Evaluating cache... X%"). Quitting is even worse, it hangs and doesn't quit at all (it does for another gmail account which has fewer emails). I waited for 10 minutes for it to quit before killing the process. It mostly maintained a 0% CPU usage, which went up to 0.6% only after every 5-15 seconds. When I killed it with C-c, it exited with segmentation fault.

So far I have been excited about using neomutt, though the handling of accounts with a lot of emails makes me rethink if it's feasible to use it as a main email client. Is there something that can be done about this at the moment?