Searching/limiting is slow because of many `auto_view` calls(?)

[rrueger]

Expected Behaviour vs Actual Behaviour

I have ~2500 mails in my mailbox.

Limiting with l ~b <term> in neomutt with auto_view text/html takes about 2 minutes.
Limiting with l ~b <term> in neomutt without auto_view text/html takes about 4 seconds.

This behaviour is observed even when the search term is a simple string (not a complex regex).
This appears to be independent of any .mailcap entries.

It appears to be that the auto_view program is called on each message as it is opened for mutt to search the body. This is evidenced by the following:

1. Append text/html; firefox '%s' &; test=test -n "$DISPLAY"; needsterminal; to .mailcap.
2. Limit (with no pattern modifier). This is fast.
3. Limit with ~b modifier and auto_view text/html.
4. Many, many tabs will be opened in firefox. This is slow.

In applications like viewing html mail, or extracting .ics attachments and printing the contents, I think it makes more sense for the auto_view program to be called only when the mail is opened to read, not to be searched.

This effect is probably(?) also seen when searching, but heavily exasperated when limiting, since the command is called for every mail in the mailbox.

Steps to Reproduce

Slow limiting

neomutt -n -F <(echo -e 'auto_view text/html\nset folder  = "<path to maildir>"') -f <large mailbox>

Fast limiting

neomutt -n -F <(echo 'set folder  = "<path to maildir>"') -f <large mailbox>

How often does this happen?

• Always

NeoMutt Version

NeoMutt 20211029
Copyright (C) 1996-2020 Michael R. Elkins <me@mutt.org>
Copyright (C) 1996-2002 Brandon Long <blong@fiction.net>
Copyright (C) 1997-2009 Thomas Roessler <roessler@does-not-exist.org>
Copyright (C) 1998-2005 Werner Koch <wk@isil.d.shuttle.de>
Copyright (C) 1999-2017 Brendan Cully <brendan@kublai.com>
Copyright (C) 1999-2002 Tommi Komulainen <Tommi.Komulainen@iki.fi>
Copyright (C) 2000-2004 Edmund Grimley Evans <edmundo@rano.org>
Copyright (C) 2006-2009 Rocco Rutte <pdmef@gmx.net>
Copyright (C) 2014-2020 Kevin J. McCarthy <kevin@8t8.us>
Copyright (C) 2015-2020 Richard Russon <rich@flatcap.org>

Many others not mentioned here contributed code, fixes,
and suggestions.

    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation; either version 2 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program; if not, write to the Free Software
    Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

To learn more about NeoMutt, visit: https://neomutt.org
If you find a bug in NeoMutt, please raise an issue at:
    https://github.com/neomutt/neomutt/issues
or send an email to: <neomutt-devel@neomutt.org>

Extra Info

• Operating System and its version
  archlinux 5.16.11-arch1-1
• Were you using multiple copies of NeoMutt at once?
  No
• Were you using 'screen' or 'tmux'?
  tmux
• Is your email local (maildir) or remote (IMAP)?
  Maildir

[roccoblues]

Hi @rrueger, can you reproduce this with the latest NeoMutt version?

[rrueger]

Unfortunately yes

Proof that the bodies are being opened by programs specified in auto_view are that many firefox tabs are opened when firefox is set to the auto_view program for the text/html mimetype.

• Neomutt version NeoMutt 20241212-29-ecf7a5-dirty
• OS archlinux 6.6.69-1-lts
• Using tmux
• Maildir

[roccoblues]

Hi @rrueger, I had a quick look at the code and confirmed what you observed.

msg_search() calls mutt_body_handler() which then calls autoview_handler() if the email body can be filtered with by mailcap.

I think that actually make sense in most cases. For example we decode encrypted mails here to search in the message.

Also for text/html you probably don't want the search to match the HTML markup? Not really sure what to do here. @flatcap @gahr any thoughts?

[lkmorlan]

I'm seeing slowness related to the limit command and I'm not sure if it is the same issue or not. In a mailbox with a few thousand emails, limit the displayed emails. Hit Enter to view an email and then return to the listing. neomutt will hang on "Sorting mailbox..." for a minute or so.