Lag on half-down/half-up scrolling on macOS

[bmclean2]

I searched for existing issues related to this, but didn't find the exact problem I am experiencing. I apologize if it has been brought-up already.

Expected Behaviour

Half-page scrolling up and down smoothly without lag

Actual Behaviour

When holding the half-up and half-down (Ctrl-U/D for me) scrolling on macOS 12/Alacritty there is a lot of screen lag and the screen lags far behind the number of registered keystrokes. Here are some tests I ran:

Using a mailbox with 11,000 messages starting at the top and scrolling to the bottom using half-down,

macOS/Alacritty or Terminal took: 80s
Ubuntu/Gnome Terminal took: 24s

When half-down scrolling through the same mailbox a second time without having closed neomutt,

macOS/Alacritty took: 20s -- the issue rectifies itself after the first scroll-through.
Ubuntu/Gnome Terminal took: 24s

This has been a long standing issue since I began using Neomutt a few years ago on macOS. Changing header cache db provider did not change the results.

Steps to Reproduce

See above

How often does this happen?

• Always

When did it start to happen?

• Since I started using Neomutt a few years ago.

NeoMutt Version

NeoMutt 20240425-1-3de005-dirty
Copyright (C) 2015-2024 Richard Russon and friends
NeoMutt comes with ABSOLUTELY NO WARRANTY; for details type 'neomutt -vv'.
NeoMutt is free software, and you are welcome to redistribute it
under certain conditions; type 'neomutt -vv' for details.

System: Darwin 21.6.0 (x86_64)
ncurses: ncurses 6.4.20221231 (compiled with 6.4.20221231)
libiconv: 1.11
libidn2: 2.3.7 (compiled with 2.3.7)
GPGME: 1.23.2
libnotmuch: 5.6.0
PCRE2: 10.43 2024-02-16
storage: tokyocabinet
compression: zlib

Configure options: --disable-doc --tokyocabinet --notmuch --pcre2 --autocrypt --with-ncurses=/usr/local/opt/ncurses --zlib

Compilation CFLAGS: -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -D__EXTENSIONS__ -D_XOPEN_SOURCE_EXTENDED -I/usr/local/Cellar/gpgme/1.23.2_1/include -I/usr/local/Cellar/libassuan/2.5.7/include -I/usr/local/opt/libgpg-error/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -DNCURSES_WIDECHAR -I/usr/local/opt/ncurses/include -I/usr/local/include -I/usr/local/Cellar/libidn2/2.3.7/include -I/usr/local/Cellar/pcre2/10.43/include -I/usr/local/Cellar/tokyo-cabinet/1.4.48/include -I/usr/local/include -O2

Compile options:
  +autocrypt +fcntl -flock -fmemopen +futimens +getaddrinfo -gnutls +gpgme
  -gsasl -gss +hcache -homespool +idn -inotify -locales_hack -lua -mixmaster
  +nls +notmuch -openssl +pcre2 +pgp -sasl +smime +sqlite +truecolor

MAILPATH="/var/mail"
PKGDATADIR="/usr/local/share/neomutt"
SENDMAIL="/usr/sbin/sendmail"
SYSCONFDIR="/usr/local/etc"

Extra Info

• Is your email local (maildir) or remote (IMAP)? maildir

[flatcap]

Hmm...
From the other issue, you index_format looks like:

set index_format = '%3C %4Z %?X?📎& ? %D %-20.20F %.70s (%-4.4c) %>'

The %X could be the cause.
To count the number of attachments, NeoMutt has to parse the entire email.

the issue rectifies itself after the first scroll-through.

Once the cache has been populated, NeoMutt can use the cached value.
Everything should be zippy again.

---

The caches are very sensitive to code changes and new releases can sometimes cause the cache to be invalidated.
We try to keep these changes to a minimum, but sometimes they're unavoidable.

If you see a persistent slowdown, please let me know.

Thanks

[bmclean2]

@flatcap That actually fixed it! Good call, thank you so much!

[bmclean2]

@flatcap Just seeking clarification on this. Removing %X does indeed fix the lag. Assuming I want to keep %X, however, what's not clear is why the cache that makes everything zippy on subsequent scrolls doesn't persist on Neomutt restart. When you say "cache," do you mean the disk-stored header/body caches, or is there some separate memory-based caching that happens? I haven't been able to get body caching to work because I think it requires the use of IMAP/POP (I use isync/mbsync), so maybe this is my issue. 🤷🏻‍♂️

[flatcap]

Just seeking clarification on this

No problem...
I've just checked the behaviour, to be certain.

The header cache stores the value for %X, but only for the emails you've seen in the Index.

Calculating %X is slow, so NeoMutt will do it on demand (for the page you're looking at)
That info will be saved to the header cache.

When you change pages, more reading needs to be done.
If you restart NeoMutt and open the same folder, the pages you've looked at before will be fast.
Anything else will be slow, once.

Unfortunately, there isn't a way to tell NeoMutt to re-cache an entire folder.
You either need to hold down <Page-Up>, or create huge macro to do it a lot of times.

---

Here are my header cache settings:

# Trailing / means create separate dirs for each folder
set header_cache = "~/.cache/mutt/"
# Fast modern database
set header_cache_backend = "lmdb"
# Enable compression; even low levels are quite effective in saving space
set header_cache_compress_level = 18
set header_cache_compress_method = "zstd"

[bmclean2]

When you change pages, more reading needs to be done. If you restart NeoMutt and open the same folder, the pages you've looked at before will be fast. Anything else will be slow, once.

My header cache settings (lmdb,zstd,10) are close to yours and when I do change-folder and come back to the original, or restart Neomutt, the lag returns. I checked again, and for some reason the lag is nearly non-existent on Linux with %X. Deleting the existing cache files did not help.