weird character � appears when autoviewing with w3m but doesn't appear when using w3m interactively or directly dumping from terminal

[milanglacier]

Expected Behaviour

when autoview with w3m, there shouldn't be weird character "�"

Actual Behaviour

There appears weird character "�"

Steps to Reproduce

the following is my mailcap file:

text/html; w3m -I %{charset} -v -F -T text/html %s; nametemplate=%s.html; needsterminal;

text/html; w3m -I %{charset} -v -F -T text/html -dump %s; nametemplate=%s.html; copiousoutput; needsterminal;

and neomuttrc:

alternative_order text/html text/enriched text/plain

When comparing autoviewing html and using <view-mailcap> in the attach page to open w3m interactively, the autoviewed version has very strange "�" character.

but it does not appear in interactive w3m:

Notably, call w3m -I 'UTF-8' -v -F -dump example.html in the terminal does also not show the weird character:

The attachment is the html file:

hello.html.zip

How often does this happen?

• It depends on the html file, for some html email, the display is fine, but for some html files, the problem always appear.

NeoMutt Version

PUT YOUR "neomutt -v" OUTPUT HERE

Copyright (C) 1996-2022 Michael R. Elkins and others.
NeoMutt comes with ABSOLUTELY NO WARRANTY; for details type 'neomutt -vv'.
NeoMutt is free software, and you are welcome to redistribute it
under certain conditions; type 'neomutt -vv' for details.

System: Darwin 21.6.0 (arm64)
ncurses: ncurses 6.3.20211021 (compiled with 6.3.20211021)
libiconv: 1.11
libidn2: 2.3.4 (compiled with 2.3.2)
GPGME: 1.17.1
OpenSSL: OpenSSL 1.1.1n 15 Mar 2022
libnotmuch: 5.6.0
storage: tokyocabinet, lmdb

Configure options: --prefix=/opt/homebrew/Cellar/neomutt/20220429 --gss --disable-idn --idn2 --lmdb --notmuch --sasl --tokyocabinet --with-gpgme=/opt/homebrew/opt/gpgme --with-lua=/opt/homebrew/opt/lua --with-ssl=/opt/homebrew/opt/openssl@1.1 --with-ui=ncurses

Compilation CFLAGS: -g -O2 -std=c99 -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -D__EXTENSIONS__ -I/opt/homebrew/opt/gpgme/include -I/opt/homebrew/Cellar/libassuan/2.5.5/include -I/opt/homebrew/opt/libgpg-error/include -I/opt/homebrew/Cellar/neomutt/20220429/include -I/opt/homebrew/opt/lua/include/lua5.4 -DNCURSES_WIDECHAR -I/opt/homebrew/opt/openssl@1.1/include

Default options:
+attach_headers_color +compose_to_sender +compress +cond_date +debug
+encrypt_to_self +forgotten_attachments +forwref +ifdef +imap +index_color
+initials +limit_current_thread +multiple_fcc +nested_if +new_mail +nntp +pop
+progress +quasi_delete +regcomp +reply_with_xorig +sensible_browser +sidebar
+skip_quoted +smtp +status_color +timeout +tls_sni +trash

Compile options:
-autocrypt +fcntl -flock -fmemopen +futimens +getaddrinfo -gnutls +gpgme +gss
+hcache -homespool +idn -inotify -locales_hack +lua -mixmaster +nls +notmuch
+openssl +pgp +regex +sasl +smime -sqlite +sun_attachment

MAILPATH="/var/mail"
PKGDATADIR="/opt/homebrew/Cellar/neomutt/20220429/share/neomutt"
SENDMAIL="/usr/sbin/sendmail"
SYSCONFDIR="/opt/homebrew/Cellar/neomutt/20220429/etc"

To learn more about NeoMutt, visit: https://neomutt.org
If you find a bug in NeoMutt, please raise an issue at:
https://github.com/neomutt/neomutt/issues
or send an email to: neomutt-devel@neomutt.org

Extra Info

• Operating System and its version

macOS 12.4

• Were you using multiple copies of NeoMutt at once?

no

• Were you using 'screen' or 'tmux'?

no

• Is your email local (maildir) or remote (IMAP)?

maildir

[jindraj]

Webmails and most of the GUI MUAs show the beginning of the message as a preview. E-mail producers generate their own customized text usually in a table cell or div with css display: none; to prevent displaying it when you open the message, but somehow it is taken into account for the preview. The mentioned characters are usually non breaking spaces or similar characters which are used to pad between the "preview" from the rest of the message to prevent actual content of the message leaking into the preview.

This is not a neomutt issue. I'm not sure if this is a correct behaviour of w3m or not.

[milanglacier]

On 12/16/22 00:58, Jakub Jindra wrote: Webmails and most of the GUI MUAs show the beginning of the message as a preview. E-mail producers generate their own customized text usually in a table cell or div with css `display: none;` to prevent displaying it when you open the message, but somehow it is taken into account for the preview. The mentioned characters are usually non breaking spaces or similar characters which are used to pad between the "preview" from the rest of the message to prevent actual content of the message leaking into the preview. This is not a neomutt issue. I'm not sure if this is a correct behaviour of w3m or not.

Thanks for all the information! I understand the rationale that why there are such kind of weird characters "�" as non breaking spaces. This is why when I call "w3m -dumo xxx.html" at terminal it outputs "spaces", however I still don't know why when in neomutt calling "w3m -dump %s; copiesoutput" w3m will output the character as non-space "�" instead of something looks like "spaces".

[jindraj]

You're right about it. It seems there's is a problem displaying these characters in neomutt pager.

[gahr]

This is specifically not a non-breaking space ( ) but spaces with zero-with-non-joiner (‌) characters in between. See line 267 and following of your hello.html file.

I've dumped the file to text then opened it in vim (w3m w3m hello.html -dump > hello.txt; vim hello.txt) and I see a number of those characters as <200c> .

Perhaps our pager should be taught to ignore those characters.

[gahr]

Please note that you'd see the same problem with a text file / text email containing those characters.