neomutt fails to decode (some?) RFC2047-encoded headers

[otizonaizit]

Since version 20180223 neomutt fails to properly decode headers encoded following RFC2047. For example, an email with Subject header:

Subject: =?UTF-8?Q?Sicherheitsl=C3=BCcke in praktisch allen IT-Systemen?=

should be shown in the message index and in the pager as:

Subject: Sicherheitslücke in praktisch allen IT-Systemen

The encoded version is shown instead. Version 20171215 works fine on the same mailbox. Also mutt 1.9.4 works on the same mailbox. My conclusion is that there must be something in neomutt that happened between 20171215 and 20180223.
By reading the RFC2047 I am not sure that such headers are actually standard conform, but they were working fine until now, so I'm guessing this is a bug in neomutt?

I attached an example mailbox (with a .txt extension to be able to attach it to the GitHub issue)
I reported this bug also in Debian, but I guess it is more relevant here.

Thank you for your work on neomutt :)))

test.txt

[gahr]

I'll look into this one

[gahr]

Oh - simply by looking at the mbox you have attached it's clear what's wrong: the subject contains spaces within an encoded-text production, which is against RFC2047, §2.

I think there was an option in previous versions that allowed NeoMutt to deal with these invalid representations. I'll see whether we can accept those in a non-breaking way.

[otizonaizit]

Ah OK, I had no doubt the headers were not RFC compliant: they come from a Microsoft client ;)

But, version 20171215 of neomutt displays the mailbox "correctly", in the sense that the headers are decoded and can be read by humans, so I think that some recent change in neomutt is causing these invalid headers to not be displayed the way they were before...

[otizonaizit]

By the way, option rfc2047_parameters still exists and it was set to yes. But I guess that option has nothing to do with the issue we are discussing here...

[gahr]

But, version 20171215 of neomutt displays the mailbox "correctly"

Yes, we used to be less strict, see

neomutt/rfc2047.c

Lines 768 to 769 in ae61285

	/* non-strict check since many MUAs will not encode spaces and question marks */
	for (q = q + 3; 0x20 <= *q && *q < 0x7f && (*q != '?' || q[1] != '='); q++)

[gahr]

I have pushed a commit to the branch devel/issue-1189 that should solve this issue, but I want to double check whether I'll break something else before submitting it for review.

[gahr]

By the way, option rfc2047_parameters still exists and it was set to yes. But I guess that option has nothing to do with the issue we are discussing here...

yeah, that's unrelated :)

[gahr]

I had no doubt the headers were not RFC compliant: they come from a Microsoft client ;)

@otizonaizit which client was that? Is there any chance that this bug was fixed in some more recent release? I just want to make sure we're not introducing funny behaviour just for the sake of an out-of-date piece of software out there...

[otizonaizit]

In the headers of the several emails with broken headers I see:

X-Mailer: MIME::Lite 3.030 ...

which seems to point to this: http://search.cpan.org/~rjbs/MIME-Lite-3.030/lib/MIME/Lite.pm

Now this is a wild guess, but given that all these messages came from the university administration and are addressed to all employees, I assume the perl library is used within or aside of a SAP system.

I am not sure the client library is "outdated". The corresponding Debian pacakge libmime-lite-perl lists version 3.030 as the last release in Nov 2013...
Not sure though how much this package is used in the wild...

[otizonaizit]

On the other hand, what is the risk of being more lenient? I mean, mutt/neomutt was doing this since at least 8 years already, right? (see 34323cc) so I fail to see the risk of re-introducing this. The more strict decoding was an unintended result of refactoring?

[gahr]

Yes I'd noticed the reference to MIME::Lite, but I couldn't find the place where that piece of software does RFC2047 encoding, so I thought that was perhaps done by another component? Admittedly, I had just glimpsed through the perl code, and also admittedly my perl sight-reading foo is not that advanced..

[gahr]

On the other hand, what is the risk of being more lenient? ... I fail to see the risk of re-introducing this

I tend to agree.

The more strict decoding was an unintended result of refactoring?

I have to be honest here, and admit that it was pretty intentional.. I have coded the regexps to be strict regarding to RFC, and hoped that mailers around wouldn't mess with them. I was wrong apparently :)

[gahr]

Merged, thanks!