RFC2231-style MIME part header continuations erroneously used for long Content-ID fields

[madduck]

On NeoMutt 20240425 (Debian unstable), so relatively recent, I just encountered the following corner-case bug:

When composing a MIME message and the Content-ID of a MIME part I enter exceeds a certain length, NeoMutt splits the Content ID across multiple lines, according to RFC2231, but RFC2231 only applies to parameters, not fields.

This is what a normal MIME part header with Content-ID looks like:

--_004_b5b53e4f78ff4f1d88c062752ba7d24falarmkittelde_
Content-Type: image/jpeg; name="image001.jpg"
Content-Disposition: inline; filename="image001.jpg"
Content-ID: <image001.jpg@01DABB52.AA1ADAD0>

If the Content-ID value exceeds a certain length, it gets broken across lines:

--_004_b5b53e4f78ff4f1d88c062752ba7d24falarmkittelde_
Content-Type: image/jpeg; name="image001.jpg"
Content-Disposition: inline; filename="image001.jpg"
Content-ID: =?utf-8?q?=3Cimage001=2Ejpg=4001DasvasdvABasdvasBasdv52=2EAAasdv?=
 =?utf-8?q?asdvas1asdvasdvaAsdvaDsdvasAasdvasdvasdvdvasdvDas0dv=3E?=

Note how the angle brackets are preserved in the encoding (=3C and =3E).

When I assign such a long content ID to a part using NeoMutt, however, it seems to think it needs to invoke RFC2231, and makes the Content-ID field a parameter of Content-Type, which is wrong:

--_004_b5b53e4f78ff4f1d88c062752ba7d24falarmkittelde_
Content-Type: image/jpeg; name="image001.jpg";
        content-id*0="image001.jpg@01DasvasdvABasdvasBasdv52.AAasdvasdvas1asdvasd";
        content-id*1="vaAsdvaDsdvasAasdvasdvasdvdvasdvDas0dv"
Content-Disposition: inline; filename="image001.jpg"

In doing so, it also loses the angle brackets, but that doesn't really matter anymore, as the MIME part is now misformed.

[gahr]

Thanks for the detailed bug report - I think I can fix it :)

[madduck]

Amazing, @gahr! Thanks! Let me know if you want me to test…

[gahr]

I see that support for Content-Id was added in #3197. The implementation put the Content-Id into the container for Content-Type parameters (perhaps for lack of a better existing place?), then it's special-cased and handled as a separate header. Unfortunately, that special-casing happens after the Content-Type parameters undergo RFC2331-transformations.

@dcpurton can you please comment on the design choice? Would putting Content-Id into a dedicated field in struct Body sound better to you? Otherwise, we need to fix the special-casing.

[gahr]

@madduck could you please give the branch devel/issue-4327 a try?

[dcpurton]

@dcpurton can you please comment on the design choice? Would putting Content-Id into a dedicated field in struct Body sound better to you? Otherwise, we need to fix the special-casing.

That's a great question…

Most likely a misunderstanding around use case of parameters. I vaguely remember at the time wishing there was a field in struct Body. Who knows why I didn't add it.

You should definitely replace with a dedicated field in struct Body.

[madduck]

@madduck could you please give the branch devel/issue-4327 a try?

Assuming I did everything right, this unfortunately does not fix the problem:

--agvk7etag6oa2zwo
Content-Type: image/png;
        content-id*0="171835140023.648213.12749145834755993267@very.long.fqdn.dom";
        content-id*1="ain.name.madduck.net"
[…]

lotus:~/code/neomutt|issue-4327|% ./neomutt -v                                  #(pts/4) 14 09:50:50.262
NeoMutt 20240425-89-194907
Copyright (C) 2015-2024 Richard Russon and friends
NeoMutt comes with ABSOLUTELY NO WARRANTY; for details type 'neomutt -vv'.
NeoMutt is free software, and you are welcome to redistribute it
under certain conditions; type 'neomutt -vv' for details.

System: Linux 6.7.9-amd64 (x86_64)
ncurses: ncurses 6.5.20240427 (compiled with 6.5.20240427)
libidn2: 2.3.7 (compiled with 2.3.7)

Configure options: 

Compilation CFLAGS: -std=c11 -fno-delete-null-pointer-checks -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -D__EXTENSIONS__ -D_XOPEN_SOURCE_EXTENDED -DNCURSES_WIDECHAR -I/usr/include -O2

Compile options:
  -autocrypt +fcntl -flock -fmemopen +futimens +getaddrinfo -gnutls -gpgme 
  -gsasl -gss -hcache -homespool +idn +inotify -locales_hack -lua -mixmaster 
  +nls -notmuch -openssl +pgp +regex -sasl +smime -sqlite +truecolor 

MAILPATH="/var/mail"
PKGDATADIR="/usr/share/neomutt"
SENDMAIL="/usr/sbin/sendmail"
SYSCONFDIR="/etc"

[gahr]

Assuming I did everything right

I fear you did not: you are running 194907, which is the last commit on the main branch.

[madduck]

I can confirm that the content ID is no longer a parameter of content-type:

Content-ID: <171836261187.679223.9223979057819607013@very.long.domain.fqdn.name.for.neomutt.madduck.net>

However, it should now be line-wrapped, see above.

[madduck]

Assuming I did everything right

I fear you did not: you are running 194907, which is the last commit on the main branch.

Certainly need to refresh my Git foo lol. Sorry.

[gahr]

I can confirm that the content ID is no longer a parameter of content-type:

Content-ID: <171836261187.679223.9223979057819607013@very.long.domain.fqdn.name.for.neomutt.madduck.net>

However, it should now be line-wrapped, see above.

Uhm.. are you suggesting that we should RFC2047-encode all long header field bodies so we can break them up? Interesting.. I don't think we do that, even for the subject - try composing a message with a very long subject without spaces, and you'll see that we don't RFC2047-encode + split it. Perhaps we should discuss this on a separate issue?

Edit: I have double checked what Apple Mail and Roundcube do, and neither encode long subjects with RFC2047 for the purpose of splitting them.

[madduck]

I cannot find a source saying that the headers should be encoded. Python's default email policy causes header lines to be encoded and folded to <=78 characters. But it doesn't seem like it's required to do this.