Message Charset / Content-Transfer-Encoding Issue

[washcroft]

When a message contains no parts (i.e. a message with a simple HTML body), that message may have been sent with a specific Charset and/or Content-Transfer-Encoding specified in the headers governing the whole message (e.g. Content-Transfer-Encoding: quoted-printable).

If link/open tracking is enabled, the message is put through the parser and the body is altered accordingly, but at the same time Postal is overwriting these two headers on the Mail object here:

postal/lib/postal/message_parser.rb

Lines 36 to 43 in 2f62baa

	if @mail.mime_type =~ /text\/plain/
	@mail.body = parse(@mail.body.decoded.dup, :text)
	@mail.content_transfer_encoding = nil
	@mail.charset = "UTF-8"
	elsif @mail.mime_type =~ /text\/html/
	@mail.body = parse(@mail.body.decoded.dup, :html)
	@mail.content_transfer_encoding = nil
	@mail.charset = "UTF-8"

...this means @mail.to_s is returning a differently encoded body. Normally this would be fine, but the new message headers are not updated in Postal's raw tables - meaning when Postal comes to deliver the message, it does so using the headers of the message as it was received, but with a body that has since been encoded differently.

This means email clients are having a hard time rendering the message, often failing miserably.

To fix this, I have commented out the four lines which are overwriting these two headers - but I suspect the proper solution needs to allow updated headers being returned from the parser?

[willpower232]

Ah good spot.

Unfortunately the message with the converted links is not preserved and I feel that is on purpose however I have seen other software only record the version with the changed links so perhaps it would be more correct to save the tracked links to the database.

Presumably an alternative would be to try and identify the current charset for the message? Not sure how feasible that is.

[washcroft]

Not storing the orignal body is OK, as only the parsed/injected body is needed for delivery post this stage of processing.

However, the headers need to be updated as well to avoid the situation where the headers say one thing, but the body is something different.

Or perhaps my solution is fine, is there a reason why Postal trys to force the Charset / Content-Transfer-Encoding?

[willpower232]

Oh I see what you mean, that is a bit outside my knowledge, I can see that the raw_message already contains the headers so hopefully there is some method available to change the headers without reconstructing the message even further.

I don't know why it forces it as it does, I could only imagine that something in the inserted links or images would require it but I don't know if there is a difference between ANSI HTML or whatever and UTF-8 HTML.

[washcroft]

The UTF8 charset is probabaly less problematic, but when the content transfer encoding is changed that breaks all kinds of things - particularly when a message is sent as quoted-printable, and the comes out of the parser as not quoted-printable. HTML can no longer be rended properly, because the equals signs in HTML tags aren't ecaped.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[willpower232]

@catphish related to #2727 ?

[catphish]

Thank you for the detailed bug report. This is fixed by 2834e2c