mails: suboptimal quoted-printable generation

[leahneukirchen]

lobste.rs sends mail where the entire value of a header is encoded as a single quoted-printable token and then split across lines. Possibly, this results in a UTF-8 multibyte sequence split across tokens,
which according to my reading of the RFCs is allowed, but resulted in at least two mail clients as rendering errors.

Perhaps a more clever version of qp can be implemented (or leveraged from some stdlib, but I couldn't find it.)

[pushcx]

Could you give an example of the bug and what a valid version of that header would look like, please?

[leahneukirchen]

puts ("foö "*30).quoted_printable(true)
=?UTF-8?Q?fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_f?=
	=?UTF-8?Q?o=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo?=
	=?UTF-8?Q?=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3?=
	=?UTF-8?Q?=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_fo=C3=B6_?=

Line 3 is split between a multibyte encoding.

Instead, something like

Subject: =?UTF-8?Q?fo=C3=B6?= =?UTF-8?Q?_fo=C3=B6?=
 =?UTF-8?Q?_fo=C3=B6?= =?UTF-8?Q?_fo=C3=B6?= =?UTF-8?Q?_fo=C3=B6?=
 =?UTF-8?Q?_fo=C3=B6?= =?UTF-8?Q?_fo=C3=B6?= =?UTF-8?Q?_fo=C3=B6?=

could be used.

[tinyplasticgreyknight]

I may take a stab at this, it's vaguely related to something I was working on recently.

[abdallahalsamman]

I was just trying to reproduce this, but I didn't get any false rendering in both gmail and thunderbird

[alanpost]

@chneukirchen,

It's been a bit better than two years since you filed this ticket including the following text:

lobste.rs sends mail where the entire value of a header is encoded as a single quoted-printable token and then split across lines. Possibly, this results in a UTF-8 multibyte sequence split across tokens,
which according to my reading of the RFCs is allowed, but resulted in at least two mail clients as rendering errors.

I don't suppose you know or remember which two mail clients you saw rendering errors in? I agree with your reading of the RFC that splitting multibyte sequences across a line is technically permitted, while also being easy to imagine how a parser could get that wrong. I'd love to see a working demonstration of the problem.

Thank you for your patience on this issue.

[briankung]

Pretty amazing, I can't even evaluate that string in the lobste.rs rails console (though I can in other pry environments):

Loading development environment (Rails 5.2.2.1)
[1] pry(main)> "fo�� "
/Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/history.rb:131:in `write': "\xC3" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/history.rb:131:in `puts'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/history.rb:131:in `save_to_file'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/history.rb:55:in `call'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/history.rb:55:in `push'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/pry_instance.rb:299:in `handle_line'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/pry_instance.rb:274:in `block (2 levels) in eval'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/pry_instance.rb:273:in `catch'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/pry_instance.rb:273:in `block in eval'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/pry_instance.rb:272:in `catch'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/pry_instance.rb:272:in `eval'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/repl.rb:77:in `block in repl'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/repl.rb:67:in `loop'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/repl.rb:67:in `repl'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/repl.rb:38:in `block in start'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/input_lock.rb:59:in `__with_ownership'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/input_lock.rb:77:in `with_ownership'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/repl.rb:38:in `start'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/repl.rb:13:in `start'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/pry-0.12.2/lib/pry/pry_class.rb:200:in `start'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-5.2.2.1/lib/rails/commands/console/console_command.rb:64:in `start'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-5.2.2.1/lib/rails/commands/console/console_command.rb:19:in `start'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-5.2.2.1/lib/rails/commands/console/console_command.rb:96:in `perform'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-5.2.2.1/lib/rails/command/base.rb:65:in `perform'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-5.2.2.1/lib/rails/command.rb:46:in `invoke'
	from /Users/briankung/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/railties-5.2.2.1/lib/rails/commands.rb:18:in `<top (required)>'
	from bin/rails:4:in `require'
	from bin/rails:4:in `<main>'

$> # pry crashes back out to shell

Looks like this issue: pry/pry-rails#69 as the comments note, it's related to the rb-readline gem.

[pushcx]

This is probably fixed now, thanks to some great spelunking by @briankung. If you're reading this comment because you got a mis-encoded email, please re-open and include:

1. the name + version of your email client
2. a screenshot of what it looks like
3. the headers you received (feel free to redact personal info)