Can't send emails containing UTF-8

[oscarcarlsson]

If I include any non-ASCII characters in a email reply I get the following error back:

<lobsters-f9PT0vjI5L@lobste.rs>: Command died with status 1:
    "/srv/lobste.rs/http/script/parse_inbound_mail". Command output:
    /srv/lobste.rs/http/vendor/bundle/ruby/2.7.0/gems/commonmarker-0.23.4/lib/commonmarker.rb:40:in
    `encode': "\\xE2" from ASCII-8BIT to UTF-8
    (Encoding::UndefinedConversionError)  from
    /srv/lobste.rs/http/vendor/bundle/ruby/2.7.0/gems/commonmarker-0.23.4/lib/commonmarker.rb:40:in
    `render_doc'  from /srv/lobste.rs/http/extras/markdowner.rb:10:in `to_html'
    from /srv/lobste.rs/http/app/models/comment.rb:340:in
    `generated_markeddown_comment'  from
    /srv/lobste.rs/http/app/models/comment.rb:241:in `comment='  from
    /srv/lobste.rs/http/script/parse_inbound_mail:48:in `<main>'

This is a snippet of the mail that caused the above - notice the Content-Type, which states utf-8:

Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable

There's (hopefully) hope on the horizion - Matter. I wouldn't hold=20
my breath for it to come out anytime soon, but once it arrives it=20
will probably be very nice.

I just use Home Assistant myself, so I don't have to care about=20
which ecosystems I'm using - they all just Work=E2=84=A2.

The issue is most likely the ™-character at the end of the message.

[pushcx]

Well, and the second bug is that you're getting emailed an exception instead of me getting emailed the exception and you getting nothing.

Thanks for reporting this with clear repro steps. This is some pretty infrequently-touched code, so I can't immediately think of a recent change that might've prompted it.

[oscarcarlsson]

I've seen this issue for at least a year, but it's rare that I include non-ascii so I haven't reported it until now. :)