Provide an example post-digest hook that reformats the content-type: multipart/digest format into an html-format digest

[Aanok]

EDIT: this issue was reused for #173 (comment) and below, feel free to skip the top

Hello!

Suddenly last week, rss2email has started sending malformed digest emails.

On Protonmail, they have the expected subject but they all appear with the following body:

No valid message body or attachments were found in this email. Please check with the sender to ensure that they are sending valid emails. The raw message is attached.\n\n

And the attachment is like so: message.eml

If sent to Gmail, they're rendered with an empty body and the individual messages split up in multiple attachments instead of a monolith.

I've reproduced this on Debian 10.8 (stable) and Arch Linux, with rss2email versions 3.9 (apt) and 3.12.3 (tarball), with sendmail and smtp configurations (although both pointing to the same Gmail account).

This is my config file: rss2email.cfg

Very importantly: rss2email can send non-digest emails perfectly well.

Regardless of this issue, thank you for rss2email, I've been using it very happily for a fair few years now :)

[Ekleog]

Hmm… do you maybe have a .eml file for something that displayed correctly before last week?

Out of the blue, my guess would be that it's Protonmail that changed the way it displays emails that consist of a list of attachments, especially as if I understand correctly it happens both with 3.9 and 3.12.3: to the best of my knowledge we haven't changed the digest format recently. (And there were 7 years of development between 3.9 and 3.12.3.)

That being said, it is very possible that since 3.9 (which was released in 2014) things have changed, but if it still does that to you with 3.9…

Happy to hear that you're liking rss2email! I myself joined the project rather recently, and people who wrote most of the code are no longer there, but I'll make sure to send good thoughts their way :)

[Aanok]

Hmm… do you maybe have a .eml file for something that displayed correctly before last week?

As far as I understand it, I can get an export from Protonmail but it will be an export of the whole message, with attachments and everything.

So here's the whole message whose attachment only I'd previously linked in the OP:
digest for archlinux 2021-03-21T21_58_30+01_00.eml

And here's a whole message from an older digest from the same feed, back from when it worked fine:
digest for archlinux 2021-02-20T06_30_23+01_00.eml

Out of the blue, my guess would be that it's Protonmail that changed the way it displays emails that consist of a list of attachments, especially as if I understand correctly it happens both with 3.9 and 3.12.3: to the best of my knowledge we haven't changed the digest format recently. (And there were 7 years of development between 3.9 and 3.12.3.)

To be clear: it's worse with Protonmail since it collates everything into a single attachment, but sending to a Gmail inbox shows the same problem. Gmail is just more clever with how it handles the attached messages, putting each one in its own attachment and being able to render their HTML content if you click on them.

Things look like this:

(clicking the attachment in Protonmail simply downloads it)

That being said, it is very possible that since 3.9 (which was released in 2014) things have changed, but if it still does that to you with 3.9…

Yeah, I'm not sure what could have been the trigger. Since I'm using a Gmail SMTP server to send the digests, maybe Gmail changed something on their end? I'll try with another SMTP service when I get a moment.

Happy to hear that you're liking rss2email! I myself joined the project rather recently, and people who wrote most of the code are no longer there, but I'll make sure to send good thoughts their way :)

Well, thanks for picking up the torch then :)

[devilgate]

Just to give you another data point. I just installed rss2email today, on Ubuntu 20.04. Here's the details:

$ r2e --full-version
rss2email 3.11
python 3.8.5 (default, Jan 27 2021, 15:41:15) 
[GCC 9.3.0]
feedparser 5.2.1
html2text (2020, 1, 16)

I don't get a message about 'No valid message body...', but I do otherwise get the experience described above. That is, each digest is an email with a set of .eml. attachments. My main email client, MailMate on the Mac, shows the digest as a single message, with the parts being separated by new from:, subject:, etc, headers.

But in Apple Mail it's shown as a single message with a set of attachments. in Gmail it looks just like the example above.

If it's relevant I'm sending using Gnu Mailman. That is, rss2email is configured to send to a Mailman list address.

[Ekleog]

@Aanok The “working” link you sent was generated by rss2email 3.9, would you happen to have a working attachment generated by rss2email 3.10 or later? (there were 5 years of development between 3.9 and 3.10 so it's hard to compare)

Also, to me it looks like this “working” email you sent actually contained a single email, and not a digest of all new RSS entries, which would hint at it actually not working…?

---

Overall, I've checked the generated output, and it looks correct to me, and the issues in display might be due to the clients supporting Content-Type: multipart/digest (defined in RFC1341) in a weird way — but GMail's way of displaying the thing looks legitimate to me from what I understand, as well as MailMate and Apple Mail from what I understand.

The only client whose behavior looks definitely wrong to me is protonmail, the other choices look more like each client deciding to display a digest as they each see best fit. Does that make sense?

[devilgate]

@Ekleog I take your point, if that's what 'digest' means. In other words, if a digest consists of a set of emails (.eml files) attached to a containing email. As you say, each client decides how to display that combination.

I'm not sure that's what people would want, though, or what they'd expect by the term 'digest'. It's certainly not what I want, nor what I expected. That would be a single email containing each of the new entries in the RSS feed, separated by their titles. Or just by a blank line if the feed entry has no title.

It's a long time since I used an old-school mailing list in digest mode, but I think that's what they did. I could be wrong, though. Maybe they sent multiple attachments and the mail clients processed them properly.

Either way, could we turn this into a feature request to get an option to work they way I want?

[Aanok]

@Aanok The “working” link you sent was generated by rss2email 3.9, would you happen to have a working attachment generated by rss2email 3.10 or later? (there were 5 years of development between 3.9 and 3.10 so it's hard to compare)

I'm afraid not :( 3.9 is what is on Debian stable, which is what runs on the machine where I normally keep rss2email in a cron job. All of a sudden it stopped working correctly, and as of now I cannot get any version to work as it used to.

Also, to me it looks like this “working” email you sent actually contained a single email, and not a digest of all new RSS entries, which would hint at it actually not working…?

Good point, I chose a bad example. Here is a working digest message:
digest for ottonieri 2020-12-31T06_30_43+01_00.eml.txt

And here is a screenshot of how digests used to be rendered, i.e. exactly like @devilgate is suggesting:

Overall, I've checked the generated output, and it looks correct to me, and the issues in display might be due to the clients supporting Content-Type: multipart/digest (defined in RFC1341) in a weird way — but GMail's way of displaying the thing looks legitimate to me from what I understand, as well as MailMate and Apple Mail from what I understand.

The only client whose behavior looks definitely wrong to me is protonmail, the other choices look more like each client deciding to display a digest as they each see best fit. Does that make sense?

It's entirely possible Protonmail changed something on their end! As a matter of fact, I would imagine the problem to be either on the receiving (Protonmail) or sending (Gmail SMTP) side, considering neither rss2email nor my system changed at all underneath. I don't mean to waste your time, but I figured you could at least use being notified if there were some providers playing badly with the program.

I'll try and find time to change my SMTP server tonight and repeat a test so we can maybe rule out one of the two sides :)

[Aanok]

There we go: tried sending the digests via Yandex, same result.

Incidentally, tried sending TO Yandex, it behaves like Gmail with a list of attachments.

[Ekleog]

@devilgate Hmm… I'm thinking the best path forward might be to provide an example of a digest post-process hook that allows one to do that from the configuration file, maybe without having an explicit option for that because the multipart/digest format is the comprehensive one. What do you think about this?

@Aanok So what I'd think would be that 3.9 always worked with protonmail and 3.12 never worked. If you're a paying customer you can maybe try to ask them if they can implement support for the multipart/digest MIME type; if not you'll probably have to rely on the above-mentioned workaround. Does that sound reasonable to you?

[Aanok]

I would put my money on both 3.9 and 3.12 working until Protonmail changed something mid March... but not a lot of it :P

As you've checked r2e is doing everything by the book as per RFC's, then yes, that definitely seems like the cleanest solution and would be perfectly fine with me.

[Ekleog]

Got it, renamed the issue — that said I must say that I personally barely have the time available to answer issues and review PRs, so this issue will definitely be “help needed”, sorry about that! :)

[squeakbat]

I'm currently using a hook to do just that. It's pretty ugly though, because feed._end_digest() expects a certain level of nesting. So it just converts each message/rfc822 part into a multipart/mixed.

I would also like to add that while the message format is strictly to spec, it's not practical if hardly any mail user agent can display it. And then there's no way to predict or control how it'll look. A single-part message gives the user some control over rendering via the css option.

Please also see discussion at #194.

[auouymous]

Does the now merged #221 fix this issue?

[squeakbat]

Not strictly speaking (the title says "an html-format digest"). However, in practice #221 is probably good enough.

To summarize for everyone else, add "digest-type = multipart/mixed" to the config file to get digest messages in multipart/mixed format, which is better supported.

It'll take a little bit of code change to do what the title asks for, because Feed._send_digest() wants to see the message in a particular format, so flattening it in a digest-post-process function will break it. Having said that, I would suggest that _send_digest() should knock it off. It's only extracting the date from the last digest part, which can and should be done before postprocess. If nothing else, setting the date has nothing to do with sending the message.

[devilgate]

I've added the new digest-type = multipart/mixed to .config/rss2email.cfg. And I cloned the latest version and ran the installer. r2e --version reports rss2email 3.14.

But when I run the command I get [ERROR] extra configuration key: digest_type.

I've obviously missed something, but what?

[devilgate]

Never mind, I had an underscore instead of a hyphen!