Bounce processing

[roshanjonah]

I see a lot of you guys have Listmonk implemented in a production env. I was wondering how you guys process the soft and hard bounces and also the complaints.

[sweetppro]

I use Amazon SES which forward the bounces/complaints to an email address.

• I then dump them all in an imap folder.
• then use an email scraper to grab all the email addresses.
• I then create an csv file, and use a random name for the name column (list monk doesn't seem to mind and parses subscribers by email)
• then I import and blacklist the csv file

[knadh]

@sweetppro that's quite cool! Hopefully we'll have bounce tracking support built into listmonk soon enough.

[roshanjonah]

@sweetppro Yes good stuff. I guess we could try something similar. Thank you for that :)
@knadh That would be amazing!!

[roshanjonah]

@knadh

1. Just building Amazon SES API integration with SNS passing JSON notifications and the system marking those subscribers as blacklisted. How are you envisioning the bounce processing system so I can follow the style and it won't be hugely different when it's released.

2. Also re duplicate subscribers...if the duplicate email is not allowed to be added again, what's a better way to store all the "lists" they are part of - can we do tags? Because generally speaking people can sign up for a monthly newsletter, product update newsletter, and so on right. Currently, if I add and the subscriber exists, its showing error although I am asking it to add to a list which it's not associated with.

3. And is there a way to skip validation of name because someone can sign up with just their email and update their profiles later or we add them later.

Thank you :)

[knadh]

1. If it's a postback mechanism like you've mentioned, that would be straight forward to implement. Offload the headache of reading inboxes, parsing messages and processing bounces to an external system that'll just notify listmonk of bounces. If this is has to be built natively into listmonk (with IMAP and POP support), it is going to be quite complex. Have to research and figure out if there are 3rd party tools available that do this and emulate the SES behaviour you described.

2. The subscribers table contains one row per subscriber e-mail, while the subscriber_lists table contains one to many (subscribers.id -> lists.id) relationships representing subscriptions. That is, a subscriber's profile is unique but they can subscribe to any number of lists.

3. No way to skip the validation, but you could just pass the e-mail as the name internally.

[roshanjonah]

1. Yup doing this to take away the headache of IMAP and POP ;) - My thinking so far is SNS sends JSON, parse for subscriber email, blacklist for complaint, and bounces - still thinking how to record for soft bounces so in the future can program for the tolerance of the number of soft bounces allowed.

2. For some odd reason, if I am adding a subscriber manually to a list which they are not subscribed to, it still errors out. Just says "The email already exists!" instead of adding the subscriber to the new list they now belong to. Is that a bug, would you say?

Thanks Kailash! You the man!

[knadh]

For some odd reason, if I am adding a subscriber manually to a list which they are not subscribed to, it still errors out. Just says "The email already exists!" instead of adding the subscriber to the new list they now belong to. Is that a bug, would you say?

Adding should throw this error. To subscribe an existing user to new lists, the action should be "edit" (PUT if you're using the API). But I see your point. There's an additional step of checking if a subscriber exists before deciding between add or edit. Maybe it should be upsert by default where it either adds or edits in place. Let me look into this.

[roshanjonah]

We are trying to develop bounce processing for SES and it passes JSON notification via SNS to a webhook we provide. In developing this webhook, we realized there is no way to add a subscriber to the blocklist without their ID, so I am thinking to add custom headers for this type of SMTP where it would insert the subscriber ID in the message headers. Most systems that pass complaints and bounces work this way where they return message headers along with the bounce/complaint types. This allows us to parse the message header and make necessary adjustments in our database. So what are your thoughts @knadh - Would this be how you have envisioned the bounce system to work in the future?

Suggesting the following items to be considered for inclusion in the message headers of an email so that the system is ready for any sort of bounce processing via API:

• Campaign ID
• List ID
• SMTP Server ID

[knadh]

That's correct. v0.7.0 added support for the List-Unsubscribe header. Similarly, listmonk should start sending campaign, list, and subscriber UUIDs in the header.

However, the current APIs work with numerical IDs, but it's not ideal to expose these. The internal queries and APIs need be tweaked to work with both IDs and UUIDs. I'll explore this.

[danilobuerger]

@knadh how does the List-Unsubscribe interact with external systems? Ref https://listmonk.app/docs/external-integration/ how would the external system know someone unsubscribed?

[knadh]

@danilobuerger sorry, missed your comment. List-Unsubscribe is to indicate to e-mail clients that there's an unsubscription option available. This is what shows the Unsubscribe button on the Gmail UI, for example.

[knadh]

I've been working on this and bounce processing will be available in the next release.

• Connect to a dedicated IMAP inbox to read bounce mails (partially working).
• Connect to a dedicated POP inbox to read bounce mails.
• Explore VERP.
• Expose a generic webhook endpoint to which bounces can be piped or posted. For instance, for local Postfix instances, you could parse local mail text and pipe the bounces to this endpoint.
• Expose ready-to-use webhook endpoints for popular services like SendGrid, SES etc.
• Settings on the UI to control the behaviour of bounce handling. eg: On X bounces [unsubscribe|blocklist|delete] subscriber.

If there are any suggestions, please share.

[danilobuerger]

@danilobuerger sorry, missed your comment. List-Unsubscribe is to indicate to e-mail clients that there's an unsubscription option available. This is what shows the Unsubscribe button on the Gmail UI, for example.

@knadh thanks for getting back. What I meant was: The List-Unsubscribe is probably populated with an url thats internal to listmonk. How does my non-listmonk system know that the user followed the unsubscribe link?

[knadh]

How does my non-listmonk system know that the user followed the unsubscribe link?

1. You can synchronize with the listmonk subscription table periodically. The table structures are really simple.
2. listmonk could have a webhook on events like subscription/unsubscription. Will have to plan and build this.

[knadh]

WIP

[c-nv-s]

Just wanted to mention that a general webhook for when users simply subscribe could also be a very useful addition.

[knadh]

Update: Bounce processing is now functional. Should be release ready in a week or so.

[cavnit]

I've been working on this and bounce processing will be available in the next release.

• Connect to a dedicated IMAP inbox to read bounce mails (partially working).
• Connect to a dedicated POP inbox to read bounce mails.
• Explore VERP.
• Expose a generic webhook endpoint to which bounces can be piped or posted. For instance, for local Postfix instances, you could parse local mail text and pipe the bounces to this endpoint.
• Expose ready-to-use webhook endpoints for popular services like SendGrid, SES etc.
• Settings on the UI to control the behaviour of bounce handling. eg: On X bounces [unsubscribe|blocklist|delete] subscriber.

If there are any suggestions, please share.

If you are using AWS SES for sending, you can use SES Configuration Sets & SNS web hooks to process bounce messages, I have it implemented in another project and it works extremely well, you also get info on the type of bounce, whether it is permanent or transient, etc. You can also get send/open etc, as well.

Happy to contribute the SNS validation/subscription and parsing code for inclusion, as well as SES Configuration Set details to anyone who needs/wants it. Works with SES SMTP as well

[knadh]

Happy to contribute the SNS validation/subscription and parsing code for inclusion, as well as SES Configuration Set details to anyone who needs/wants it

Yes sure, that would be helpful. Support for webhooks is already in place. SES and SendGrid webhook parsing have to be plugged in.

[cavnit]

SNS calls are made to a single endpoint with:

• Subscribe
• Unsubscribe
• Notification

Subscribe is called when you setup the endpoint in SNS, to confirm the subscription and activate the call in sns,
Unsubscribe is the reverse, though I have never used it.
Notification contains the actual details of the SES Sending

• Bounce
• Click
• Delivery
• Open
• Reject
• Send

In order for the SNS calls to be enabled you need to add a Configuration Set to SES and configure which notifications you want to receive, and then when you send each email, you need to add a Mail Header with the configuration set.

I will send a pull request once done, and then look at documenting it all as well.

[knadh]

Please don't send a PR to the bounce branch yet @cavnit. The branch is still WIP and I already have Sendgrid and SES postback logic partially done. Also, the branch's head is regularly updated and force pushed. Will update here once the loose ends are all wound up.

[knadh]

I've pushed the SES webhook processing WIP here:

listmonk/cmd/bounce.go

Line 219 in 936dd22

func parseSESNotif(c echo.Context, app *App) (models.Bounce, error) {

The SES implementation is based on this doc. The doc doesn't mention anything about an SNS envelope or validation though.

[cavnit]

To setup the sns web hook you need too confirm the subscription, you don't need to validate the call, but then anyone can make calls to the web hook.

https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html

The validation just makes sure that the incoming post request is actually from AWS SNS

[cavnit]

There is no need for an auth key, AWS signs all the requests you just validate the signature on the payload, putting the auth key in a query string leaves it open to abuse in my opinion. Validating the payload is pretty simple,

[knadh]

The validation just makes sure that the incoming post request is actually from AWS SNS

Yep. Was looking for this. The SES notification docs make no mention of SNS signatures for validation.

[knadh]

@cavnit that was very helpful! The signature verification would've otherwise been painful to figure out. SNS/SES confirmation and bounce processing are now fully implemented:

listmonk/internal/bounce/webhooks/ses/ses.go

Line 21 in 45ffc17

// AWS signature/validation logic borrowed from @cavnit's contrib:

Took me a while to figure out as it wasn't apparent in the SES notification documentation.

• SNS topic is created and a "subscription" (URL of the listmonk webhook endpoint) is added. SNS sends a SubscriptionConfirmation notification with a subscription URL that should then be retrieved with a GET request. Without this, delivery events are not posted.
• SES notifications are connected to the SNS topic and delivery events start getting posted to the same webhook endpoint.
• All these messages have to be validated with Amazon's certificate specified in the SigningCertURL in the payload. To avoid downloading certs on every event request, they are downloaded once and cached in-memory.

[cavnit]

Thats great, the only other part that might be missing is the "Configuration-Set" if you want to get send, delivered, click, open stats, as you need to set that when sending the email in the header, but it is probably better for a separate issue.

[knadh]

I think the bounce branch is ready for production. I'm in the process of writing integration tests now. It supports:

• scanning POP for bounce messages. IMAP is not included in this version as it is (unsurprisingly) problematic. Also, bounce mailboxes are not generally IMAP so it seems like a fair trade-off for now.
• Internal API for recording bounces (/webhooks/bounce).
• Public SES/SNS bounce notifications endpoint (/webhooks/service/ses).
• Public SendGrid bounce notifications (signed) endpoint (/webhooks/service/sendgrid)
• UI: Settings -> Bounces. Subscribers -> Bounces. Individual subscriber modals.

@roshanjonah if you could test the bounce branch with your working SES environments, that would be very helpful.

[roshanjonah]

@knadh Sorry it took so long - got caught up with work. I can't seem to get it working because I am getting this error. I ran DB upgrade too. Looks like a table is missing?

2021/07/29 10:56:08 init.go:196: connecting to db: maindbxsrv:5432/domain
** IMPORTANT: Take a backup of the database before upgrading.
continue (y/n)?  2021/07/29 10:56:08 upgrade.go:58: no upgrades to run. Database is up to date.

2021/07/29 10:53:00 main.go:86:  (#cdd8037 2021-07-29T10:51:49+0000)
2021/07/29 10:53:00 init.go:105: reading config: /home/domain/orgx.domain.com/config.toml
2021/07/29 10:53:00 init.go:196: connecting to db: maindbxsrv:5432/domain
2021/07/29 10:53:00 init.go:224: error preparing SQL queries: Error preparing query 'get-campaign-stats': pq: relation "bounces" does not exist

[knadh]

@roshanjonah sorry, been busy myself. There is a new bounces table and a new bounce.* keys in the settings table.

[knadh]

This is now merged into master 1be8c7d

[MaximilianKohler]

I don't understand how this works. Once again, your docs https://listmonk.app/docs/bounces/ aren't very helpful.

I checked "enable bounce processing"; "bounce count =1; blocklist".

I checked "Enable bounce webhooks" even though I don't understand what it does.

I'm using Amazon SES so I checked "enable SES".

I sent a test campaign to:
bounce@simulator.amazonses.com, complaint@simulator.amazonses.com

But it didn't seem to register any bounces, and come to think of it, there's nowhere that "complaints" are mentioned/tracked either.

"Enable bounce mailbox", don't understand it. "SMTP server's host address" so something from Amazon SES? But it's equating it to "pop.yoursite.com" which is very confusing. I know gmail has pop settings, but that's it.

The docs suggest that if my "from" email has a POP3 mailbox behind it (it should; it's a gmail) then it can receive bounce e-mails. But I didn't get any bounce email from sending the campaign to bounce@simulator.amazonses.com.

I looked up more bounce emails https://stackoverflow.com/questions/17699307/service-for-testing-bounced-email-handling and tested/confirmed that bounce-test@service.socketlabs.com is working as a bounce email. I sent a campaign to it; 0 bounces, and 0 emails to the "from" email notifying about bounces.

---

EDIT: see solution a few comments down.

[cavnit]

@MaximilianKohler You need to setup SNS to send the bounce details, at the bottom of the https://listmonk.app/docs/bounces/ page look at the part on "External Webhooks" you will see the first entry describes how to set it up for SES

[MaximilianKohler]

@MaximilianKohler You need to setup SNS to send the bounce details, at the bottom of the https://listmonk.app/docs/bounces/ page look at the part on "External Webhooks" you will see the first entry describes how to set it up for SES

Did not seem to work.

SNS subscriptions https://user-images.githubusercontent.com/3606996/204659010-dc872421-9af4-41ca-979c-1ea841a5a3c8.jpg

Verified identities https://user-images.githubusercontent.com/3606996/201532497-6536cd13-720f-44b0-8a7e-e2b390a94782.jpg

Domain https://user-images.githubusercontent.com/3606996/204658897-8dfb1137-47d5-4866-a544-b7f35462ad45.jpg

Moreover, I see in the AWS console (suppression list), that bounce-test@service.socketlabs.com has already been blocklisted. When I send a campaign to them it would be important to know that it was never delivered to them since they are blocklisted.

Also, if Sendy can implement it automatically it should be easy for Listmonk to do so as well.

---

For https://listmonk.yoursite.com/webhooks/service/ses, if I'm running listmonk on a subdomain (https://listmonk.mydomain.com), presumably I use https://listmonk.mydomain.com/webhooks/service/ses not https://listmonk.listmonk.mydomain.com/webhooks/service/ses ?

EDIT: Nope, https://listmonk.listmonk.mydomain.com/webhooks/service/ses does not work at all. Given that the SNS status of the Listmonk subscription I created is "Confirmed", it seems like it should be working properly.

[cavnit]

Feel free to use Sendy then if it better meets your needs

[MaximilianKohler]

Feel free to use Sendy then if it better meets your needs

It doesn't; that's why I'm trying to test Listmonk. I only mentioned Sendy because when I went into those AWS settings I noticed that Sendy had set it up automatically.

[bamboowonder]

@MaximilianKohler ,I doubt the tone of your initial post inspires any of the volunteers on here to want to help. This software is free. If you are having trouble, ask nicely, or better yet offer to improve the documentation. The sole developer also does this in his free time. Sendy is paid software, so that dev has made some of the complicated parts easier by utilizing Amazon's api to setup all the bounce processing.
It is a pain, and complicated to get all the moving parts of SNS, SES and Listmonk setup. Then on top of that having a secure proxy in front. The documentation assumes that anyone brave enough to set up their own email delivery system has patience, some solid background knowledge etc etc.
From your post it sounds like the issue is at the SNS/SES level.
If I remember correctly when I set it up, until SNS had completed it's verification that the webhook was functioning, it stayed in pending. I realized that my proxy wasn't allowing the SNS notifications to reach Listmonk.

[MaximilianKohler]

I am indeed creating a proper guide/documentation while I spend weeks trying to set this up (I'll go post it in #120). I am frustrated because I know very well that the people creating this project could have typed out the directions in a few minutes. But they make everything so vague, so that only extremely knowledgeable people can use this, which is totally unnecessary.

I see this project has dozens of contributors spending years creating it, then no one bothers to create directions simple enough for the average person to be able to follow.

If I remember correctly when I set it up, until SNS had completed it's verification that the webhook was functioning, it stayed in pending. I realized that my proxy wasn't allowing the SNS notifications to reach Listmonk.

In my screenshots, mine says verified (and it was verified pretty much immediately.

[knadh]

I am frustrated because I know very well that the people creating this project could have typed out the directions in a few minutes.

Sorry @MaximilianKohler, that would be a misconception. I've no experience with SNS/SES apart from setting them up once (which I remember struggling with) for testing the feature while I was developing it. Never found the time to study and understand it and to document it properly either. If you manage to do this, please do contribute to the docs.

[MaximilianKohler]

Sorry @MaximilianKohler, that would be a misconception. I've no experience with SNS/SES apart from setting them up once (which I remember struggling with) for testing the feature while I was developing it. Never found the time to study and understand it and to document it properly either. If you manage to do this, please do contribute to the docs.

Damn :(

Well... you can see in my screenshots how another similar program did it automatically, and split it up into "bounces" and "complaints". Perhaps that's also why Listmonk can't distinguish between bounces and complaints?

I don't know how to contribute to the docs, but I can leave details in the issues here.

I found a "workaround":

AWS keeps its own "suppression list": https://us-west-1.console.aws.amazon.com/ses/home?region=us-west-1#/suppression-list

Only way to export it is with the command line interface (CLI): https://docs.aws.amazon.com/ses/latest/dg/sending-email-suppression-list.html#sending-email-suppression-list-view-entries - outputs in javascript.

Save output to clipboard:
aws sesv2 list-suppressed-destinations|clip - requires sudo apt install geomview (200MB)
or file:
aws sesv2 list-suppressed-destinations > filename.txt - then have to download the file somehow (can view with ls), or output into pastebin:
cat filename.txt | curl -F 'clbin=<-' https://clbin.com/
Then:
Convert javascript array to CSV: https://www.convertsimple.com/convert-javascript-array-to-csv/ - delete the outmost curly brackets.

---

This doesn't seem like a fully functioning workaround, because when I opt for SES bounce email notifications, I get more notifications than emails-added-to-the-bounce-list. I would suspect that those extras would be caught with Listmonk, but not sure.

[MaximilianKohler]

Maybe this is why it doesn't work?

https://aws.amazon.com/premiumsupport/knowledge-center/ses-email-opens-clicks/

To apply the configuration set that you created to your email, you must pass the configuration set in the headers of your email

[MaximilianKohler]

Solved:

I found the answer on the Mautic issues mautic/mautic#7497 (comment)

Don't enable the raw message when you are creating the subscription with HTTP/S.

I also tested cyberphen's solution of "applying the notification settings to the 'from email' as well as/instead of the domain", and that seemed to have no impact. The "from email" seems irrelevant.

There are also other necessary steps I found and added to my guide.

The Mautic docs are erroneous, outdated, and incomplete as well. I was able to piece together most things in the guide I made. Feel free to update your docs with it, or even simply link to it. I don't know how to mess with the docs, and I don't know if you'd like my changes. But the Mautic docs shouldn't be the only/main reference there given how flawed they are.

Now that I've gotten most things to work, Listmonk is actually quite a good program. There are things that are even better than some paid alternatives. Thank you very much for making it free and open source.

The only disappointment is the fact that likely hundreds of people went through the same steps I did and not a single one bothered to leave instructions for others.

I initially immediately dismissed Listmonk as a viable option after looking at https://listmonk.app/#download and the linked docs. I think the stuff that's there right now is just gibberish to the vast majority of people. But with the right directions it should be relatively easy to set up for most people.

Also, from what I've been able to determine, Amazon SES only logs hard bounces in the suppression list. Soft bounces and domain errors are only logged via email notifications and webhooks. So it's probably not a great idea to rely solely on SES's suppression list. https://repost.aws/questions/QUFi5N0DVBRyqAioWQTXfuSA/se-ss-suppression-list-misses-many-bounces-that-otherwise-get-emailed-to-me-via-email-feedback-forwarding-do-i-need-to-worry-about-these-it-would-be-nice-if-there-was-an-option-to-catch-these-extras

Once Listmonk separates soft bounces to its own customizable category, that would be a nice improvement.

[Slach]

@MaximilianKohler could you make pull request with your instuctions to
https://github.com/knadh/listmonk-site ?

[MaximilianKohler]

@MaximilianKohler could you make pull request with your instuctions to https://github.com/knadh/listmonk-site ?

Sorry, I have no idea how. I took a look and it seems too complicated for me. If it was like editing a wiki I could do it, but I'm no coder.

I'll keep updating the guide-comment I made in the other thread with whatever I learn.

[Slach]

@MaximilianKohler just open
https://github.dev/knadh/listmonk-site/blob/master/docs/docs/bounces.md

[MaximilianKohler]

@MaximilianKohler just open https://github.dev/knadh/listmonk-site/blob/master/docs/docs/bounces.md

Cool. Done. Maybe if that's accepted I'll try more.

I was thinking another option would be to enable the wiki and create an "installation guide" page, and I could paste my guide in there.

[Slach]

yep, cool, looks great
https://github.com/knadh/listmonk-site/pull/38/files
I hope @knadh will accept this PR ;) thanks for your errort

[knadh]

Thanks for the PR. It's merged.