Fix #10282 "Campaign Response by Recipient Activity" (Status page) ha…

[chris001]

…s recently become unreliable. And others.

With this I implement RFC8085, "Signaling One-Click Functionality for List Email Headers" to restore the accurate working of the SuiteCRM opt-out feature for Newsletters and Email Campaigns, by preventing email services from automatically opting-out subscribers by crawling links in the email message, which before this Fix, used to load the opt-out link, and perform the opt-out.

All major email services (Microsoft, Gmail, Yahoo, etc) are crawling the links in email messages, to protect users from malware, and that crawling has been triggering the unsubscribe ("opt-out") feature, the "Remove Me" link in the footer of every Campaign and Newsletter bulk email message.

The new way, required in 2024 for all mail sent to Microsoft, Gmail, and Yahoo (and many, many more cloud hosted email services), is for senders to add new headers in the email, telling the email client app exactly which URL to request, with a special POST action that crawlers don't use, to enable the user to do a one-click unsubscribe ("opt-out"), from inside their email app, for this subscriber, for this particular email Campaign or Newsletter. So the subscriber no longer has to leave their email client app, and visit the Remove Me web page on the CRM.

Description

1. Improve reliability in detecting the web app's URL, in case the admin has left 'site_url' blank. This is required for constructing the opt-out URL.
2. Fix some missing global statements for $log and $mod_strings.
3. Accurantly construct the headers required for RFC8085 one-click unsubscribe.

Motivation and Context

The top Email services have been crawling all the links in the Campaign and Newsletter emails for a long time. One load of the opt-out (unsubscribe) link was actually unsubscribing every single lead or target who received the email, before they could even open it and read it!

How To Test This

Without this Fix, try sending a Campaign or Newsletter to only gmail, yahoo, or microsoft hosted email address. All of your emails will unsubsribe immediately. That's the automated link scanners loading the opt-out link to check for malware!

Apply this Fix. Send a Campaign or Newsletter to only Gmail, Yahoo, and Microsoft hosted emil. The headers for RFC8085 should be present, and none of the recipients will unsubscribe until they have a chance to open up the email message and look at it. In the email app, the subscriber will see a new "Unsubscribe" button, located near the headers. Clicking on this new "Unsubscribe" button should perform the opt-out from the SuiteCRM Newsletter or Email Campaign, from inside the email app.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Final checklist

• My code follows the code style of this project found here.
• My change requires a change to the documentation.
• I have read the How to Contribute guidelines.

[SinergiaCRM]

I have tested your code and I think I have found a mistake.
On RemoveMe.php you only accept POST requests now, but the actual link which is also included on every campaign email sent does not make a POST request.

[chris001]

Thank you for testing and finding that! Yes, it's allowing only POST to unsubscribe. To re-enable the link in the email unsubscribe working, I'll update the code to, when a user (or bot) goes to the normal web link, it will show the user (and the crawler bot) a page with an Unsubscribe button. When they click it (which will do the POST) it will unsubscribe the user. The anti-malware crawler bots will not click that button, they only scan page content, sometimes follow normal links.

[chris001]

@SinergiaCRM Updated. Please test again both methods of unsubscribe - click "unsubscsribe" in header, and click "remove me" link at end of email body.

[SinergiaCRM]

Hi,

now the link included on campaigns works like a charm.
Nevertheless, I have not been able to make the "cancel subscription" button that gmails adds automatically.
The POST is received but it seems that information sent is not the expected.
Clicking the "old" link (the one automatically included by SuiteCRM) I receive as postdata: entryPoint=removeme&identifier=6e70adcf-5972-fb71-d812-65b4c67c6127&List-Unsubscribe=One-Click
but clicking on google button I only receive
List-Unsubscribe=One-Click.

Headers included seem to be ok (in fact the gmail button is showing):
List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: <https://XXXXXXXX.org/index.php?entryPoint=removeme&identifier=6e70adcf-5972-fb71-d812-65b4c67c6127>

Besides a copuple of comments:

1. You are using hardcoded literals. It would be great if they could be personalized and / or internationalized.
2. In a future it would be great if that page could be personalized as well (Organization logo, etc)
3. When building the UnsubscribePOSTURL, probably using $sugar_config['site_url'] as being done in EmailMan.php would be better

[chris001]

OK @SinergiaCRM thanks for your feedback! I simplified the code, both methods of unsubscribing should be working now (click on New URL on Header of email, and click on Classic Tracker URL on Footer of email).
Answering your concerns:

1. Hardcoded literals. Yes, when both methods of opt-out are working 100%, I'll add internationalized strings, and basic .tpl page for the classic Tracker URL unsubscribe.
2. Logo on page: Will use the uploaded organization logo from Admin. Custom .tpl page with logo, if it exists.
3. UnsubscribePOSTURL. Good idea. I'll update this.

Please test this update, and post your feedback!

[SinergiaCRM]

Hi @chris001 ,
I tested your changes. Seem to work except for one thing: the UUID validation.
My IDs dis not pass the validation and they are correct (at least they have been automatically generated by SuiteCRM).
Example: 1d13c02f-3785-45df-2b3b-65b7eaf9b5fb.

[chris001]

Thanks for checking @SinergiaCRM this sounds like a bug, I'll take a look.

[chris001]

@SinergiaCRM I added a fix, it should now recognize the UUID. Please try, and post back your results!

[SinergiaCRM]

Hi @chris001 ,

both methods seem to work properly with gmail.

[SuiteBot]

This pull request has been mentioned on SuiteCRM. There might be relevant details there:

https://community.suitecrm.com/t/campaign-email-view-click-immediately/91992/4