Unable to sign up for Test Pilot newsletter outside of US networks (Romania)

[SoftVision-EmilPasca]

[Affected versions]:

• Firefox 54.0.1 and above

[Affected Platforms]:

• All Windows
• All Mac
• All Linux

[Prerequisites]:

• The "testpilot.env" string type pref was manually added in "about:config" page with "stage" value.
• The "extensions.webapi.testing" boolean type pref was manually added in "about:config" page with "true" value.

[Steps to reproduce]:

1. Open the Firefox with the profile from prerequisite and navigate to https://testpilot.stage.mozaws.net page.
2. From the bottom of the page click on the text box for newsletter sign up.
3. Enter a valid mail, check the "I'm ok with..." checkbox and click the "Sign Up Now" button.
4. Open the email client and check for an email from Test Pilot.

[Expected result]:

• A confirmation email from Test Pilot was received on the email client.

[Actual results]:

• No confirmation email is received from Test Pilot.

[marsjaninzmarsa]

Confirmation email should be sent only if you're not yet signed for any of Mozilla Newsletters, are you sure that's not the case?

[fzzzy]

I see a POST to https://basket.mozilla.org/news/subscribe/ when the form is submitted, so either basket was having a problem or the email was already subscribed.

[SoftVision-PaulOiegas]

@fzzzy Don't think so, I had the same problem on my end. Also checked the subscription from an old test email and my testing email wasn't subscribed. After 2 attempts of subscribing, nothing came on that email.

[SoftVision-PaulOiegas]

I'll recheck with a brand new email address to make sure it's still broken.

[pmac]

I think basket is okay. Can someone point me to the code that sends the request to basket? I know that there have been issues with sending basket an invalid language. I filed #2403 to help deal with this, but it could well be something to do with that again. I tried to reproduce from the URL above but I apparently need some special Fx config that I don't have. Is this also happening in prod?

[pmac]

I tried prod with a test email and got the confirmation email. So things seem fine in prod at least.

[fzzzy]

Is the email perhaps being caught in a spam filter?

[SoftVision-PaulOiegas]

Sorry guys, just did a new Gmail email address (no spam filters) and tried with it on both Stage and Production servers. Tried both the methods, from homepage bottom form and from the welcome pop up displayed after installing Test Pilot. No newsletter email received... Checked all the folders and nothing.

[fzzzy]

I'm going to close this since the list is working, just not from the softvision offices apparently. Hopefully we can find a way (vpn?) that they can test this.

[SoftVision-PaulOiegas]

@fzzzy Somehow, both me and Cosmin forgot to test this at home after work (please don't ask 😄). We'll do this today after work.
However we have done some additional testing:

1. Tested the newsletter signup via mobile data connection using our phones.
   -This didn't worked, as no emails were received on our phones.
2. Using US random VPN in browser Network settings
   -This one worked and the email was correctly received by Cosmin.

Depending on the result we get after we test at home, this could still be a problem. If we receive the email at home, it could mean that Softvision IP is blocked. If we don't receive anything, this could mean that there is a wider filter/block. Since Test Pilot is not only for US people, this could end up as a problem.

We'll update you today on this matter.

[fzzzy]

@SoftVision-PaulOiegas Thanks. We need to figure out where to file this bug against the "basket" service we use.

[pmac]

You can file a bugzilla bug if you'd like.

[SoftVision-PaulOiegas]

Guys, I'm gonna reopen this since I haven't managed to sign up from home either. So this is something broader than the Softvision network. I'll rename the bug to reflect this.

We'll try with other VPN's too (Germany), in order to see if this only occurs for Romania or for other EU countries too. We only managed to sign up using US VPN.

[pmac]

Just to make sure, we're using new real email addresses for these tests right? I often use the restmail.net service for this kind of thing. You can have it email any-user-at-all@restmail.net (where "any-user-at-all" is literally any username) and then check that mail at e.g. restmail.net/mail/any-user-at-all@restmail.net. It will be a JSON representation of the email received. That should help narrow it down to something in the Salesforce system vs. something after email leaves salesforce.

[pmac]

I just ran another such test at https://testpilot.firefox.com/experiments and received the confirmation email.

[SoftVision-PaulOiegas]

Just to make sure, we're using new real email addresses for these tests right?

We only used real gmail and yahoo addresses for testing this. But as I've told you, with a set US VPN, everything was ok. But while on local IP's, no email came trough.

[pmac]

We only used real gmail and yahoo addresses for testing this.

And unique ones every time right? If the email has ever been confirmed before you won't get a confirmation email. I'm only continuing to ask because I can't find anything that would cause the difference depending on IP or geolocation. It shouldn't be Salesforce filtering because it doesn't get the end-user's IP, only basket does, and we don't do IP filtering.

[pmac]

And to be really clear, by "unique" I mean that things like "user+test1@gmail.com" and "user+test2@gmail.com" are unique and would be a good test.

[pmac]

Nevermind... I believe I've found the issue. Our basket instance in the AWS Frankfurt region was not scaled appropriately. So requests coming from europe were most likely hitting this instance. I've just scaled up the workers so all of the tests may start coming in now as they've been queued but not processed. Let me know. Apologies for the confusion, but thanks for staying on it.

[fzzzy]

@pmac Wow, thanks for looking into it more! Hope that fixes it!

[SoftVision-PaulOiegas]

And unique ones every time right? If the email has ever been confirmed before you won't get a confirmation email.

Sorry for late response but we are on +2 GMT (Romania), so sadly we were out of office when you replied.
To answer your question, we used both new and unique emails and also old testing emails which were unsubscribed after each sign up. If you unsubscribe from Mozilla newsletters, you can reuse the email address.

Nevermind... I believe I've found the issue.

I think you are right. Just used one of my old email addresses and successfully subscribed twice to the newsletter (with an unsubscribe between).
Good catch 👍 and thanks for insisting on this!

However, before closing this as fixed I want to clarify one thing. In the past, for the first subscribe attempt, the user received an email of confirmation. And for a 2-nd subscribe attempt (unsubscribe + resubscribe) on the same address, an email that promoted a couple of active experiments. Right now, even for 2 consecutive subscribes, I received the first confirmation email only and no sign of experiment promotion.
@johngruen @clouserw Was this intended due to the newly launched experiments and the reminder I logged #2594? Or we should consider this an issue? In the reminder you guys said that a change is prepared for 25th of July. Just want to confirm that this is the right behavior.

[ghost]

https://docs.google.com/presentation/d/1JaNxwBMu0_uYSt-PxDgRtBPfQXmPdeGa5-nIuacBOHg/edit#slide=id.g1fac4dc0b6_0_5 has what the email messages should be. There should be a confirmation and then a welcome email. Slides 6 & 7

[SoftVision-PaulOiegas]

@wresuolc double checked this and the following flow seems to be in place right now:

Brand new email address

1. Sign up for the first time -> You get the first email that requires confirmation.
2. Confirm the subscription -> You get the second email, the Welcome one.
3. Unsubscribe + re-subscribe -> You get the same Welcome email, not the 3rd one that promotes experiments.

Already used email that was unsubscribed

1. Sign up -> You get the second email, the Welcome one.
2. Unsubscribe + re-subscribe -> You get the same Welcome email, not the 3rd one that promotes experiments.

Already subscribed email

1. Sign up -> You don't get any email.

So, either the 3rd email was temporary removed from the chain in order to rewrite it to promote the 3 new experiments (#2594). Either we lost it somewhere along the way. I haven't checked on DE, but I'm pretty sure the EN-US localization had the 3rd email too, because I tested it a lot.

Is there a chance that fluent-react could have caused this?

[fzzzy]

I don't think anything in our codebase would have caused this. The email configuration is probably in some database related to the email list service.

[pmac]

Which emails to send is fully the responsibility of Salesforce. Whoever is managing things on that side is who would know. @jessilyndavis may know, or would at least know who would know.

[jessilyndavis]

Thanks @pmac!

@SoftVision-PaulOiegas

The flow you articulate is correct. There is not a 3rd email and you only get welcomed to the program once (this prevents someone from getting multiple welcome emails if a bot signs them up multiple times in a row).

There is only 1 welcome email that's set live. If you got a 3rd email in your testing, it is because you signed up right before a manual email was sent to the entire subscriber base.

If you want to have a 3rd email to be sent automatically after signup, a welcome series would need to be setup. I'm meeting with John on Monday to talk about the Test Pilot email program: https://bugzilla.mozilla.org/show_bug.cgi?id=1384730

I'm adding this github thread to that bug.

Thanks!

[SoftVision-PaulOiegas]

@jessilyndavis Thanks for the details! What I wanted to say is that we previously had a 3rd email. That one was sent to an user that already confirmed the subscription and received the 2nd email, but unsubscribed and resubscribed. The email promoted new experiments and other details.

Anyway, I think this will be covered in the Bugzilla issue you mentioned above.

What kind of relationships are you looking to develop?
We would like to send emails to our mailing list directly to keep participants up to date with new experiments, experiment updates etc.

Probably the email was removed since the content was no longer appropriate.

I'll leave this open until the bugzilla issue is fixed and retest this.
Thanks!

[SoftVision-CosminMuntean]

It seems that this issue is again reproducible, the newsletter confirmation email is not sent. I am not sure if this is reproducible only for Romania or for other EU countries too.
Also, I have used a US VPN and the confirmation email was sent.

[SoftVision-CosminMuntean]

Ciprian, Carmen and Emil just tested this issue and they cannot reproduce it. All of them received the confirmation email.
So probably my machine or my email accounts have some issues.

[ghost]

Thanks for checking again Cosmin (and everyone). Sorry your email is broken. ✉️