Move FxA Activity data from a view to the queue

[pmac]

The Queue of events we get from FxA includes login events that have the same info as the posts we get from FxA's basket proxy to /fxa-activity/. We should consume these events instead of having FxA send us these POST requests.

NOTE: These posts are not idempotent, so we need to take special care in the transition to be sure we don't duplicate or drop any data.

See the discussion in #35 (review) for reference.

NOTE 2: We need to be sure we're handling the extra subscriptions for unique welcome emails implemented in the basket-proxy:

https://github.com/mozilla/fxa-basket-proxy/blob/13ed0f5206f9150566135d3a0973ac358cf1c0f7/lib/events/index.js#L96-L111

https://github.com/mozilla/fxa-basket-proxy/blob/13ed0f5206f9150566135d3a0973ac358cf1c0f7/lib/config.js#L57-L69

[pmac]

cc @rfk

[rfk]

I'm assuming that calls to /fxa-activity/ are not idempotent, but is that true in practice? I have a vague recollection of there being an issue where we were missing some login events, and one of the theories was that they might be so close together that they were generating duplicate rows in the underlying data extension.

[pmac]

I'm pretty sure that's true. We just record whatever update we get for this one. I'm not sure it would matter much in the long run if there were dupes, but it's best kept to a minimum I'm sure.

[pmac]

Just spoke to @rfk and we've decided that the best path forward for doing this data cut-over is to use the timestamps on the queue messages. The procedure should be basically as follows:

1. Decide on a timestamp value before which everything will go over the basket-proxy, and after which everything will happen over the queue
2. Code that timestamp into the fxa-basket-proxy so that it will stop sending basket requests after that timestamp
3. Code that timestamp into basket so that it will start accepting these queue messages after that timestamp.
4. Profit

We'll need to get the work from 2 and 3 done first with the timestamp configurable on both sides, then choose said timestamp and deploy. I'll go ahead and get started on supporting these messages and only doing so after a specific value in the timestamp field.

[pmac]

Note: timestamp key in the message will be ts and will be a float in seconds since unix epoch.

see mozilla/fxa-auth-server#2168

[rfk]

I think the timestamps should be live in the event stream as of now, are you able to confirm from your end?

[pmac]

@rfk confirmed! I see the following in a message on the queue:

{
  "ts": 1508897239.207,
  "event": "login"
}

Obviously that's a snippet of the message, but it looks correct and is all I need to go on. Thanks!

[rfk]

WIP on the basket-proxy-side killswitch in https://github.com/mozilla/fxa-basket-proxy/pull/48/files

[pmac]

@rfk I think we're both ready now. Just need to decide on a reasonable cutover timestamp and we can both deploy. I don't have the release trains and complexities like you, but I also don't know if they apply to the basket-proxy service. So please let me know when you think your work might hit prod and I'll be ready.

[rfk]

@pmac side note, any chance I can get added to the mozmeao org in some capacity? The notification emails from this org are going to my personal email, which means I regularly forget about them during work hours :-(

Github apparently lets me choose the notification email address on an org-by-org basis, but I can only see how to do it for orgs that I'm a member of.

[pmac]

@rfk I've invited you to be a collaborator on basket. Hopefully that also invites you into the org. Let me know if that isn't sufficient.

[rfk]

That did the trick, thanks!

[rfk]

The work on our side should go out next week, in train-99: https://bugzilla.mozilla.org/show_bug.cgi?id=1413334

Can we QA your changes here in dev or stage to ensure we've confident of doing the right thing? For example, perhaps we should simulate the cutover in stage on both sides and see whether it all works correctly.

[pmac]

Definitely. Just give me a cutover value and we'll give it a try.