Push transaction data to SQS

[solarissmoke]

Push data about completed transactions to SQS.

Information required

1. API for pushing data to SQS.
2. What information should be sent and its structure.

Technical notes

My initial thoughts for implementation are that this should be done asynchronously via a worker process.

There is an open question about how much resiliency is required in terms of retrying failed push, and whether we need to store basic transaction data (a Braintree reference number) in Wagtail as a fallback.

[philwakefield]

What are the implications of having to store the Braintree ref no? Is there a data protection or security concern by having this in our system?

[solarissmoke]

I don't think that storing the transaction reference should be a problem - doesn't tell you anything about the transaction itself.

[patjouk]

1. Our current donate platform talks to SQS directly. You might be able to use boto3 for that. Doing it asynchronously is a good idea.

2. Basket has a donate process that looks for different event types: donation, crm_petition_data (@alanmoo do we need this one?), newsletter_signup_data, and DEFAULT (probably a catch-all, I'll ask around). I couldn't find a doc for the message structure, so I'm asking people around. Meanwhile, we can get a good idea of what basket expects from the code:

• donations
• DEFAULT

I'll add more if I manage to put my hands on some docs.

[solarissmoke]

Thanks. Yes, we will do this async.

I think we can probably pick out the data structure from the code you've linked to, but it would be helpful to have a written spec that we can refer to. If you don't find any docs then I will write up a spec based on the code.

[alanmoo]

I've got a call tomorrow morning (US) that touches on the approach for this, so hopefully I'll have more details by Thursday.

[alanmoo]

Lucie's got a PR open to document some Basket things, but for extra data that we're going to be sending from Braintree to Basket, you can define the keys that need to be added to the package and support for them will get added by the Basket team.

[solarissmoke]

Is there a possibility of setting up sandbox SQS/Basket instances so that we can test end-to-end data flows? Based on where we are currently, I should be able to have an initial pass at the SQS data push ready next week.

[alanmoo]

Is this still blocked?

[patjouk]

Yes, because there's the question of the "net amount" field. When we have an answer there, we will need to ask Pmac to do the necessary changes on basket side: I guess it will be easier to add a temporary event type so that both donate platforms can work side to side and do a switch on launch day.

[alanmoo]

Ok. I'm waiting to hear back from Braintree about that.

[alanmoo]

I think I've mentioned this elsewhere, but we're not going to be dealing with "net amounts" from Braintree, and the stakeholders are fine with that. Based on the API I spec'd out, we should be able to start passing data to Basket as it stands, and then improve Basket to be able to handle any new data that's coming in.

[solarissmoke]

Thanks. We should already be sending data to Basket (see #203 (comment)) but will need a bit of time/help from someone with access to the SQS queue and the Basket back end to help us verify that this is working correctly - we have no visibility on errors once the data is sent to SQS.

[alanmoo]

@cadecairos or @patjouk can one of you sync up with Samir to find a time for solving this?

[patjouk]

I've contacted Samir on Slack.

…

On Wed, Sep 11, 2019 at 11:24 PM Alan Mooiman ***@***.***> wrote: @cadecairos <https://github.com/cadecairos> or @patjouk <https://github.com/patjouk> can one of you sync up with Samir to find a time for solving this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADVOHBYSO2R6PDZYVJFWN3QJFO2LANCNFSM4H7L4U2Q> .

[solarissmoke]

I'm going to close this on the basis that we are now successfully passing data to SQS - let's open new issues for specific data structure issues after you've tested.