Skip to content
This repository has been archived by the owner on Jun 2, 2021. It is now read-only.

Integration with external CRMs #10

Open
tttp opened this issue Feb 23, 2020 · 6 comments
Open

Integration with external CRMs #10

tttp opened this issue Feb 23, 2020 · 6 comments
Labels
help wanted Extra attention is needed question Further information is requested
Projects

Comments

@tttp
Copy link
Member

tttp commented Feb 23, 2020

the aim of proca is to be able to push the signatures and supporters contact details to the organisers' communication tools (CRM, mailinglist...).

This is triggered by an event in proca (eg new petition signature), might need a different workflow based on the event data (eg push it to the mailing list only if the signatory optsin).

In the future, these events/workflow could be chained, like:

  • signature requiring a double opt-in
  • event pushing to a component sending a transactional email
  • user clicks on opt-in link
  • update the signature status (from requested to optin)
  • event pushing the signature data to the organiser CRM

Shall we write ad-hoc components to integrate with various CRMs (eg one for civi, one for mailchimp...)?

Or should we already integrate with a event/agent & automation system?
at these opensource ones seems to be interesting:

Requirements for going external:

  • handling failures (eg a mailing list can't process adding new signatories)
  • being able to have alerts/overview if one queue isn't consumed (eg a CRM becomes unresponsive and lots of signatures aren's pushed there)
  • having a community
  • if possible having already integration with existing mailinglist managers/CRM

and ideally having a batch mode (ie. instead of triggering one call per event, being able to batch them, like all the signatures in the past minute or by batch of 100)

I think we should go with an existing event/workflow system eventually, not sure when

Right now, we need a system to export signatures, at minima as a CSV ;)

@tttp tttp added question Further information is requested help wanted Extra attention is needed labels Feb 23, 2020
@tttp
Copy link
Member Author

tttp commented Feb 23, 2020

@marcinkoziej this needs clarification and brainstorm when you are back from holidays ;)

@marcinkoziej marcinkoziej added this to Backlog and questions in MVP Mar 2, 2020
@marcinkoziej
Copy link
Contributor

The action pipeline

My idea for conceptualizing all this post-processing is an action "pipeline", which is an ordered set of functions which will be called in asynchronous manner on the signature (or different action).

I think that pipeline will be set per-action_page, because there could be maybe different opt in policies, and different CRM's to post to.

So for instance, for a new signature:

  • It is created and saved. It's pipeline position is 0.
  • The worker batch job checks for first step in the pipeline for this signature. It is double-opt-in email. This step is run. The content of the mail is retrieved (somehow, from Ghost maybe?) and email is sent. A "pending" state is stored for this pipeline step. Because there is nothing else to do, stop.
  • On the next check, the signature is at position 1 and pending a confirmation. Nothing to be done. Maybe, after some timeout, a second confirmation email can be sent, to nag the person to confirm.
  • Member clicks confirmation link, signature's pending state is removed, and its position in the pipeline is advanced to 2. Next step is to post to CRM queue. It's done synchronously and position is advanced to 3. This is "notify MP step". It has parameters configured: thresholds + email to MP template. If threshold is met, it sends the MP notification. Advances to pipeline step 4.
  • If pipeline is finished, mark this signature pipeline as complete.

An external service could be called by "webhook" pipeline steps. There would be two types:

  • AsyncWebhook - send data to webhook (perhaps using a json template to format data) and advance to next pipeline step
  • SyncWebhook - send data to webhook and mark this step as Pending. Release the pending state when external system will call a webhook.

Webhooks and DOS

I think we should read a little bit more about webhook delivery issues, I found this:
https://techblog.commercetools.com/webhooks-the-devil-in-the-details-ca7f7982c24f
and after reading it I realized that proca should also not implement this functionality. If we say "let's keep the system minimal, let the external system send emails to targets, etc", then why should we implement a message queue? Let's make AsyncWebhook and SyncWebhook post to RabbitMQ or other queues, and the client can use standard MQ APIs to get messahes without losses and at pace they want. We assume we setup proca<->MQ communication will be ok with just resending messages on delivery failure (say, if MQ is down for a while, the background worker will resend messages to it).

@marcinkoziej
Copy link
Contributor

Automation systems

These look very coooool. Maybe we can set up one of them to check out how easy/hard it is to use them? And what can be done with them? Can they implement the double opt in mechanics?

@tttp
Copy link
Member Author

tttp commented Mar 30, 2020

So what you suggest (trying to make explicit limitations/simplifications):

  • there is one and only one workflow per signature/actionPage, and the steps can't be done in parallel (say you want to push the signature to the CRM and an analytic system, it won't go on analytics until the CRM has processed it)

Proca is responsible to maintain the state (position in the workflow) for every signature.

What about we turn that on its head and we let the external systems handle the state, eg instead of pushing each signature to the CRM (+risk of DoSing it), we let the external system fetch (give me up to 100 signatures after id=xyz - or timestamp)

the CRM - or external system- being in charge of knowing what it fetched last and what it needs to fetch

to make it more responsive, we can have a webhook to notify (max once per every x second) than a new signature has been collected

yeah, worth looking at automation system, great idea!

@marcinkoziej
Copy link
Contributor

Yes, they would not be done in parallel.
This could have a consequence, that if the primary recipient (CRM) is down, the signature is not pushed to secondary system (analytic system, although usually you connect it to CRM and not action tool yeah?). The win: more predictable consistency among distributed system.

@marcinkoziej
Copy link
Contributor

Writing integrations (using push or pull) is now a lot easier with @proca/cli

MVP automation moved this from Backlog and questions to Done Oct 20, 2020
@marcinkoziej marcinkoziej reopened this Oct 20, 2020
MVP automation moved this from Done to In progress Oct 20, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
help wanted Extra attention is needed question Further information is requested
Projects
MVP
  
In progress
Development

No branches or pull requests

2 participants