Integration with external CRMs

[tttp]

the aim of proca is to be able to push the signatures and supporters contact details to the organisers' communication tools (CRM, mailinglist...).

This is triggered by an event in proca (eg new petition signature), might need a different workflow based on the event data (eg push it to the mailing list only if the signatory optsin).

In the future, these events/workflow could be chained, like:

• signature requiring a double opt-in
• event pushing to a component sending a transactional email
• user clicks on opt-in link
• update the signature status (from requested to optin)
• event pushing the signature data to the organiser CRM

Shall we write ad-hoc components to integrate with various CRMs (eg one for civi, one for mailchimp...)?

Or should we already integrate with a event/agent & automation system?
at these opensource ones seems to be interesting:

• mautic
• beehive
• flogo

Requirements for going external:

• handling failures (eg a mailing list can't process adding new signatories)
• being able to have alerts/overview if one queue isn't consumed (eg a CRM becomes unresponsive and lots of signatures aren's pushed there)
• having a community
• if possible having already integration with existing mailinglist managers/CRM

and ideally having a batch mode (ie. instead of triggering one call per event, being able to batch them, like all the signatures in the past minute or by batch of 100)

I think we should go with an existing event/workflow system eventually, not sure when

Right now, we need a system to export signatures, at minima as a CSV ;)

[tttp]

@marcinkoziej this needs clarification and brainstorm when you are back from holidays ;)

[marcinkoziej]

The action pipeline

My idea for conceptualizing all this post-processing is an action "pipeline", which is an ordered set of functions which will be called in asynchronous manner on the signature (or different action).

I think that pipeline will be set per-action_page, because there could be maybe different opt in policies, and different CRM's to post to.

So for instance, for a new signature:

• It is created and saved. It's pipeline position is 0.
• The worker batch job checks for first step in the pipeline for this signature. It is double-opt-in email. This step is run. The content of the mail is retrieved (somehow, from Ghost maybe?) and email is sent. A "pending" state is stored for this pipeline step. Because there is nothing else to do, stop.
• On the next check, the signature is at position 1 and pending a confirmation. Nothing to be done. Maybe, after some timeout, a second confirmation email can be sent, to nag the person to confirm.
• Member clicks confirmation link, signature's pending state is removed, and its position in the pipeline is advanced to 2. Next step is to post to CRM queue. It's done synchronously and position is advanced to 3. This is "notify MP step". It has parameters configured: thresholds + email to MP template. If threshold is met, it sends the MP notification. Advances to pipeline step 4.
• If pipeline is finished, mark this signature pipeline as complete.

An external service could be called by "webhook" pipeline steps. There would be two types:

• AsyncWebhook - send data to webhook (perhaps using a json template to format data) and advance to next pipeline step
• SyncWebhook - send data to webhook and mark this step as Pending. Release the pending state when external system will call a webhook.

Webhooks and DOS

I think we should read a little bit more about webhook delivery issues, I found this:
https://techblog.commercetools.com/webhooks-the-devil-in-the-details-ca7f7982c24f
and after reading it I realized that proca should also not implement this functionality. If we say "let's keep the system minimal, let the external system send emails to targets, etc", then why should we implement a message queue? Let's make AsyncWebhook and SyncWebhook post to RabbitMQ or other queues, and the client can use standard MQ APIs to get messahes without losses and at pace they want. We assume we setup proca<->MQ communication will be ok with just resending messages on delivery failure (say, if MQ is down for a while, the background worker will resend messages to it).

[marcinkoziej]

Automation systems

These look very coooool. Maybe we can set up one of them to check out how easy/hard it is to use them? And what can be done with them? Can they implement the double opt in mechanics?

[tttp]

So what you suggest (trying to make explicit limitations/simplifications):

• there is one and only one workflow per signature/actionPage, and the steps can't be done in parallel (say you want to push the signature to the CRM and an analytic system, it won't go on analytics until the CRM has processed it)

Proca is responsible to maintain the state (position in the workflow) for every signature.

What about we turn that on its head and we let the external systems handle the state, eg instead of pushing each signature to the CRM (+risk of DoSing it), we let the external system fetch (give me up to 100 signatures after id=xyz - or timestamp)

the CRM - or external system- being in charge of knowing what it fetched last and what it needs to fetch

to make it more responsive, we can have a webhook to notify (max once per every x second) than a new signature has been collected

yeah, worth looking at automation system, great idea!

[marcinkoziej]

Yes, they would not be done in parallel.
This could have a consequence, that if the primary recipient (CRM) is down, the signature is not pushed to secondary system (analytic system, although usually you connect it to CRM and not action tool yeah?). The win: more predictable consistency among distributed system.

[marcinkoziej]

Writing integrations (using push or pull) is now a lot easier with @proca/cli