Merge pull request #17669 from seamuslee001/flexmailer_core

[REF] Ship Flexmailer extension with Core

.gitignore

@@ -1,8 +1,9 @@
 *~
 *.bak
 .use-civicrm-setup
-/ext/
+/ext/*
 !/ext/sequentialcreditnotes
+!/ext/flexmailer
 backdrop/
 bower_components
 CRM/Case/xml/configuration

ext/flexmailer/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Change Log
+
+## v0.2-alpha1
+
+* Override core's `Mailing.preview` API to support rendering via
+  Flexmailer events.
+* (BC Break) In the class `DefaultComposer`, change the signature for
+  `createMessageTemplates()` and `applyClickTracking()` to provide full
+  access to the event context (`$e`).

ext/flexmailer/README.md

@@ -0,0 +1,12 @@
+# org.civicrm.flexmailer
+
+FlexMailer (`org.civicrm.flexmailer`) is an email delivery engine for CiviCRM v4.7+.  It replaces the internal guts of CiviMail.  It is a
+drop-in replacement which enables *other* extensions to provide richer email features.
+
+* [Introduction](docs/index.md)
+* [Installation](docs/install.md)
+* [Development](docs/develop/index.md)
+    * [CheckSendableEvent](docs/develop/CheckSendableEvent.md)
+    * [WalkBatchesEvent](docs/develop/WalkBatchesEvent.md)
+    * [ComposeBatchEvent](docs/develop/ComposeBatchEvent.md)
+    * [SendBatchEvent](docs/develop/SendBatchEvent.md)

ext/flexmailer/docs/develop/CheckSendableEvent.md

@@ -0,0 +1,29 @@
+
+The `CheckSendableEvent` (`EVENT_CHECK_SENDABLE`) determines whether a draft mailing is fully specified for delivery.
+
+For example, some jurisdictions require that email blasts provide contact
+information for the organization (eg street address) and an opt-out link.
+Traditionally, the check-sendable event will verify that this information is
+provided through a CiviMail token (eg `{action.unsubscribeUrl}`).
+
+But what happens if you implement a new template language (e.g. Mustache) with
+a different mail-merge notation? The validation will need to be different.
+In this example, we verify the presence of a Mustache-style token, `{{unsubscribeUrl}}`.
+
+```php
+<?php
+function mustache_civicrm_container($container) {
+  $container->addResource(new \Symfony\Component\Config\Resource\FileResource(__FILE__));
+  $container->findDefinition('dispatcher')->addMethodCall('addListener',
+    array(\Civi\FlexMailer\Validator::EVENT_CHECK_SENDABLE, '_mustache_check_sendable')
+  );
+}
+
+function _mustache_check_sendable(\Civi\FlexMailer\Event\CheckSendableEvent $e) {
+  if ($e->getMailing()->template_type !== 'mustache') return;
+
+  if (strpos('{{unsubscribeUrl}}', $e->getMailing()->body_html) === FALSE) {
+    $e->setError('body_html:unsubscribeUrl', E::ts('Please include the token {{unsubscribeUrl}}'));
+  }
+}
+```

ext/flexmailer/docs/develop/ComposeBatchEvent.md

@@ -0,0 +1,62 @@
+The `ComposeBatchEvent` builds the email messages.  Each message is represented as a `FlexMailerTask` with a list of `MailParams`.
+
+Some listeners are "under the hood" -- they define less visible parts of the message, e.g.
+
+ * `BasicHeaders`  defines `Message-Id`, `Precedence`, `From`, `Reply-To`, and others.
+ * `BounceTracker` defines various headers for bounce-tracking.
+ * `OpenTracker` appends an HTML tracking code to any HTML messages.
+
+The heavy-lifting of composing the message content is also handled by a listener, such as
+`DefaultComposer`. `DefaultComposer` replicates traditional CiviMail functionality:
+
+ * Reads email content from `$mailing->body_text` and `$mailing->body_html`.
+ * Interprets tokens like `{contact.display_name}` and `{mailing.viewUrl}`.
+ * Loads data in batches.
+ * Post-processes the message with Smarty (if `CIVICRM_SMARTY` is enabled).
+
+The traditional CiviMail semantics have some problems -- e.g.  the Smarty post-processing is incompatible with Smarty's
+template cache, and it is difficult to securely post-process the message with Smarty.  However, changing the behavior
+would break existing templates.
+
+A major goal of FlexMailer is to facilitate a migration toward different template semantics.  For example, an
+extension might (naively) implement support for Mustache templates using:
+
+```php
+<?php
+function mustache_civicrm_container($container) {
+  $container->addResource(new \Symfony\Component\Config\Resource\FileResource(__FILE__));
+  $container->findDefinition('dispatcher')->addMethodCall('addListener',
+    array(\Civi\FlexMailer\FlexMailer::EVENT_COMPOSE, '_mustache_compose_batch')
+  );
+}
+
+function _mustache_compose_batch(\Civi\FlexMailer\Event\ComposeBatchEvent $event) {
+  if ($event->getMailing()->template_type !== 'mustache') return;
+
+  $m = new Mustache_Engine();
+  foreach ($event->getTasks() as $task) {
+    if ($task->hasContent()) continue;
+    $contact = civicrm_api3('Contact', 'getsingle', array(
+      'id' => $task->getContactId(),
+    ));
+    $task->setMailParam('text', $m->render($event->getMailing()->body_text, $contact));
+    $task->setMailParam('html', $m->render($event->getMailing()->body_html, $contact));
+  }
+}
+```
+
+This implementation is naive in a few ways -- it performs separate SQL queries for each recipient; it doesn't optimize
+the template compilation; it has a very limited range of tokens; and it doesn't handle click-through tracking.  For
+more ideas about these issues, review `DefaultComposer`.
+
+> FIXME: Core's `TokenProcessor` is useful for batch-loading token data.
+> However, you currently have to use `addMessage()` and `render()` to kick it
+> off -- but those are based on CiviMail template notation.  We should provide
+> another function that doesn't depend on the template notation -- so that
+> other templates can leverage our token library.
+
+> **Tip**: When you register a listener for `EVENT_COMPOSE`, note the weight.
+> The default weight puts your listener in the middle of pipeline -- right
+> before the `DefaultComposer`.  However, you might want to position
+> relative to other places -- e.g.  `WEIGHT_PREPARE`, `WEIGHT_MAIN`,
+> `WEIGHT_ALTER`, or `WEIGHT_END`.

ext/flexmailer/docs/develop/RunEvent.md

@@ -0,0 +1,36 @@
+The `RunEvent` (`EVENT_RUN`) fires as soon as FlexMailer begins processing a job.
+
+CiviMail has a recurring task -- `Job.process_mailings` -- which identifies scheduled/pending mailings.  It determines
+the `Mailing` and `MailingJob` records, then passes control to `FlexMailer` to perform delivery.  `FlexMailer`
+immediately fires the `RunEvent`.
+
+!!! note "`RunEvent` fires for each cron-run."
+
+    By default, FlexMailer uses `DefaultBatcher` which obeys the traditional CiviMail throttling behavior.  This can
+    limit the number of deliveries performed within a single cron-run.  If you reach this limit, then it stops
+    execution.  However, after 5 or 10 minutes, a new *cron-run* begins.  It passes control to FlexMailer again, and
+    then we pick up where we left off.  This means that one `Mailing` and one `MailingJob` could require multiple
+    *cron-runs*.
+
+    The `RunEvent` would fire for *every cron run*.
+
+To listen to the `RunEvent`:
+
+```php
+<?php
+function example_civicrm_container($container) {
+  $container->addResource(new \Symfony\Component\Config\Resource\FileResource(__FILE__));
+  $container->findDefinition('dispatcher')->addMethodCall('addListener',
+    array(\Civi\FlexMailer\FlexMailer::EVENT_RUN, '_example_run')
+  );
+}
+
+function _example_run(\Civi\FlexMailer\Event\RunEvent $event) {
+  printf("Starting work on job #%d for mailing #%d\n", $event->getJob()->id, $event->getMailing()->id);
+}
+```
+
+!!! note "Stopping the `RunEvent` will stop FlexMailer."
+
+    If you call `$event->stopPropagation()`, this will cause FlexMailer to
+    stop its delivery process.

ext/flexmailer/docs/develop/SendBatchEvent.md

@@ -0,0 +1,25 @@
+The `SendBatchEvent` (`EVENT_SEND`) takes a batch of recipients and messages, and it delivers the messages.  For example, suppose you wanted to
+replace the built-in delivery mechanism with a batch-oriented web-service:
+
+```php
+<?php
+function example_civicrm_container($container) {
+  $container->addResource(new \Symfony\Component\Config\Resource\FileResource(__FILE__));
+  $container->findDefinition('dispatcher')->addMethodCall('addListener',
+    array(\Civi\FlexMailer\FlexMailer::EVENT_SEND, '_example_send_batch')
+  );
+}
+
+function _example_send_batch(\Civi\FlexMailer\Event\SendBatchEvent $event) {
+  $event->stopPropagation(); // Disable standard delivery
+
+  $context = stream_context_create(array(
+    'http' => array(
+      'method'  => 'POST',
+      'header'  => 'Content-type: application/vnd.php.serialize',
+      'content' => serialize($event->getTasks()),
+    ),
+  ));
+  return file_get_contents('https://example.org/batch-delivery', false, $context);
+}
+```

ext/flexmailer/docs/develop/WalkBatchesEvent.md

@@ -0,0 +1,26 @@
+The `WalkBatchesEvent` examines the recipient list and pulls out a subset for whom you want to send email.  This is useful if you need strategies for
+chunking-out deliveries.
+
+The basic formula for defining your own batch logic is:
+
+```php
+<?php
+function example_civicrm_container($container) {
+  $container->addResource(new \Symfony\Component\Config\Resource\FileResource(__FILE__));
+  $container->findDefinition('dispatcher')->addMethodCall('addListener',
+    array(\Civi\FlexMailer\FlexMailer::EVENT_WALK, '_example_walk_batches')
+  );
+}
+
+function _example_walk_batches(\Civi\FlexMailer\Event\WalkBatchesEvent $event) {
+  $event->stopPropagation(); // Disable standard delivery
+
+  while (...) {
+    $tasks = array();
+    $task[] = new FlexMailerTask(...);
+    $task[] = new FlexMailerTask(...);
+    $task[] = new FlexMailerTask(...);
+    $event->visit($tasks);
+  }
+}
+```

ext/flexmailer/docs/develop/index.md

@@ -0,0 +1,136 @@
+## Unit tests
+
+The [headless unit tests](https://docs.civicrm.org/dev/en/latest/testing/#headless) are based on PHPUnit and `cv`. Simply run:
+
+```
+$ phpunit5
+```
+
+## Events
+
+!!! tip "Symfony Events"
+
+    This documentation references the [Symfony EventDispatcher](http://symfony.com/components/EventDispatcher).
+    If this is unfamiliar, you can read [a general introduction to Symfony events](http://symfony.com/doc/2.7/components/event_dispatcher.html)
+    or [a specific introduction about CiviCRM and Symfony events](https://docs.civicrm.org/dev/en/latest/hooks/setup/symfony/).
+
+FlexMailer is an *event* based delivery system. It defines a few events:
+
+* [CheckSendableEvent](CheckSendableEvent.md): In this event, one examines a draft mailing to determine if it is complete enough to deliver.
+* [RunEvent](RunEvent.md): When a cron-worker starts processing a `MailingJob`, this event fires. It can be used to initialize resources... or to completely circumvent the normal process.
+* [WalkBatchesEvent](WalkBatchesEvent.md): In this event, one examines the recipient list and pulls out a subset for whom you want to send email.
+* [ComposeBatchEvent](ComposeBatchEvent.md): In this event, one examines the mail content and the list of recipients -- then composes a batch of fully-formed email messages.
+* [SendBatchEvent](SendBatchEvent.md): In this event, one takes a batch of fully-formed email messages and delivers the messages.
+
+These events are not conceived in the same way as a typical *CiviCRM hook*; rather, they resemble *pipelines*.  For each event, several listeners
+have an opportunity to weigh-in, and the *order* of the listeners is important.  As such, it helps to *inspect* the list of listeners.  You can do
+this with the CLI command, `cv`:
+
+```
+$ cv debug:event-dispatcher /flexmail/
+[Event] civi.flexmailer.checkSendable
++-------+------------------------------------------------------------+
+| Order | Callable                                                   |
++-------+------------------------------------------------------------+
+| #1    | Civi\FlexMailer\Listener\Abdicator->onCheckSendable()      |
+| #2    | Civi\FlexMailer\Listener\RequiredFields->onCheckSendable() |
+| #3    | Civi\FlexMailer\Listener\RequiredTokens->onCheckSendable() |
++-------+------------------------------------------------------------+
+
+[Event] civi.flexmailer.walk
++-------+---------------------------------------------------+
+| Order | Callable                                          |
++-------+---------------------------------------------------+
+| #1    | Civi\FlexMailer\Listener\DefaultBatcher->onWalk() |
++-------+---------------------------------------------------+
+
+[Event] civi.flexmailer.compose
++-------+-------------------------------------------------------+
+| Order | Callable                                              |
++-------+-------------------------------------------------------+
+| #1    | Civi\FlexMailer\Listener\BasicHeaders->onCompose()    |
+| #2    | Civi\FlexMailer\Listener\ToHeader->onCompose()        |
+| #3    | Civi\FlexMailer\Listener\BounceTracker->onCompose()   |
+| #4    | Civi\FlexMailer\Listener\DefaultComposer->onCompose() |
+| #5    | Civi\FlexMailer\Listener\Attachments->onCompose()     |
+| #6    | Civi\FlexMailer\Listener\OpenTracker->onCompose()     |
+| #7    | Civi\FlexMailer\Listener\HookAdapter->onCompose()     |
++-------+-------------------------------------------------------+
+
+[Event] civi.flexmailer.send
++-------+--------------------------------------------------+
+| Order | Callable                                         |
++-------+--------------------------------------------------+
+| #1    | Civi\FlexMailer\Listener\DefaultSender->onSend() |
++-------+--------------------------------------------------+
+```
+
+The above listing shows the default set of listeners at time of writing. (Run the command yourself to see how they appear on your system.)
+The default listeners behave in basically the same way as CiviMail's traditional BAO-based delivery system (respecting `mailerJobSize`,
+`mailThrottleTime`, `mailing_backend`, `hook_civicrm_alterMailParams`, etal).
+
+There are a few tricks for manipulating the pipeline:
+
+* __Register new listeners__. Each event has its own documentation which describes how to do this.
+* __Manage the priority__. When registering a listener, the `addListener()` function accepts a `$priority` integer. Use this to move up or down the pipeline.
+
+    !!! note "Priority vs Order"
+
+        When writing code, you will set the *priority* of a listener. The default is `0`, and the usual range is `2000` (first) to `-2000` (last).
+
+        <!-- The default listeners have priorities based on the constants `FlexMailer::WEIGHT_PREPARE` (1000), `FlexMailer::WEIGHT_MAIN` (0),
+        `FlexMailer::WEIGHT_ALTER` (-1000), and `FlexMailer::WEIGHT_END` (-2000). -->
+
+        At runtime, the `EventDispatcher` will take all the listeners and sort them by priority. This produces the *order*, which simply counts up (`1`, `2`, `3`, ...).
+
+* __Alter a listener__. Most listeners are *services*, and you can manipulate options on these services. For example, suppose you wanted to replace the default bounce-tracking mechanism.
+  Here's a simple way to disable the default `BounceTracker`:
+
+    ```php
+    <?php
+    \Civi::service('civi_flexmailer_bounce_tracker')->setActive(FALSE);
+    ```
+
+    Of course, this change needs to be made before the listener runs. You might use a global hook (like `hook_civicrm_config`), or you might
+    have your own listener which disables `civi_flexmailer_bounce_tracker` and adds its own bounce-tracking.
+
+    Most FlexMailer services support `setActive()`, which enables you to completely replace them.
+
+    Additionally, some services have their own richer methods. In this example, we modify the list of required tokens:
+
+    ```php
+    <?php
+    $tokens = \Civi::service('civi_flexmailer_required_tokens')
+      ->getRequiredTokens();
+
+    unset($tokens['domain.address']);
+
+    \Civi::service('civi_flexmailer_required_tokens')
+      ->setRequiredTokens($tokens);
+    ```
+
+## Services
+
+Most features in FlexMailer are implemented by *services*, and you can override or manipulate these features if you understand the corresponding service.
+For more detailed information about how to manipulate a service, consult its docblocks.
+
+* Listener services (`CheckSendableEvent`)
+     * `civi_flexmailer_required_fields` (`RequiredFields.php`): Check for fields like "Subject" and "From".
+     * `civi_flexmailer_required_tokens` (`RequiredTokens.php`): Check for tokens like `{action.unsubscribeUrl}` (in `traditional` mailings).
+* Listener services (`WalkBatchesEvent`)
+     * `civi_flexmailer_default_batcher` (`DefaultBatcher.php`): Split the recipient list into smaller batches (per CiviMail settings)
+* Listener services (`ComposeBatchEvent`)
+     * `civi_flexmailer_basic_headers` (`BasicHeaders.php`): Add `From:`, `Reply-To:`, etc
+     * `civi_flexmailer_to_header` (`ToHeader.php`): Add `To:` header
+     * `civi_flexmailer_bounce_tracker` (`BounceTracker.php`): Add bounce-tracking codes
+     * `civi_flexmailer_default_composer` (`DefaultComposer.php`): Read the email template and evaluate any tokens (based on CiviMail tokens)
+     * `civi_flexmailer_attachments` (`Attachments.php`): Add attachments
+     * `civi_flexmailer_open_tracker` (`OpenTracker.php`): Add open-tracking codes
+     * `civi_flexmailer_test_prefix` (`TestPrefix.php`): Add a prefix to any test mailings
+     * `civi_flexmailer_hooks` (`HookAdapter.php`): Backward compatibility with `hook_civicrm_alterMailParams`
+* Listener services (`SendBatchEvent`)
+     * `civi_flexmailer_default_sender` (`DefaultSender.php`): Send the batch using CiviCRM's default delivery service
+* Other services
+     * `civi_flexmailer_html_click_tracker` (`HtmlClickTracker.php`): Add click-tracking codes (for HTML messages)
+     * `civi_flexmailer_text_click_tracker` (`TextClickTracker.php`): Add click-tracking codes (for plain-text messages)
+     * `civi_flexmailer_api_overrides` (`Services.php.php`): Alter the `Mailing` APIs

ext/flexmailer/docs/index.md

@@ -0,0 +1,13 @@
+FlexMailer (`org.civicrm.flexmailer`) is an email delivery engine for CiviCRM v4.7+.  It replaces the internal guts of CiviMail.  It is a
+drop-in replacement which enables *other* extensions to provide richer email features.
+
+By default, FlexMailer supports the same user interfaces, delivery algorithms, and use-cases as CiviMail.  After activating FlexMailer, an
+administrator does not need to take any special actions.
+
+The distinguishing improvement here is under-the-hood: it provides better APIs and events for extension-developers.  For example,
+other extensions might:
+
+* Change the template language
+* Manipulate tracking codes
+* Rework the delivery mechanism
+* Redefine the batching algorithm

ext/flexmailer/docs/install.md

@@ -0,0 +1,19 @@
+To download the latest alpha or beta version:
+
+```bash
+$ cv dl --dev flexmailer
+```
+
+To download the latest, bleeding-edge code:
+
+```bash
+$ cv dl org.civicrm.flexmailer@https://github.com/civicrm/org.civicrm.flexmailer/archive/master.zip
+```
+
+To download the latest, bleeding-edge code from git:
+
+```bash
+$ cd $(cv path -x .)
+$ git clone https://github.com/civicrm/org.civicrm.flexmailer.git
+$ cv en flexmailer
+```