Fix various threading issues, particularly in Gmail

Summary:
  - Add an explicit multiplexing option, and enable it by default. This is necessary for Mail.app to coexist with other clients ("Re:" breaks outlook at the very least, and generally sucks in the common case), and allows users with flexible clients to enable subject variance.
  - Add an option for subject line variance. Default to not varying the subject, so mail no longer says [Committed], [Closed], etc. This is so the defaults thread correctly in Gmail (not entirely sure this actually works).
  - Add a preference to enable subject line variance.
  - Unless all mail is multiplexed, don't enable or respect the "Re" or "vary subject" preferences. These are currently shown and respected in non-multiplex cases, which creates inconsistent results.

NOTE: @jungejason @nh @vrana This changes the default behavior (from non-multiplexing to multiplexing), and might break Facebook's integration. You should be able to keep the same behavior by setting the options appropriately, although if you can get the new defaults working they're probably better.

Test Plan:
Send mail from Maniphest, Differential and Audit. Updated preferences. Enabled/disabled multiplexing. Things seem OK?

NOTE: I haven't actually been able to repro the Gmail threading issue so I'm not totally sure what's going on there, maybe it started respecting "Re:" (or always has), but @cpiro and @20after4 both reported it independently. This fixes a bunch of bugs in any case and gives us more conservative set of defaults.

I'll see if I can buff out the Gmail story a bit but every client is basically a giant black box of mystery. :/

Reviewers: btrahan, vrana, jungejason, nh

Reviewed By: btrahan

CC: cpiro, 20after4, aran

Maniphest Tasks: T1097, T847

Differential Revision: https://secure.phabricator.com/D2206

Author: epriestley

conf/default.conf.php

@@ -145,6 +145,35 @@
   // Domain used to generate Message-IDs.
   'metamta.domain'              => 'example.com',
 
+  // When a message is sent to multiple recipients (for example, several
+  // reviewers on a code review), Phabricator can either deliver one email to
+  // everyone (e.g., "To: alincoln, usgrant, htaft") or separate emails to each
+  // user (e.g., "To: alincoln", "To: usgrant", "To: htaft"). The major
+  // advantages and disadvantages of each approach are:
+  //
+  //   - One mail to everyone:
+  //     - Recipients can see To/Cc at a glance.
+  //     - If you use mailing lists, you won't get duplicate mail if you're
+  //       a normal recipient and also Cc'd on a mailing list.
+  //     - Getting threading to work properly is harder, and probably requires
+  //       making mail less useful by turning off options.
+  //     - Sometimes people will "Reply All" and everyone will get two mails,
+  //       one from the user and one from Phabricator turning their mail into
+  //       a comment.
+  //     - Not supported with a private reply-to address.
+  //   - One mail to each user:
+  //     - Recipients need to look in the mail body to see To/Cc.
+  //     - If you use mailing lists, recipients may sometimes get duplicate
+  //       mail.
+  //     - Getting threading to work properly is easier, and threading settings
+  //       can be customzied by each user.
+  //     - "Reply All" no longer spams all other users.
+  //     - Required if private reply-to addresses are configured.
+  //
+  // In the code, splitting one outbound email into one-per-recipient is
+  // sometimes referred to as "multiplexing".
+  'metamta.one-mail-per-recipient'  => true,
+
   // When a user takes an action which generates an email notification (like
   // commenting on a Differential revision), Phabricator can either send that
   // mail "From" the user's email address (like "alincoln@logcabin.com") or
@@ -166,6 +195,7 @@
   //      mostly never arrive.
   'metamta.can-send-as-user'    => false,
 
+
   // Adapter class to use to transmit mail to the MTA. The default uses
   // PHPMailerLite, which will invoke "sendmail". This is appropriate
   // if sendmail actually works on your host, but if you haven't configured mail
@@ -329,9 +359,18 @@
 
   // Mail.app on OS X Lion won't respect threading headers unless the subject
   // is prefixed with "Re:". If you enable this option, Phabricator will add
-  // "Re:" to the subject line of all mail which is expected to thread.
+  // "Re:" to the subject line of all mail which is expected to thread. If
+  // you've set 'metamta.one-mail-per-recipient', users can override this
+  // setting in their preferences.
   'metamta.re-prefix' => false,
 
+  // If true, allow MetaMTA to change mail subjects to put text like
+  // '[Accepted]' and '[Commented]' in them. This makes subjects more useful,
+  // but might break threading on some clients. If you've set
+  // 'metamta.one-mail-per-recipient', users can override this setting in their
+  // preferences.
+  'metamta.vary-subjects' => true,
+
 
 // -- Auth ------------------------------------------------------------------ //
 

src/applications/audit/editor/comment/PhabricatorAuditCommentEditor.php

@@ -283,12 +283,12 @@ private function sendMail(
     $reply_handler = self::newReplyHandlerForCommit($commit);
 
     $prefix = PhabricatorEnv::getEnvConfig('metamta.diffusion.subject-prefix');
-    $subject = "{$prefix} [{$verb}] {$name}: {$summary}";
+    $subject = "{$prefix} {$name}: {$summary}";
+    $vary_subject = "{$prefix} [{$verb}] {$name}: {$summary}";
 
     $threading = self::getMailThreading($commit->getPHID());
     list($thread_id, $thread_topic) = $threading;
 
-    $is_new     = !count($other_comments);
     $body       = $this->renderMailBody(
       $comment,
       "{$name}: {$summary}",
@@ -311,8 +311,13 @@ private function sendMail(
     $phids = array_merge($email_to, $email_cc);
     $handles = id(new PhabricatorObjectHandleData($phids))->loadHandles();
 
+    // NOTE: Always set $is_new to false, because the "first" mail in the
+    // thread is the Herald notification of the commit.
+    $is_new = false;
+
     $template = id(new PhabricatorMetaMTAMail())
       ->setSubject($subject)
+      ->setVarySubject($subject)
       ->setFrom($comment->getActorPHID())
       ->setThreadID($thread_id, $is_new)
       ->addHeader('Thread-Topic', $thread_topic)
@@ -332,7 +337,7 @@ private function sendMail(
 
   public static function getMailThreading($phid) {
     return array(
-      '<diffusion-audit-'.$phid.'>',
+      'diffusion-audit-'.$phid,
       'Diffusion Audit '.$phid,
     );
   }

src/applications/differential/mail/base/DifferentialMail.php

@@ -34,7 +34,14 @@ abstract class DifferentialMail {
   protected $replyHandler;
   protected $parentMessageID;
 
-  abstract protected function renderSubject();
+  protected function renderSubject() {
+    $revision = $this->getRevision();
+    $title = $revision->getTitle();
+    $id = $revision->getID();
+    return "D{$id}: {$title}";
+  }
+
+  abstract protected function renderVarySubject();
   abstract protected function renderBody();
 
   public function setActorHandle($actor_handle) {
@@ -70,10 +77,11 @@ public function send() {
       throw new Exception('No "To:" users provided!');
     }
 
-    $cc_phids    = $this->getCCPHIDs();
-    $subject     = $this->buildSubject();
-    $body        = $this->buildBody();
-    $attachments = $this->buildAttachments();
+    $cc_phids     = $this->getCCPHIDs();
+    $subject      = $this->buildSubject();
+    $vary_subject = $this->buildVarySubject();
+    $body         = $this->buildBody();
+    $attachments  = $this->buildAttachments();
 
     $template = new PhabricatorMetaMTAMail();
     $actor_handle = $this->getActorHandle();
@@ -85,6 +93,7 @@ public function send() {
 
     $template
       ->setSubject($subject)
+      ->setVarySubject($vary_subject)
       ->setBody($body)
       ->setIsHTML($this->shouldMarkMailAsHTML())
       ->setParentMessageID($this->parentMessageID)
@@ -197,6 +206,10 @@ protected function buildSubject() {
     return trim($this->getSubjectPrefix().' '.$this->renderSubject());
   }
 
+  protected function buildVarySubject() {
+    return trim($this->getSubjectPrefix().' '.$this->renderVarySubject());
+  }
+
   protected function shouldMarkMailAsHTML() {
     return false;
   }
@@ -320,8 +333,7 @@ public function getRevision() {
 
   protected function getThreadID() {
     $phid = $this->getRevision()->getPHID();
-    $domain = PhabricatorEnv::getEnvConfig('metamta.domain');
-    return "<differential-rev-{$phid}-req@{$domain}>";
+    return "differential-rev-{$phid}-req";
   }
 
   public function setComment($comment) {

src/applications/differential/mail/ccwelcome/DifferentialCCWelcomeMail.php

@@ -18,9 +18,8 @@
 
 final class DifferentialCCWelcomeMail extends DifferentialReviewRequestMail {
 
-  protected function renderSubject() {
-    $revision = $this->getRevision();
-    return 'Added to CC: '.$revision->getTitle();
+  protected function renderVarySubject() {
+    return '[Added to CC] '.$this->renderSubject();
   }
 
   protected function renderBody() {

src/applications/differential/mail/comment/DifferentialCommentMail.php

@@ -75,13 +75,9 @@ protected function getMailTags() {
     return $tags;
   }
 
-  protected function renderSubject() {
+  protected function renderVarySubject() {
     $verb = ucwords($this->getVerb());
-    $revision = $this->getRevision();
-    $title = $revision->getTitle();
-    $id = $revision->getID();
-    $subject = "[{$verb}] D{$id}: {$title}";
-    return $subject;
+    return "[{$verb}] ".$this->renderSubject();
   }
 
   protected function getVerb() {

src/applications/differential/mail/diffcontent/DifferentialDiffContentMail.php

@@ -25,8 +25,8 @@ public function __construct(DifferentialRevision $revision, $content) {
     $this->content = $content;
   }
 
-  protected function renderSubject() {
-    return "Content: ".$this->getRevision()->getTitle();
+  protected function renderVarySubject() {
+    return '[Content] '.$this->renderSubject();
   }
 
   protected function renderBody() {

src/applications/differential/mail/exception/DifferentialExceptionMail.php

@@ -36,6 +36,10 @@ protected function renderSubject() {
     return "Exception: unable to process your mail request.";
   }
 
+  protected function renderVarySubject() {
+    return $this->renderSubject();
+  }
+
   protected function buildBody() {
     $exception = $this->exception;
     $original_body = $this->originalBody;

src/applications/differential/mail/newdiff/DifferentialNewDiffMail.php

@@ -18,7 +18,7 @@
 
 final class DifferentialNewDiffMail extends DifferentialReviewRequestMail {
 
-  protected function renderSubject() {
+  protected function renderVarySubject() {
     $revision = $this->getRevision();
     $line_count = $revision->getLineCount();
     $lines = ($line_count == 1 ? "1 line" : "{$line_count} lines");
@@ -29,21 +29,7 @@ protected function renderSubject() {
       $verb = 'Updated';
     }
 
-    $revision_id = $revision->getID();
-    $revision_title = $revision->getTitle();
-
-    return "[{$verb}, {$lines}] D{$revision_id}: {$revision_title}";
-  }
-
-  protected function buildSubject() {
-    if (!$this->isFirstMailToRecipients()) {
-      return parent::buildSubject();
-    }
-
-    $prefix = $this->getSubjectPrefix();
-    $subject = $this->renderSubject();
-
-    return trim("{$prefix} {$subject}");
+    return "[{$verb}, {$lines}] ".$this->renderSubject();
   }
 
   protected function renderBody() {

src/applications/maniphest/editor/transaction/ManiphestTransactionEditor.php

@@ -266,16 +266,18 @@ private function sendEmail($task, $transactions, $email_to, $email_cc) {
         "  ".$reply_instructions."\n";
     }
 
-    $thread_id = '<maniphest-task-'.$task->getPHID().'>';
+    $thread_id = 'maniphest-task-'.$task->getPHID();
     $task_id = $task->getID();
     $title = $task->getTitle();
     $prefix = $this->getSubjectPrefix();
-    $subject = trim("{$prefix} [{$action}] T{$task_id}: {$title}");
+    $subject = trim("{$prefix} T{$task_id}: {$title}");
+    $vary_subject = trim("{$prefix} [{$action}] T{$task_id}: {$title}");
 
     $mailtags = $this->getMailTags($transactions);
 
     $template = id(new PhabricatorMetaMTAMail())
       ->setSubject($subject)
+      ->setVarySubject($vary_subject)
       ->setFrom($transaction->getAuthorPHID())
       ->setParentMessageID($this->parentMessageID)
       ->addHeader('Thread-Topic', 'Maniphest Task '.$task->getID())

src/applications/metamta/replyhandler/base/PhabricatorMailReplyHandler.php

@@ -80,22 +80,26 @@ final public function multiplexMail(
 
     $result = array();
 
-    // If private replies are not supported, simply send one email to all
-    // recipients and CCs. This covers cases where we have no reply handler,
-    // or we have a public reply handler.
-    if (!$this->supportsPrivateReplies()) {
-      $mail = clone $mail_template;
-      $mail->addTos(mpull($to_handles, 'getPHID'));
-      $mail->addCCs(mpull($cc_handles, 'getPHID'));
-
-      if ($this->supportsPublicReplies()) {
-        $reply_to = $this->getPublicReplyHandlerEmailAddress();
-        $mail->setReplyTo($reply_to);
+    // If MetaMTA is configured to always multiplex, skip the single-email
+    // case.
+    if (!PhabricatorMetaMTAMail::shouldMultiplexAllMail()) {
+      // If private replies are not supported, simply send one email to all
+      // recipients and CCs. This covers cases where we have no reply handler,
+      // or we have a public reply handler.
+      if (!$this->supportsPrivateReplies()) {
+        $mail = clone $mail_template;
+        $mail->addTos(mpull($to_handles, 'getPHID'));
+        $mail->addCCs(mpull($cc_handles, 'getPHID'));
+
+        if ($this->supportsPublicReplies()) {
+          $reply_to = $this->getPublicReplyHandlerEmailAddress();
+          $mail->setReplyTo($reply_to);
+        }
+
+        $result[] = $mail;
+
+        return $result;
       }
-
-      $result[] = $mail;
-
-      return $result;
     }
 
     // Merge all the recipients together. TODO: We could keep the CCs as real

src/applications/metamta/replyhandler/base/__init__.php

@@ -6,6 +6,7 @@
 
 
 
+phutil_require_module('phabricator', 'applications/metamta/storage/mail');
 phutil_require_module('phabricator', 'applications/metamta/storage/receivedmail');
 phutil_require_module('phabricator', 'applications/phid/constants');
 phutil_require_module('phabricator', 'infrastructure/env');