Allow transaction publishers to pass binary data to workers

Summary:
Ref T8672. Ref T9187. Root issue in at least one case is:

  - User makes a commit including a file with some non-UTF8 text (say, a Japanese file full of Shift-JIS).
  - We pass the file to the TransactionEditor so it can inline or attach the patch if the server is configured for these things.
    - When inlining patches, we convert them to UTF8 before inlining. We must do this since the rest of the mail is UTF8.
    - When attaching patches, we send them in the original encoding (as file attachments). This is correct, and means we need to give the worker the raw patch in whatever encoding it was originally in: we can't just convert it to utf8 earlier, or we'd attach the wrong patch in some cases.
  - TransactionEditor does its thing (e.g., creates the commit), then gets ready to send mail about whatever it did.
  - The publishing work now happens in the daemon queue, so we prepare to queue a PublishWorker and pass it the patch (with some other data).
  - When we queue workers, we serialize the state data with JSON.

So far, so good. But this is where things go wrong:

  - JSON can't encode binary data, and can't encode Shift-JIS. The encoding silently fails and we ignore it.

Then we get to the worker, and things go wrong-er:

  - Since the data is bad, we fatal. This isn't a permanent failure, so we continue retrying the task indefinitely.

This applies several fixes:

  # When queueing tasks, fail loudly when JSON encoding fails.
  # In the worker, fail permanently when data can't be decoded.
  # Allow Editors to specify that some of their data is binary and needs special handling.

This is fairly messy, but some simpler alternatives don't seem like good ways forward:

  - We can't convert to UTF8 earlier, because we need the original raw patch when adding it as an attachment.
  - We could encode //only// this field, but I suspect some other fields will also need attention, so that adding a mechanism will be worthwhile. In particular, I suspect filenames //may// be causing a similar problem in some cases.
  - We could convert task data to always use a serialize()-based binary safe encoding, but this is a larger change and I think it's correct that things are UTF8 by default, even if it makes a bit of a mess. I'd rather have an explicit mess like this than a lot of binary data floating around.

The change to make `LiskDAO` will almost certainly catch some other problems too, so I'm going to hold this until after `stable` is cut. These problems were existing problems (i.e., the code was previously breaking or destroying data) so it's definitely correct to catch them, but this will make the problems much more obvious/urgent than they previously were.

Test Plan:
  - Created a commit with a bunch of Shift-JIS stuff in a file.
  - Tried to import it.

Prior to patch:

  - Broken PublishWorker with distant, irrelevant error message.

With patch partially applied (only new error checking):

  - Explicit, local error message about bad key in serialized data.

With patch fully applied:

  - Import went fine and mail generated.

Reviewers: chad

Reviewed By: chad

Subscribers: devurandom, nevogd

Maniphest Tasks: T8672, T9187

Differential Revision: https://secure.phabricator.com/D13939

Author: epriestley

src/applications/audit/editor/PhabricatorAuditEditor.php

@@ -949,6 +949,12 @@ protected function getCustomWorkerState() {
     );
   }
 
+  protected function getCustomWorkerStateEncoding() {
+    return array(
+      'rawPatch' => self::STORAGE_ENCODING_BINARY,
+    );
+  }
+
   protected function loadCustomWorkerState(array $state) {
     $this->rawPatch = idx($state, 'rawPatch');
     $this->affectedFiles = idx($state, 'affectedFiles');

src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php

@@ -69,6 +69,8 @@ abstract class PhabricatorApplicationTransactionEditor
   private $feedNotifyPHIDs = array();
   private $feedRelatedPHIDs = array();
 
+  const STORAGE_ENCODING_BINARY = 'binary';
+
   /**
    * Get the class name for the application this editor is a part of.
    *
@@ -2637,6 +2639,21 @@ protected function addCustomFieldsToMailBody(
   }
 
 
+  /**
+   * @task mail
+   */
+  private function runHeraldMailRules(array $messages) {
+    foreach ($messages as $message) {
+      $engine = new HeraldEngine();
+      $adapter = id(new PhabricatorMailOutboundMailHeraldAdapter())
+        ->setObject($message);
+
+      $rules = $engine->loadRulesForAdapter($adapter);
+      $effects = $engine->applyRules($rules, $adapter);
+      $engine->applyEffects($effects, $adapter, $rules);
+    }
+  }
+
 
 /* -(  Publishing Feed Stories  )-------------------------------------------- */
 
@@ -3060,9 +3077,13 @@ final private function getWorkerState() {
       $state[$property] = $this->$property;
     }
 
+    $custom_state = $this->getCustomWorkerState();
+    $custom_encoding = $this->getCustomWorkerStateEncoding();
+
     $state += array(
       'excludeMailRecipientPHIDs' => $this->getExcludeMailRecipientPHIDs(),
-      'custom' => $this->getCustomWorkerState(),
+      'custom' => $this->encodeStateForStorage($custom_state, $custom_encoding),
+      'custom.encoding' => $custom_encoding,
     );
 
     return $state;
@@ -3080,6 +3101,21 @@ protected function getCustomWorkerState() {
   }
 
 
+  /**
+   * Hook; return storage encoding for custom properties which need to be
+   * passed to workers.
+   *
+   * This primarily allows binary data to be passed to workers and survive
+   * JSON encoding.
+   *
+   * @return dict<string, string> Property encodings.
+   * @task workers
+   */
+  protected function getCustomWorkerStateEncoding() {
+    return array();
+  }
+
+
   /**
    * Load editor state using a dictionary emitted by @{method:getWorkerState}.
    *
@@ -3097,7 +3133,10 @@ final public function loadWorkerState(array $state) {
     $exclude = idx($state, 'excludeMailRecipientPHIDs', array());
     $this->setExcludeMailRecipientPHIDs($exclude);
 
-    $custom = idx($state, 'custom', array());
+    $custom_state = idx($state, 'custom', array());
+    $custom_encodings = idx($state, 'custom.encoding', array());
+    $custom = $this->decodeStateFromStorage($custom_state, $custom_encodings);
+
     $this->loadCustomWorkerState($custom);
 
     return $this;
@@ -3143,16 +3182,85 @@ private function getAutomaticStateProperties() {
     );
   }
 
-  private function runHeraldMailRules(array $messages) {
-    foreach ($messages as $message) {
-      $engine = new HeraldEngine();
-      $adapter = id(new PhabricatorMailOutboundMailHeraldAdapter())
-        ->setObject($message);
+  /**
+   * Apply encodings prior to storage.
+   *
+   * See @{method:getCustomWorkerStateEncoding}.
+   *
+   * @param map<string, wild> Map of values to encode.
+   * @param map<string, string> Map of encodings to apply.
+   * @return map<string, wild> Map of encoded values.
+   * @task workers
+   */
+  final private function encodeStateForStorage(
+    array $state,
+    array $encodings) {
+
+    foreach ($state as $key => $value) {
+      $encoding = idx($encodings, $key);
+      switch ($encoding) {
+        case self::STORAGE_ENCODING_BINARY:
+          // The mechanics of this encoding (serialize + base64) are a little
+          // awkward, but it allows us encode arrays and still be JSON-safe
+          // with binary data.
+
+          $value = @serialize($value);
+          if ($value === false) {
+            throw new Exception(
+              pht(
+                'Failed to serialize() value for key "%s".',
+                $key));
+          }
 
-      $rules = $engine->loadRulesForAdapter($adapter);
-      $effects = $engine->applyRules($rules, $adapter);
-      $engine->applyEffects($effects, $adapter, $rules);
+          $value = base64_encode($value);
+          if ($value === false) {
+            throw new Exception(
+              pht(
+                'Failed to base64 encode value for key "%s".',
+                $key));
+          }
+          break;
+      }
+      $state[$key] = $value;
     }
+
+    return $state;
+  }
+
+
+  /**
+   * Undo storage encoding applied when storing state.
+   *
+   * See @{method:getCustomWorkerStateEncoding}.
+   *
+   * @param map<string, wild> Map of encoded values.
+   * @param map<string, string> Map of encodings.
+   * @return map<string, wild> Map of decoded values.
+   * @task workers
+   */
+  final private function decodeStateFromStorage(
+    array $state,
+    array $encodings) {
+
+    foreach ($state as $key => $value) {
+      $encoding = idx($encodings, $key);
+      switch ($encoding) {
+        case self::STORAGE_ENCODING_BINARY:
+          $value = base64_decode($value);
+          if ($value === false) {
+            throw new Exception(
+              pht(
+                'Failed to base64_decode() value for key "%s".',
+                $key));
+          }
+
+          $value = unserialize($value);
+          break;
+      }
+      $state[$key] = $value;
+    }
+
+    return $state;
   }
 
 }

src/applications/transactions/worker/PhabricatorApplicationTransactionPublishWorker.php

@@ -26,9 +26,14 @@ protected function doWork() {
    * Load the object the transactions affect.
    */
   private function loadObject() {
-    $data = $this->getTaskData();
     $viewer = PhabricatorUser::getOmnipotentUser();
 
+    $data = $this->getTaskData();
+    if (!is_array($data)) {
+      throw new PhabricatorWorkerPermanentFailureException(
+        pht('Task has invalid task data.'));
+    }
+
     $phid = idx($data, 'objectPHID');
     if (!$phid) {
       throw new PhabricatorWorkerPermanentFailureException(

src/infrastructure/storage/lisk/LiskDAO.php

@@ -1651,7 +1651,7 @@ protected function applyLiskDataSerialization(array &$data, $deserialize) {
             if ($deserialize) {
               $data[$col] = json_decode($data[$col], true);
             } else {
-              $data[$col] = json_encode($data[$col]);
+              $data[$col] = phutil_json_encode($data[$col]);
             }
             break;
           default: