Migrate attachments from IndexedDB to file system #2129

gasi-signal · 2018-03-15T00:49:02Z

Relates to #1589.

gasi-signal

Annotated PR for your convenience, @scottnonnenberg-signal 😄

gasi-signal · 2018-03-20T23:13:33Z

.eslintignore

 !js/views/settings_view.js
-!js/backup.js
-!js/database.js


Just alphabetized it (using Sublime Sort Lines) and found some dupes.

gasi-signal · 2018-03-20T23:13:49Z

app/attachments.js

+const toArrayBuffer = require('to-arraybuffer');
+
+
+const PATH = 'attachments.noindex';


This is the piece that helps prevent indexing on macOS.

gasi-signal · 2018-03-20T23:15:03Z

app/attachments.js

+
+  const prefix = name.slice(0, 2);
+  return path.join(prefix, name);
+};


Apologies, this file was written before we had our discussion about module.exports so I haven’t converted it yet. Also, it’s nice to have => for some short functions without return which would require to have function keyword.

No worries, I'll amend my previous comment: exports.xxxx is totally reasonable since export xxx is the future. :0)

I'll get my preferred order in files when we move to that via export function xxx.

gasi-signal · 2018-03-20T23:16:23Z

js/models/conversations.js

@@ -656,10 +657,13 @@
          profileKey = storage.get('profileKey');
        }

+        const loadData = Attachment.loadData(migrationContext.readAttachmentData);
+        const attachmentsWithData =
+            await Promise.all(messageWithSchema.attachments.map(loadData));


I tried to surgically add places where we have to upgrade messages or load attachment metadata. I’d appreciate you thinking about places that I might have missed (and are therefore not in this PR) based on your extensive experience with the codebase — thanks 👀🙇

There are only a few potential sources which could create a new message:

incoming, from contact

incoming, sent to contact but from your own device

import

sending, created locally

There are a lot of places where 'sync' messages are sent, but those will never have attachments. Though perhaps you do want to upgrade their schema as well?

Thanks, that helps. I think I got (1) onMessageReceived, (2) onSentMessage, (4) Conversation::sendMessage. Let me look at (3) imported messages (Update: On hold until I get the app startup migration working.)

Re: sync messages. Good point. Indeed, right now they don’t have attachments but we may have other schema upgrades in the future. I’ll see if I can fit this in here, or at least do Message::initializeSchema so they’ll be picked up by our background task in the future.

gasi-signal · 2018-03-20T23:17:06Z

js/models/messages.js

+        initialize: function(attributes) {
+            if (_.isObject(attributes)) {
+                this.set(TypedMessage.initializeSchemaVersion(attributes));
+            }


Had to add if guard for when we call new Whisper.Message().

gasi-signal · 2018-03-20T23:17:55Z

js/models/messages.js

@@ -427,6 +444,7 @@
                        }
                    }
                    message.set({
+                        schemaVersion  : dataMessage.schemaVersion,


This is a place we have to look out for when adding new properties. Haven’t had time to think about how to make this less likely to be forgotten in the future.

What about in the constructor of the Message object?

I added Signal.Types.Message.initializeSchema to Whisper.Message::initialize constructor. But sadly that won’t cover handleDataMessage as it mutates Whisper.Message through this set call. In my experience, code is much easier to reason about when the constructor is the only place to construct a data type and it ensures the data type is valid after successful execution (Make Invalid States Impossible to Represent).

In fact, when you work with immutable data types, there simply is no such a thing as a set after the fact to mutate an existing ‘instance’. Yes, you can have set later but it returns a new data type. This means required fields (non-Maybes) have to be set in the constructor. Functions that do validation and data type construction are often called smart constructors. This article gives a good overview: https://github.com/paf31/purescript-book/blob/cf0b510052495e884059667e0585156cf68e354b/text/chapter14.md#smart-constructors

Hm. Maybe it would make sense to use a different technique to go from message to dataMessage? What keys don't we want to come through? What did you see when you spent some time with this part of the code?

Yes, ideally we’d only use constructors as the sole way to create an instance of a model so there is a single point to determine what data is valid vs not. Right now we have a mixture of new Whisper.Message(…) and later various set calls which makes it harder to reason about the final state of Message. For specifics, I’d have to spend more time with it but at a high-level I see a few problematic patterns:

We treat incoming, outgoing, verified-change, group update, end session, etc. messages all the same, i.e. as Whisper.Message. I understand we want to render them at the same time and persist them as a union (all properties flat and explicit type field) in messages store at the storage level, but it’d be nice to make a distinction at the app level. This would allow us to break down handleDataMessage into shorter functions, e.g. handleIncomingMessage, handleOutgoingMessage, handleVerifiedChangeMessage, etc. Happy to share how this would be modeled if we had support for union types.

We interweave pure construction of the data type as well as side-effects such as Message::save and Conversation::save. It’d be nice to break the two apart so we could test the side-effect free data type construction using unit tests.

What keys don't we want to come through?

For Whisper.Message specifically we seem to be doing fairly well to pick data vs rejecting it. Going through my own database, I found dataMessage: null in type: 'outgoing' which seems to be astray. Other than that, it’d be simply nice if there was a single new Whisper.Message(…) call that showed all the properties we initialize with.

gasi-signal · 2018-03-20T23:18:23Z

js/modules/types/attachment.js

+
+//      hasData :: Attachment -> Boolean
+exports.hasData = attachment =>
+  attachment.data instanceof ArrayBuffer || ArrayBuffer.isView(attachment.data);


One example where module.exports would make the function longer. But not a huge deal 😄

BTW, this definition of data is what window.crypto.subtle expects for encryption.

gasi-signal · 2018-03-20T23:19:53Z

js/modules/types/attachment/migrate_data_to_file_system.js

+
+  const attachmentWithoutData = Object.assign({}, attachment, { path });
+  delete attachmentWithoutData.data;
+  return attachmentWithoutData;


Grr, forgot to use omit again. Can fix it before merging.

It's not a big deal, I'm just honestly surprised that it's not second-nature for you! :0)

I didn’t start using Lodash until recently, that’s why. Deleting fields from a data type is also not something you do in statically typed languages 😉

Fixed it. Thanks for the tip. This is one example where yes, we could say that comment is not worth it but I appreciate you having made it and I learned something new 😄

gasi-signal · 2018-03-20T23:20:54Z

libtextsecure/sendmessage.js

+            '`attachment.data` must be an `ArrayBuffer` or `ArrayBufferView`; got: ' +
+            typeof attachment.data
+          ));
+        }


This helped improve our error messaging in case of problems and highlights the error sooner.

gasi-signal · 2018-03-20T23:21:32Z

package.json

-    "spectron": "^3.8.0"
+    "spectron": "^3.8.0",
+    "string-to-arraybuffer": "^1.0.0",
+    "tmp": "^0.0.33"


Oh, I guess tmp is now not needed since you added it to dependencies.

Removed it from devDependencies

scottnonnenberg-signal

Overall, great work. Some design things to think about, some questions about dependencies. And a strong recommendation to avoid ` in strings unless we really need it. :0)

scottnonnenberg-signal · 2018-03-22T21:45:39Z

app/attachments.js

+//      getPath :: AbsolutePath -> AbsolutePath
+exports.getPath = (userDataPath) => {
+  if (!isString(userDataPath)) {
+    throw new TypeError('`userDataPath` must be a string');


Hm. The use of ` to denote a name is bit confusing when scanning this text, since that character is used for strings as well. I don't thing we get enough value out of it here to justify that cost.

Can you clarify what the particular concern is? The rationale is to use the Markdown syntax we’re so used to as developers for denoting identifiers. I feel '\'userDataPath\' must be a string is noisier and just 'userDataPath must be a string' can get confusing once we have more generic arguments such as 'data must be a string', etc.

Backticks are harder for me to scan and understand. It's a twitch thing, and makes the code take more time to parse. IMO, markdown syntax is only useful when it gets converted to markdown.

I'm not really worried about a confusing error message like 'data must be a string' because we'll have the stack and therefore line numbers to help us out with addressing the error. If we're really looking for clarity, we might want to put the name of the method in the error.

scottnonnenberg-signal · 2018-03-22T21:51:26Z

js/models/conversations.js

@@ -656,10 +657,13 @@
          profileKey = storage.get('profileKey');
        }

+        const loadData = Attachment.loadData(migrationContext.readAttachmentData);
+        const attachmentsWithData =
+            await Promise.all(messageWithSchema.attachments.map(loadData));


There are only a few potential sources which could create a new message:

incoming, from contact

incoming, sent to contact but from your own device

import

sending, created locally

There are a lot of places where 'sync' messages are sent, but those will never have attachments. Though perhaps you do want to upgrade their schema as well?

scottnonnenberg-signal · 2018-03-22T21:57:19Z

js/models/messages.js

+          const attachments = this.get('attachments');
+          const deleteData =
+            Attachment.deleteData(migrationContext.deleteAttachmentData);
+          await Promise.all(attachments.map(deleteData));


As the last line of a function, how do we decide to use return or await? Is it because you explicitly don't want to make the results of those deleteData() calls available to any caller?

Yes, I want onDestroy to have return type void. I’ll add an explicit return; to make it clear 👍

scottnonnenberg-signal · 2018-03-22T21:59:17Z

js/models/messages.js

@@ -427,6 +444,7 @@
                        }
                    }
                    message.set({
+                        schemaVersion  : dataMessage.schemaVersion,


What about in the constructor of the Message object?

scottnonnenberg-signal · 2018-03-22T22:04:26Z

js/models/messages.js

+          this.revokeImageUrl();
+          const attachments = this.get('attachments');
+          const deleteData =
+            Attachment.deleteData(migrationContext.deleteAttachmentData);


In terms of API, I would like to have the final deleteData available to me here as an end-user, instead of needing to assemble it with context. I can see why we arrange things this way at declaration, keeping things pure, keeping things easy to test and compose. But I think we could simplify things here by having things be pre-assembled.

Agreed, that’d be nice. I will have to think about how to best expose it since the regular upgrade pipeline just needs the writeAttachmentData function on its own.

scottnonnenberg-signal · 2018-03-22T22:15:18Z

test/app/attachments_test.js

@@ -0,0 +1,105 @@
+const fse = require('fs-extra');
+const path = require('path');
+const stringToArrayBuffer = require('string-to-arraybuffer');


Looks to me like we're pulling this in for a very simple function:

function str2ab (str) { var array = new Uint8Array(str.length); for(var i = 0; i < str.length; i++) { array[i] = str.charCodeAt(i); } return array.buffer }

I feel like the risk we take on in adding another dependency (even if a dev dependency) is not worthwhile.

scottnonnenberg-signal · 2018-03-22T22:17:25Z

app/attachments.js

+
+  const prefix = name.slice(0, 2);
+  return path.join(prefix, name);
+};


No worries, I'll amend my previous comment: exports.xxxx is totally reasonable since export xxx is the future. :0)

I'll get my preferred order in files when we move to that via export function xxx.

scottnonnenberg-signal · 2018-03-22T22:21:41Z

package.json

@@ -77,6 +78,7 @@
    "spellchecker": "^3.4.4",
    "testcheck": "^1.0.0-rc.2",
    "tmp": "^0.0.33",
+    "to-arraybuffer": "^1.0.1",


Can you talk about the need for this dependency?

Unlike string-to-arraybuffer, at least this function is has a nice performance optimization we probably wouldn't have written otherwise: https://github.com/jhiesey/to-arraybuffer/blob/6502d9850e70ba7935a7df4ad86b358fc216f9f0/index.js#L3-L27

scottnonnenberg-signal · 2018-03-22T22:23:17Z

js/modules/types/attachment/migrate_data_to_file_system.js

+
+  const attachmentWithoutData = Object.assign({}, attachment, { path });
+  delete attachmentWithoutData.data;
+  return attachmentWithoutData;


It's not a big deal, I'm just honestly surprised that it's not second-nature for you! :0)

gasi-signal

Thanks for the review and the valuable feedback, especially pointing out where we create, send, and receive messages.

Re: to-arraybuffer. I need that to convert from Node.js Buffer (fs.readFile) to ArrayBuffer used in our UI.

gasi-signal · 2018-03-23T14:59:25Z

js/models/conversations.js

@@ -656,10 +657,13 @@
          profileKey = storage.get('profileKey');
        }

+        const loadData = Attachment.loadData(migrationContext.readAttachmentData);
+        const attachmentsWithData =
+            await Promise.all(messageWithSchema.attachments.map(loadData));


Thanks, that helps. I think I got (1) onMessageReceived, (2) onSentMessage, (4) Conversation::sendMessage. Let me look at (3) imported messages (Update: On hold until I get the app startup migration working.)

Re: sync messages. Good point. Indeed, right now they don’t have attachments but we may have other schema upgrades in the future. I’ll see if I can fit this in here, or at least do Message::initializeSchema so they’ll be picked up by our background task in the future.

gasi-signal · 2018-03-23T21:50:50Z

@scottnonnenberg-signal Added the following changes based on feedback: https://github.com/signalapp/Signal-Desktop/pull/2129/files/47df1d52a4005362cba339cfe24f1041c12dec2a..9d2706c59e18585f9d1ddd4e134aee24b4d749e9

gasi-signal · 2018-03-26T14:55:42Z

@scottnonnenberg Latest changes based on our discussion: https://github.com/signalapp/Signal-Desktop/pull/2129/files/9d2706c59e18585f9d1ddd4e134aee24b4d749e9..ead5d04f9f121a96172480c96d7fc26285a58e44

scottnonnenberg-signal

Looks great! That attachment API is now much cleaner at all the call sites. :0)

Allows us to use `fs` with promises. Turns out it’s already a transitive dependency.

Useful for creating random temporary directories for testing.

Used for testing attachment data migration.

Using hashes, we get the benefit of deduplication but if a user receives two messages with the same attachment, deleting one would delete it for both since they are only stored once. To avoid the complexity of tracking number of references, we simply generate random file names similar to iMessage on MacOS (?) and Signal Android.

We already use `tmp`.

Allows us to pass in Electron/Node.js specific functions. This allows better unit testing in Mocha where we don’t have access to Electron APIs.

Prevent double rendering of attachments by multiple entries into `MessageView::render` using promises.

See: https://apple.stackexchange.com/a/92785

Our tests rely on that.

This way we can avoid an extra dependency.

Makes `migrationContext` obsolete.

The function is simple enough to inline and this allows us to reduce our dependencies surface area.

- [x] Generate random file names. - [x] Generate random file paths that prevent too many files per folder using fan-out. - [x] Create attachment directory in user data folder. - [x] Investigate operating system file indexing on: - [x] Windows: Confirmed that `AppData` is not indexed by default. - [x] macOS: Confirmed that `~/Library` files are not indexed by default. Searching system files using Spotlight requires multi-step opt-in: https://lifehacker.com/5711409/how-to-search-for-hidden-packaged-and-system-files-in-os-x. More info https://apple.stackexchange.com/a/92785. Added `.noindex` suffix to `attachments` folder. - [x] Linux: n/a - [x] Save incoming attachment files to disk - [x] On received - [x] On sync - [x] Save outgoing attachments files to disk before sending - [x] Display attachments either from disk or memory in attachment view. Be robust to multiple render passes. - [x] Test that missing attachment on disk doesn’t break app. Behavior: Message is displayed without attachment. - [x] Delete attachment files when message is deleted. Relates to #1589.

gasi-signal · 2018-03-26T20:51:27Z

Rebased (no changes) and merged (before CI). Will address any unlikely CI issues after all three PRs are in development.

@colefranz

Update to electron 1.8.4 (#2186) Migrate all attachments from IndexedDB to file system in the background (#2208, #2193, #2165, #2162, #2129) Save attachments to disk when importing Chrome app export (#2212) Fixed: Read receipts setting would not be synchronized along with re-link (#2218) Fixed: Clicking conversation in left pane when already selected would remove focus on message composition field - thanks @colefranz! (#2032) Fixed: Searching for the phone number of an existing contact, then selecting 'start conversation' would erase contact details (#2191) Fixed: Selecting Settings menu option multiple times would open multiple instances of settings view - thanks @navdeepsinghkhalsa (#2167) Dev: - Relax Node.js version requirements (#2203) - Fix a few typos in documentation - thanks @Vinnl (#2171) - Update issue template to mention that translation should be via Transifex (#2157)

@colefranz

Update to electron 1.8.4 (signalapp#2186) Migrate all attachments from IndexedDB to file system in the background (signalapp#2208, signalapp#2193, signalapp#2165, signalapp#2162, signalapp#2129) Save attachments to disk when importing Chrome app export (signalapp#2212) Fixed: Read receipts setting would not be synchronized along with re-link (signalapp#2218) Fixed: Clicking conversation in left pane when already selected would remove focus on message composition field - thanks @colefranz! (signalapp#2032) Fixed: Searching for the phone number of an existing contact, then selecting 'start conversation' would erase contact details (signalapp#2191) Fixed: Selecting Settings menu option multiple times would open multiple instances of settings view - thanks @navdeepsinghkhalsa (signalapp#2167) Dev: - Relax Node.js version requirements (signalapp#2203) - Fix a few typos in documentation - thanks @Vinnl (signalapp#2171) - Update issue template to mention that translation should be via Transifex (signalapp#2157)

@colefranz

v1.7.0-beta.3 Update to electron 1.8.4 (#2186) Migrate all attachments from IndexedDB to file system in the background (#2208, #2193, #2165, #2162, #2129) Save attachments to disk when importing Chrome app export (#2212) Fixed: Read receipts setting would not be synchronized along with re-link (#2218) Fixed: Clicking conversation in left pane when already selected would remove focus on message composition field - thanks @colefranz! (#2032) Fixed: Searching for the phone number of an existing contact, then selecting 'start conversation' would erase contact details (#2191) Fixed: Selecting Settings menu option multiple times would open multiple instances of settings view - thanks @navdeepsinghkhalsa (#2167) Dev: - Relax Node.js version requirements (#2203) - Fix a few typos in documentation - thanks @Vinnl (#2171) - Update issue template to mention that translation should be via Transifex (#2157)

@colefranz

Note: this release is equivalent to v1.7.0-beta.3. The changes listed below are compared to the previous production release, v1.6.1. Update to electron 1.8.4 (#2186) Migrate all attachments from IndexedDB to file system in the background (#2208, #2193, #2165, #2162, #2129) Save attachments to disk when importing Chrome app export (#2212) New option in settings: delete all application data (383e02e, #2144 and #2153) Remove all configuration in database when we discover we are unlinked (9acb189 and 1c6d91b) Delete everything in database when we link with a different phone number from previous link (9acb189) Windows: Delete all data on uninstall (c855597) Fixed: Read receipts setting would not be synchronized along with re-link (#2218) Fixed: Clicking conversation in left pane when already selected would remove focus on message composition field - thanks @colefranz! (#2032) Fixed: Searching for the phone number of an existing contact, then selecting 'start conversation' would erase contact details (#2191) Fixed: Selecting Settings menu option multiple times would open multiple instances of settings view - thanks @navdeepsinghkhalsa (#2167) Dev: - Redact file paths in anything that goes to the log on disk (#2110) - When top-level process errors happen, don't show dialog with stack trace (#2110) - Add `nsp` to CI runs (fd056e1) - Add eslint-plugin-mocha to disallow exclusive tests using *.only (#2110) - Preparation for encrypted backups (cea42bd) - Updates to structure of exported data - messages.zip, flat list of attachments (6d8f4b7) - Relax Node.js version requirements (#2203) - Fix a few typos in documentation - thanks @Vinnl (#2171) - Update issue template to mention that translation should be via Transifex (#2157)

gasi-signal force-pushed the create-schema-version-message-index branch 2 times, most recently from 14aef88 to 55b7350 Compare March 16, 2018 18:37

gasi-signal added the WIP label Mar 16, 2018

gasi-signal force-pushed the migrate-attachments-to-file-system branch 2 times, most recently from d8d3199 to a677b31 Compare March 19, 2018 21:38

gasi-signal force-pushed the create-schema-version-message-index branch from 55b7350 to bde8436 Compare March 19, 2018 23:30

gasi-signal changed the base branch from create-schema-version-message-index to development March 20, 2018 18:35

gasi-signal force-pushed the migrate-attachments-to-file-system branch 3 times, most recently from 0aef6a2 to 47df1d5 Compare March 20, 2018 23:00

gasi-signal changed the title ~~WIP: Migrate attachments from IndexedDB to file system~~ Migrate attachments from IndexedDB to file system Mar 20, 2018

gasi-signal requested a review from scottnonnenberg-signal March 20, 2018 23:12

gasi-signal added Ready for Review and removed WIP labels Mar 20, 2018

gasi-signal commented Mar 20, 2018

View reviewed changes

scottnonnenberg-signal reviewed Mar 22, 2018

View reviewed changes

gasi-signal commented Mar 23, 2018

View reviewed changes

gasi force-pushed the migrate-attachments-to-file-system branch from 642f376 to ead5d04 Compare March 26, 2018 14:53

scottnonnenberg-signal approved these changes Mar 26, 2018

View reviewed changes

gasi-signal added 9 commits March 26, 2018 16:38

Add fs-extra as direct dependency

1442265

Allows us to use `fs` with promises. Turns out it’s already a transitive dependency.

Add tempy as dev dependency

925ddbd

Useful for creating random temporary directories for testing.

Add string-to-array-buffer as dev dependency

14f5828

Used for testing attachment data migration.

Add writeAttachmentData module

d9de6da

🎨 Use Buffer.from

8a94ca5

Remove tempy dependency

0fc2868

We already use `tmp`.

Add Attachment.migrateDataToFileSystem

ebe2a76

Add context to Message.upgradeSchema

aa82a2a

Allows us to pass in Electron/Node.js specific functions. This allows better unit testing in Mocha where we don’t have access to Electron APIs.

gasi-signal added 21 commits March 26, 2018 16:38

Remove duplicate ESLint ignore entry

f031ac3

Remove Vim modeline

b5721c2

Whitelist js/views/message_view.js for ESLint

97e3b49

Load attachment data before rendering

e1c1b1a

Prevent double rendering of attachments by multiple entries into `MessageView::render` using promises.

🎨 Use camelCase for mutable variable

aba2611

Remove Vim modeline

833a203

Use assert.deepEqual instead of Lodash isEqual

f6f0359

Add test for Attachments.readData

9b636a1

Add Attachments.deleteData

10afcf1

Add Attachment.deleteData

279b3f8

Delete attachment files on disk upon message delete

9d25aa4

Prevent potential NPE

b3251a8

Prevent macOS Spotlight indexing of attachments folder

add5d4a

See: https://apple.stackexchange.com/a/92785

Make MessageView::onUnload synchronous

ef2be2c

Our tests rely on that.

Use Lodash omit instead of delete

2ad0fd7

Remove tmp as dev dependency

232e906

Add stringToArrayBuffer utility

70cdd2b

This way we can avoid an extra dependency.

Use new stringToArrayBuffer function

06e7bca

Use double quotes for identifiers in error messages

867f73b

Wire up Message / Attachment migration functions on startup

0c06fff

Makes `migrationContext` obsolete.

Remove string-to-arraybuffer dependency

7eaa6ef

The function is simple enough to inline and this allows us to reduce our dependencies surface area.

gasi-signal force-pushed the migrate-attachments-to-file-system branch from a86ac5d to 7eaa6ef Compare March 26, 2018 20:43

gasi-signal merged commit 7eaa6ef into development Mar 26, 2018

gasi-signal deleted the migrate-attachments-to-file-system branch March 26, 2018 20:51

		const toArrayBuffer = require('to-arraybuffer');


		const PATH = 'attachments.noindex';

Migrate attachments from IndexedDB to file system #2129

Migrate attachments from IndexedDB to file system #2129

Conversation

gasi-signal commented Mar 15, 2018 • edited

gasi-signal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gasi-signal Mar 23, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scottnonnenberg-signal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gasi-signal Mar 23, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gasi-signal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gasi-signal commented Mar 23, 2018

gasi-signal commented Mar 26, 2018

scottnonnenberg-signal left a comment

Choose a reason for hiding this comment

gasi-signal commented Mar 26, 2018 • edited

gasi-signal commented Mar 15, 2018 •

edited

gasi-signal Mar 23, 2018 •

edited

gasi-signal Mar 23, 2018 •

edited

gasi-signal commented Mar 26, 2018 •

edited