End to End Encryption Support - Java client #731

saandrews · 2017-08-30T19:02:35Z

Motivation

Added Java client changes to encrypt messages published to pulsar. Once encrypted, it only allows the consumer with the right set of keys to be able to decrypt the original message published by the producer.

Modifications

Changes are implemented as per PIP-4.

Addresses issue #633

merlimat

Just did a very brief initial pass. Will look it thoroughly later today.

Have you looked at other existing crypto libraries, such as https://github.com/google/tink that support both C++ and Java and also have abstractions and hooks for key management?

merlimat · 2017-08-30T19:10:24Z

pulsar-common/src/main/proto/PulsarApi.proto

@@ -34,6 +34,11 @@ message KeyValue {
 	required string value = 2;
 }

+message KeyByteValue {


string and bytes types are effectively identical. We could use the KeyValue message type here.

I see. One of the recommendation is to use byte array when keys are stored. Since it's encrypted key, we should be ok with string. Will change it.

Yes, in both cases you get the ByteString which is essentially a byte array

Converting the encrypted key byte to String require it be encoded(e.g: Base64). For an encrypted key of size 256, the encoded value is 344 bytes long. Since this is sent in every message, it will unnecessarily increase the size. Unless there is a better way to convert byte array to string and back, I would prefer to use bytes

I don't think you need to used base64. In protobuf wire protocol, bytes and string are exactly the same type and they're also exposed in the same way in the Java API through ByteString class. ByteString can hold both.

You are right, ByteString can hold both. However, in our proto, the the "value" in KeyValue is defined as string, which can't hold ByteString.

In this use case, the result of encrypted key is a byte array which can be easily converted to ByteString without additional bytes. However, converting to String would require additional bytes.

Actually, just checked with Andrews and the type is different in Java (string --> String and bytes --> ByteString)

Thanks Matteo. Let us keep the KeyByteValue since it's efficient.

merlimat · 2017-08-30T19:11:40Z

pulsar-common/src/main/proto/PulsarApi.proto

@@ -60,6 +65,12 @@ message MessageMetadata {
 	//optional sfixed64 checksum = 10;
 	// differentiate single and batch message metadata
 	optional int32 num_messages_in_batch = 11 [default = 1];
+	// Comman separated encryption key name(s) used to encrypt data key


If encryption_keys is effectively a map, the comment should no mention "comma-separated"

Missed to update the comment. Will update it.

merlimat · 2017-08-30T19:18:36Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/ConsumerConfiguration.java

@@ -24,6 +24,8 @@
 import java.io.Serializable;
 import java.util.concurrent.TimeUnit;

+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;


This doesn't seem to be used

merlimat · 2017-08-30T19:30:39Z

pulsar-common/src/main/proto/PulsarApi.proto

@@ -110,6 +121,7 @@ enum ProtocolVersion {
 	v8 = 8;  // Added CommandConsumerStats - Client fetches broker side consumer stats
 	v9 = 9;  // Added end of topic notification
 	v10 = 10;// Added proxy to broker
+	v11 = 11;// Added encryption_key metadata


There's no need to bump up the protocol version if only adding new fields. Unknown fields are anyway ignored.

rdhabalia · 2017-08-31T01:12:05Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+            ClientCnx currentCnx) {
+
+        if (msgMetadata.getEncryptionKeysCount() == 0) {
+            return payload.retain();


Is there any reason to retain() payload in this method? because I don't see it released for this retain()?

Seems unnecessary as well.

I had to put it back since encryptedPayload is released explicitly.

rdhabalia · 2017-08-31T01:15:22Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+        if (conf.getCryptoKeyReader() == null) {
+
+            if (conf.getCryptoFailureAction() == ConsumerCryptoFailureAction.CONSUME) {
+                log.warn("CryptoKeyReader interface is not implemented. Consuming encrypted message.");


can we also log subscription and consumerName in msg?

rdhabalia · 2017-08-31T01:20:57Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+
+        // Create msgCrypto if not created already
+        if (this.msgCrypto == null) {
+            this.msgCrypto = new MessageCrypto(false);


though this method will not be called concurrently but as this method is not thread-safe, should we initialize msgCrypto in constructor if conf.getCryptoKeyReader() present.?

The idea is to create it on demand only when the incoming message has a key. But I agree, it's cleaner to do it in constructor.

rdhabalia · 2017-08-31T01:23:52Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+        }
+
+        log.error("[{}][{}] Failed to decrypt message {}", topic, subscription, messageId);
+        if (conf.getCryptoFailureAction() == ConsumerCryptoFailureAction.CONSUME) {


as this logic is duplicate, so should we create a method with this logic?

Slightly different message printed. Since it's just 6 lines, I would leave them.

rdhabalia · 2017-08-31T01:45:15Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+
+    }
+
+    private static final Logger log = LoggerFactory.getLogger(ProducerImpl.class);


MessageCrypto.class

Good catch. Fixed.

rdhabalia · 2017-09-05T21:37:56Z

pulsar-common/src/main/proto/PulsarApi.proto

 	}

-	optional ValidationError validation_error = 4;
+	optional ValidationError validation_error = 5;


We don't have to change field value here?

Isn't it supposed to be set to a value outside the ValidationError enum?

AFAIK - It is the field number, not the value - so needs to remain 4.

Why would we send BatchDeSerializeError if consumer received invalid message?

rdhabalia · 2017-09-06T17:27:02Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+
+        } catch (NoSuchAlgorithmException | NoSuchProviderException | NoSuchPaddingException e) {
+
+            cipher = null;


I think we may not need it as cipher is anyway null here.

KeyGenerator.getInstance() exception is also handled here. At that point, cipher is already initialized.

rdhabalia · 2017-09-06T18:49:15Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+            try {
+                addPublicKeyCipher(keyName, keyReader);
+            } catch (CryptoException e) {
+                throw new RuntimeException(e.getMessage());


I think we should throw Checked exception else, caller of this method ProducerImpl is not catching the exception and couldn't complete the future. So, client will keep waiting of the producer-create future.??

Good catch. Fixed.

rdhabalia · 2017-09-06T18:51:34Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+    private boolean decryptDataKey(String keyName, byte[] encryptedDataKey, CryptoKeyReader keyReader) {
+
+        // Prefix "private-key." into key name and read the key value using callback
+        String privKeyName = "private-key." + keyName;


is there any reason of adding prefix? do we have to document this convention so, CryptoKeyReader can implement it?

Yes, we need it to differentiate the public/private keys. I will update the doc.

rdhabalia · 2017-09-06T19:12:20Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+                ecParam = ECNamedCurveTable.getByOID(ecOID);
+                if (ecParam == null) {
+                    throw new PEMException("Unable to find EC Parameter for the given curve oid: "
+                            + ((ASN1ObjectIdentifier) pemObj).getId());


can avoid duplicate casting ecOID.getId() ?

Good catch, will fix it.

rdhabalia · 2017-09-06T20:53:07Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+        try (PEMParser pemReader = new PEMParser(keyReader)) {
+            X9ECParameters ecParam = null;
+
+            Object pemObj = pemReader.readObject();


PEMParser. readObject() can return null so, should we handle null value ?

rdhabalia · 2017-09-06T20:56:03Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+        if (keyName == null) {
+            return false;
+        }
+        encryptedDataKey.remove(keyName);


return encryptedDataKey.remove(keyName) == null?

rdhabalia · 2017-09-06T20:58:13Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+            throws PulsarClientException {
+
+        if (encKeys.isEmpty()) {
+            return payload.retain();


is there any reason of retaining the payload here?

It's not necessary.

rdhabalia · 2017-09-06T20:59:22Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java

+        try {
+            encryptedPayload = msgCrypto.encrypt(conf.getEncryptionKeys(), msgMetadata, compressedPayload);
+        } catch (PulsarClientException e) {
+            // TODO: Provide option to fail the request vs. proceed with warning


should we log here?

Added warning msg.

rdhabalia · 2017-09-06T21:28:13Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+        // Convert key from byte to PivateKey
+        PrivateKey privateKey;
+        try {
+            privateKey = loadPrivateKey(keyValue);


Should we cache this privateKey for a given key-name. Because, for a given key-name, value of the key will not be changed. Else we will end up calling loadPrivateKey for alternate message if we have multiple producers (because each producer has different dataKey which will fail at line: 479 and it forces to recalculate dataKey.

We could cache it. However, the data key is decrypted at the beginning and when the key changes, which is for every 100k/1m messages or every 4 hours(yet to add this logic).
Another reason is to be able to handle cases where the key value itself is changed at run time. If we cache it, we also have to store the key and verify if it's same.
Lastly keeping memory footprint of private key is a bad idea since memory dump could have its trace and could be a security issue.

rdhabalia · 2017-09-06T21:32:20Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+        if (dataKey != null) {
+            ByteBuf decryptedData = decryptData(msgMetadata, payload);
+            // If decryption succeeded, data is non null
+            if (decryptedData != null) {


should we keep cache for dataKey?
eg. if we have two producers and each has different dataKey then here on every msg coming from different producer will not be able to decrypt data and requires to derive dataKey again.?

Good point. Implemented cache with time based expiry based on last access.

saandrews · 2017-09-13T18:16:20Z

retest this please

saandrews · 2017-09-13T18:17:06Z

@merlimat @rdhabalia Do you have any more comments?

jai1

Gave it one look - seems good so far
Will go through MessageCrypto.java again before approving.

jai1 · 2017-09-13T17:18:02Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/ConsumerCryptoFailureAction.java

+    FAIL,    // This is the default option to fail consume until crypto succeeds 
+    DISCARD, // Message is silently acknowledged and not delivered to the application
+    CONSUME  // Deliver the encrypted message to the application. It's the application's
+             // responsibility to decrypt the message. If message is also compressed, decompression will fail   


Want to mention what will happen in case of batch messages??

Done. It's already explained in the design.

jai1 · 2017-09-13T17:21:08Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerConfiguration.java

+     * Returns true if encryption keys are added
+     *  
+     */
+    public boolean isEncryptionEnabled() {


Not sure if this method is required in Configuration class because usually we just have setters and getters here.

jai1 · 2017-09-13T22:37:34Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java

+            return decryptedData;
+        }
+
+        log.error("[{}][{}] Failed to decrypt message {}", topic, subscription, messageId);


Redundant logging-
Don't require this line if you add messageId in the log messages below.

jai1 · 2017-09-13T23:22:30Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerConfiguration.java

+     * If batch messaging is enabled, the batched message is encrypted.
+     *
+     */
+    public void addEncryptionKey(String key) {


Need to mention that this option is not to be used if batch messaging is enables and add appropriate asserts.

Batch message will work as long as proper keys are used. It's a problem only for cases with decryption failure, which is not very different from non batch message failure.

jai1 · 2017-09-13T23:25:44Z

pulsar-common/src/main/proto/PulsarApi.proto

 	}

-	optional ValidationError validation_error = 4;
+	optional ValidationError validation_error = 5;


AFAIK - It is the field number, not the value - so needs to remain 4.

jai1

LGTM - few minor suggestions

jai1 · 2017-09-13T23:40:16Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+import java.security.spec.InvalidKeySpecException;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;


Few imports not needed

jai1 · 2017-09-13T23:49:33Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+
+        Reader keyReader = new StringReader(new String(keyBytes));
+        PublicKey publicKey = null;
+        try (org.bouncycastle.openssl.PEMParser pemReader = new org.bouncycastle.openssl.PEMParser(keyReader)) {


Any reason for full class names????

jai1 · 2017-09-13T23:54:57Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+                pemObj = pemReader.readObject();
+            }
+
+            if (pemObj instanceof org.bouncycastle.cert.X509CertificateHolder) {


Any reason for full class names????

jai1 · 2017-09-13T23:57:40Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+                ECPublicKeySpec keySpec = new ECPublicKeySpec(((BCECPublicKey) publicKey).getQ(), ecSpec);
+                publicKey = (PublicKey) keyFactory.generatePublic(keySpec);
+            }
+        } catch (IOException | NoSuchAlgorithmException | NoSuchProviderException | InvalidKeySpecException e) {


Why not let this exception pass through - why to create a new one??

jai1 · 2017-09-14T00:27:51Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/CryptoKeyReader.java

+ */
+package org.apache.pulsar.client.api;
+
+public interface CryptoKeyReader {


I think we should mention that in case of Consumer the input will be "private-key." + keyName and "public-key." + keyName in case of producer

Because otherwise, the user will confuse it for the key name it gives in ProducerCong.addEncryptionKey

Good point. I have copied it from the doc here.

jai1 · 2017-09-14T00:43:57Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+
+        // Update message metadata with encrypted data key
+        encKeys.forEach(keyName -> {
+            msgMetadata.addEncryptionKeys(KeyByteValue.newBuilder().setKey(keyName)


May be add assertion to ensure the key name is in the encryptedDataKeyMap

jai1 · 2017-09-14T01:10:33Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+        KeyByteValue keyByteValue = encKeys.stream().filter(kbv -> {
+
+            byte[] encDataKey = kbv.getValue().toByteArray();
+            if (!decryptDataKey(kbv.getKey(), encDataKey, keyReader)) {


return decryptDataKey(kbv.getKey(), encDataKey, keyReader);

jai1 · 2017-09-14T05:15:15Z

retest this please

jai1 · 2017-09-14T17:33:56Z

retest this please

rdhabalia · 2017-09-14T17:38:27Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java

+            this.msgCrypto = new MessageCrypto(true);
+
+            // Regenerate data key cipher at fixed interval
+            keyGenExecutor = Executors.newSingleThreadScheduledExecutor();


it seems we are not closing this executor.??

rdhabalia · 2017-09-14T17:39:58Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java

+
+            // Regenerate data key cipher at fixed interval
+            keyGenExecutor = Executors.newSingleThreadScheduledExecutor();
+            keyGenExecutor.scheduleWithFixedDelay(new Runnable() {


instead we can directly use lambda

keyGenExecutor.scheduleWithFixedDela(()->{ //logic }, 0L, 4L, TimeUnit.HOURS);

jai1

Minor suggestions

jai1 · 2017-09-14T18:58:10Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java

+        try {
+            encryptedPayload = msgCrypto.encrypt(conf.getEncryptionKeys(), msgMetadata, compressedPayload);
+        } catch (PulsarClientException e) {
+            // TODO: Provide option to fail the request vs. proceed with warning


Remove TODO

jai1 · 2017-09-14T18:59:40Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java

+                        producerCreatedFuture.completeExceptionally(e);
+                    }
+                }
+            }, 0L, 4L, TimeUnit.HOURS);


Can this 4 hour be configurable in case the client is too paranoid and want to change the symmetric key more frequently?

jai1 · 2017-09-14T19:05:16Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.PooledByteBufAllocator;
+
+public class MessageCrypto {


Just a suggestion - Have a different class for Producer and Consumer Message Crypto since except for secureRandom the two flow have nothing in common:-
a. Producer uses loadPublicKey and encrypt function
b. Consumer uses loadPrivateKey and decrypt* functions

I think we should not separate it. One provider should do both encryption and decryption. eg: CompressionCodec, CheckSumProvider, Encoder .

rdhabalia · 2017-09-14T23:52:37Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/CryptoKeyReader.java

+     * called at the time of producer creation as well as consumer receiving messages.
+     * Hence, application should not make any blocking calls within the implementation. 
+     * While reading public key from the producer, the key name will have prefix "public-key." added to it.
+     * Similarly, to get private key from consumer, prefix "private-key." is used.


just a thought : instead adding prefix business, should we have two apis

byte [] getPrivateKey(String keyName); byte [] getPublicKey(String keyName);

I like this proposal. @msb-at-yahoo Do you have any objection?

rdhabalia · 2017-09-15T00:02:43Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java

+                try {
+                    msgCrypto.addPublicKeyCipher(conf.getEncryptionKeys(), conf.getCryptoKeyReader());
+                } catch (CryptoException e) {
+                    if (!producerCreatedFuture.isDone()) {


Should we log warn message, else there is no way to figure out if key stopped getting generated?

Added warn msg.

rdhabalia · 2017-09-15T18:55:26Z

@merlimat as we want to merge this PR as part of 1.20, can you also please review this PR when you get a chance.

rdhabalia

Overall LGTM.. just few last minor comments..

rdhabalia · 2017-09-15T22:52:23Z

pulsar-client/src/main/java/org/apache/pulsar/client/api/ProducerConfiguration.java

+     */
+    public void addEncryptionKey(String key) {
+        if (this.encryptionKeys == null) {
+            this.encryptionKeys = new ConcurrentOpenHashSet<String>(); 


by default concurrency is 16 for ConcurrentOpenHashSet which we may not need here. should we make it this.encryptionKeys = new ConcurrentOpenHashSet<>(256, 1);

We could even start with 16 entries, since most use case won't have more than one key.

rdhabalia · 2017-09-15T23:45:50Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+    public MessageCrypto(boolean keyGenNeeded) {
+
+        encryptedDataKeyMap = new ConcurrentHashMap<String, byte[]>();
+        dataKeyCache = CacheBuilder.newBuilder().expireAfterAccess(6, TimeUnit.HOURS)


if producer regenerates key at every 4 hours and should we also cache up to 4 hours only?

We would require few secs/mins more than 4hrs for ideal use case.

rdhabalia · 2017-09-15T23:49:15Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageCrypto.java

+                // Taking a small performance hit here if the hash collides. When it
+                // retruns a different key decryption fails. At this point, we would
+                // call decryptDataKey to refresh the cache and come here again to decrypt.
+                decryptedData = decryptData(storedSecretKey, msgMetadata, payload);


if key is not present then dataKeyCache returns null. so, instead decryptData(..) with null key, can't we just return if storedSecretKey==null?

I think we may use Optional if we are returning null intentionally..

null is also returned when decryption fails for other reasons. Also even if it fails with one key, we would attempt to decrypt using the remaining keys.

saandrews added this to the 1.20.0-incubating milestone Aug 30, 2017

saandrews requested review from merlimat, rdhabalia and jai1 August 30, 2017 19:02

merlimat reviewed Aug 30, 2017

View reviewed changes

merlimat added the type/feature The PR added a new feature or issue requested a new feature label Aug 30, 2017

merlimat assigned saandrews Aug 30, 2017

rdhabalia reviewed Aug 31, 2017

View reviewed changes

rdhabalia reviewed Sep 5, 2017

View reviewed changes

saandrews force-pushed the master branch from cb75849 to 41bf3bd Compare September 6, 2017 05:47

rdhabalia reviewed Sep 6, 2017

View reviewed changes

saandrews force-pushed the master branch 3 times, most recently from b2f6f83 to d1faece Compare September 11, 2017 18:04

jai1 reviewed Sep 13, 2017

View reviewed changes

saandrews force-pushed the master branch from d741aea to 430ac6f Compare September 14, 2017 00:13

jai1 reviewed Sep 14, 2017

View reviewed changes

saandrews force-pushed the master branch 2 times, most recently from a73aae5 to 72a9c18 Compare September 14, 2017 07:01

saandrews force-pushed the master branch from 72a9c18 to a460428 Compare September 14, 2017 17:34

rdhabalia reviewed Sep 14, 2017

View reviewed changes

saandrews force-pushed the master branch 3 times, most recently from 80d376d to 0e36311 Compare September 14, 2017 19:04

jai1 approved these changes Sep 14, 2017

View reviewed changes

rdhabalia reviewed Sep 14, 2017

View reviewed changes

rdhabalia reviewed Sep 15, 2017

View reviewed changes

saandrews force-pushed the master branch from 0e36311 to 6aec354 Compare September 15, 2017 00:29

apache deleted a comment from jai1 Sep 15, 2017

apache deleted a comment from saandrews Sep 15, 2017

saandrews force-pushed the master branch from 6aec354 to 880a654 Compare September 15, 2017 17:39

rdhabalia reviewed Sep 15, 2017

View reviewed changes

End to End Encryption Support - Java client

6772320

saandrews force-pushed the master branch from 06bbaeb to 04b6b55 Compare September 16, 2017 01:36

End to End Encryption Support - Java client

d71d117

saandrews force-pushed the master branch from 04b6b55 to d71d117 Compare September 16, 2017 03:12

saandrews merged commit b5bdd79 into apache:master Sep 16, 2017

eaba mentioned this pull request Oct 15, 2019

Support - Message encryption apache/pulsar-dotpulsar#8

Open

GPrabhudas mentioned this pull request Jun 24, 2021

Encryption support in Go client apache/pulsar-client-go#552

Closed


		}

		private static final Logger log = LoggerFactory.getLogger(ProducerImpl.class);


		} catch (NoSuchAlgorithmException \| NoSuchProviderException \| NoSuchPaddingException e) {

		cipher = null;

End to End Encryption Support - Java client #731

End to End Encryption Support - Java client #731

Conversation

saandrews commented Aug 30, 2017

Motivation

Modifications

merlimat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saandrews commented Sep 13, 2017

saandrews commented Sep 13, 2017

jai1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jai1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jai1 commented Sep 14, 2017

jai1 commented Sep 14, 2017