Add s3 support for staging local dependencies in sparkctl #144 #179

mrow4a · 2018-06-07T12:43:58Z

Description

Adds support for staging local dependencies in sparkctl (solves #144 )

Implementation follows the guidelines from Google Storage Interoperability (https://cloud.google.com/storage/docs/interoperability) and S3 CLI docs (https://docs.aws.amazon.com/cli/latest/topic/s3-config.html)
By default, implementation uses --upload-to-endpoint set to https://storage.googleapis.com and --upload-to-region default empty region

To bo done

Implement S3 support
Adjust GCS
Update docs

How did I test

export AWS_ACCESS_KEY_ID=<redacted>
export AWS_SECRET_ACCESS_KEY=<redacted>
./sparkctl create \
./jobs/spark-pi.yaml \
--upload-to s3a://<gcs-bucket>

export GOOGLE_APPLICATION_CREDENTIALS=~/<redacted>.json
./sparkctl create \
./jobs/spark-pi.yaml \
--upload-to gs://<gcs-bucket>

@liyinan926 @prasanthkothuri

liyinan926 · 2018-06-10T19:43:26Z

@kow3ns.

mrow4a · 2018-06-11T15:01:58Z

Do you have some suggestions to the code itself? I am not sure if I can start adding documentation and update go-deps definitions, as I am not sure whether implementation needs some adjustments.

liyinan926 · 2018-06-11T15:51:59Z

@mrow4a GCS also supports regional access and allows users to specify a region. I plan to use the new flag region in this PR for that. However, the default storage class of GCS is multi-regional so by default region needs to be empty for GCS. What's the reason for forcing regional access for S3? Other than that, I think the code looks good in general.

liyinan926

Please run go fmt to format the code.

liyinan926 · 2018-06-11T15:54:27Z

sparkctl/cmd/s3.go

+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/s3"
+	"path/filepath"


Golang built in packages should be on top.

liyinan926 · 2018-06-11T15:54:44Z

sparkctl/cmd/s3.go

+	if err != nil {
+		return nil, err
+	}
+


Unnecessary extra line.

liyinan926 · 2018-06-11T15:57:00Z

sparkctl/cmd/s3.go

+
+
+	uploader := &s3Uploader{
+		client: s3manager.NewUploader(sess),


Does the client need to be closed when done?

I did not find any note that this is necessary in aws sdk

mrow4a · 2018-06-12T16:18:00Z

sparkctl/README.md

+```
+
+NOTE: In Spark 2.3.0 with init-containers used for Spark with Kubernetes as resource manager, 
+the dependencies are not mounted in proper location (e.g. application file is not mounted in jars folder)


It seems the problem I had previously with s3a deps - the init container incorrectly fetches (mounts) e.g. main application file. I think I need to investigate issue more with spark/master @liyinan926

Spark 2.4.0 (master branch) solves the issue

mrow4a · 2018-06-12T16:19:41Z

sparkctl/cmd/s3.go

+			return ""
+		}
+
+		return fmt.Sprintf("%s://%s.%s/%s",


I ensured to be fully compatible with AWS, and thus it requires <endpoint-protocol>://<bucket>.<endpoint-url>/path/to/file scheme.

mrow4a · 2018-06-13T12:41:39Z

@liyinan926 I have been debugging S3 issue with Spark 2.3.0

I found that the S3 dependencies are wrongly fetched because of spark-init container. I have been testing with Spark 2.4.0 (apache/spark:master) and all works fine. Please see the analysis log

Configuration

$ cat /opt/spark/conf/spark-defaults.conf

spark.hadoop.fs.s3a.endpoint      cs3.cern.ch
spark.hadoop.fs.s3a.access.key    <redacted>
spark.hadoop.fs.s3a.secret.key    <redacted>
spark.hadoop.fs.s3a.impl          org.apache.hadoop.fs.s3a.S3AFileSystem
spark.eventLog.enabled            true
spark.eventLog.dir                s3a://spark-cluster-test/spark-events
spark.history.fs.logDirectory     s3a://spark-cluster-test/spark-events
spark.logConf true

LOG with Spark 2.3.0 LOCAL (Docker image "gitlab-registry.cern.ch/db/spark-service/docker-registry/spark-on-k8s:v2.3.0-xrootd-s3"):

$ /opt/spark/bin/spark-submit --master local --class org.sparkservice.sparkrootapplications.examples.SparkPi s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar

2018-06-13 09:41:16 INFO  SparkContext:54 - Added JAR s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar at s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar with timestamp 1528875676885
2018-06-13 09:41:16 INFO  Executor:54 - Starting executor ID driver on host localhost
...
2018-06-13 09:41:18 INFO  Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2018-06-13 09:41:18 INFO  Executor:54 - Fetching s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar with timestamp 1528875676885
2018-06-13 09:41:18 INFO  Utils:54 - Fetching s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar to /tmp/spark-abb7b212-9329-42a4-bcf5-8653794db86b/userFiles-b81721c0-d093-4ba2-a893-2250b2ecb7af/fetchFileTemp8539294104865314818.tmp
2018-06-13 09:41:18 INFO  Executor:54 - Adding file:/tmp/spark-abb7b212-9329-42a4-bcf5-8653794db86b/userFiles-b81721c0-d093-4ba2-a893-2250b2ecb7af/spark-service-examples_2.11-0.0.1.jar to class loader
2018-06-13 09:41:18 INFO  Executor:54 - Finished task 0.0 in stage 0.0 (TID 0). 910 bytes result sent to driver

JOB COMPLETED WITH SUCCESS

LOG with Spark 2.3.0 KUBERNETES (Docker image "gitlab-registry.cern.ch/db/spark-service/docker-registry/spark-on-k8s:v2.3.0"):

$ kubectl describe sparkapplication spark-pi
    Name:         spark-pi
    Namespace:    default
    Labels:       <none>
    Annotations:  <none>
    API Version:  sparkoperator.k8s.io/v1alpha1
    Kind:         SparkApplication
    Spec:
        Image:                  gitlab-registry.cern.ch/db/spark-service/docker-registry/spark-on-k8s:v2.3.0-xrootd-s3
        Image Pull Policy:      Always
        Main Application File:  s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar
        Main Class:             org.sparkservice.sparkrootapplications.examples.SparkPi

$ kubectl logs spark-pi-dc261b23e2e730bfb8a883d1a4d67588-driver -c spark-init
...
2018-06-13 09:40:46 INFO  SparkPodInitContainer:54 - Downloading remote jars: Some(s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar,s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar)
2018-06-13 09:40:46 INFO  SparkPodInitContainer:54 - Downloading remote files: Some(s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/root4j-0.1.6.jar)
2018-06-13 09:40:48 INFO  SparkPodInitContainer:54 - Finished downloading application dependencies.
...

$ kubectl logs spark-pi-dc261b23e2e730bfb8a883d1a4d67588-driver
...
exec /sbin/tini -s -- /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/bin/java -Dspark.driver.cores=1.000000 -Dspark.kubernetes.executor.label.version=2.3.0 -Dspark.driver.memory=1024m -Dspark.kubernetes.driver.annotation.sparkoperator.k8s.io/ownerreference=ChBTcGFya0FwcGxpY2F0aW9uGghzcGFyay1waSIkMTA3NWI0YTEtNmVkZC0xMWU4LWIyYjgtZmExNjNlZjc2ZTdkKh1zcGFya29wZXJhdG9yLms4cy5pby92MWFscGhhMQ== -Dspark.app.name=spark-pi -Dspark.kubernetes.container.image.pullPolicy=Always -Dspark.kubernetes.executor.label.sparkoperator.k8s.io/app-name=spark-pi -Dspark.kubernetes.namespace=default -Dspark.driver.blockManager.port=7079 -Dspark.kubernetes.driver.limit.cores=1000m -Dspark.hadoop.fs.s3a.endpoint=<redacted> -Dspark.kubernetes.driver.label.sparkoperator.k8s.io/launched-by-spark-operator=true -Dspark.kubernetes.submission.waitAppCompletion=false -Dspark.kubernetes.driver.label.version=2.3.0 -Dspark.hadoop.fs.s3a.secret.key=<redacted> -Dspark.master=k8s://https://10.254.0.1:443 -Dspark.executor.cores=2 -Dspark.executor.memory=2048m -Dspark.kubernetes.container.image=gitlab-registry.cern.ch/db/spark-service/docker-registry/spark-on-k8s:v2.3.0-xrootd-s3 -Dspark.jars=s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar,s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar -Dspark.kubernetes.initContainer.configMapKey=spark-init.properties -Dspark.submit.deployMode=cluster -Dspark.hadoop.fs.s3a.access.key=<redacted> -Dspark.driver.host=spark-pi-dc261b23e2e730bfb8a883d1a4d67588-driver-svc.default.svc -Dspark.driver.port=7078 -Dspark.kubernetes.authenticate.driver.serviceAccountName=spark -Dspark.history.fs.update.interval=1s -Dspark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem -Dspark.kubernetes.driver.pod.name=spark-pi-dc261b23e2e730bfb8a883d1a4d67588-driver -Dspark.kubernetes.driver.label.sparkoperator.k8s.io/app-id=spark-pi-2416137625 -Dspark.eventLog.dir=s3a://spark-cluster-test/spark-events -Dspark.eventLog.enabled=true -Dspark.kubernetes.executor.label.sparkoperator.k8s.io/launched-by-spark-operator=true -Dspark.kubernetes.initContainer.configMapName=spark-pi-dc261b23e2e730bfb8a883d1a4d67588-init-config -Dspark.executor.instances=1 -Dspark.app.id=spark-03326e75bafc4620b300ba8f5adbb67e -Dspark.history.fs.logDirectory=s3a://spark-cluster-test/spark-events -Dspark.kubernetes.executor.podNamePrefix=spark-pi-dc261b23e2e730bfb8a883d1a4d67588 -Dspark.files=s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/root4j-0.1.6.jar -Dspark.kubernetes.driver.label.sparkoperator.k8s.io/app-name=spark-pi -Dspark.kubernetes.executor.label.sparkoperator.k8s.io/app-id=spark-pi-2416137625 -cp ':/opt/spark/jars/*:/var/spark-data/spark-jars/spark-service-examples_2.11-0.0.1.jar:/var/spark-data/spark-jars/spark-service-examples_2.11-0.0.1.jar' -Xms1024m -Xmx1024m -Dspark.driver.bindAddress=10.100.41.3 org.sparkservice.sparkrootapplications.examples.SparkPi
Error: Could not find or load main class org.sparkservice.sparkrootapplications.examples.SparkPi
...
JOB COMPLETED WITH ERROR (Could not find or load main class org.sparkservice.sparkrootapplications.examples.SparkPi)

LOG with Spark 2.4.0 KUBERNETES (Docker image "gitlab-registry.cern.ch/db/spark-service/docker-registry/spark-on-k8s:master"):

$ kubectl logs spark-pi-1528892994323-driver -c spark-init
Error from server (BadRequest): container spark-init is not valid for pod spark-pi-1528892994323-driver

$ kubectl logs spark-pi-1528892994323-driver
2018-06-13 14:30:05 INFO  SparkContext:54 - Added JAR s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar at s3a://spark-cluster-test/spark-app-dependencies/default/spark-pi/spark-service-examples_2.11-0.0.1.jar with timestamp 1528893005323

JOB COMPLETED WITH SUCCESS

liyinan926 · 2018-07-03T12:54:55Z

@mrow4a just came back from vacation. Will take a look soon.

mrow4a · 2018-07-03T13:04:26Z

@liyinan926 You can try with S3 interoperability - but generally I got error in Spark 2.3 on fetching from s3, and needed to build Spark from master branch (without init containers).

liyinan926 · 2018-07-27T22:00:18Z

@mrow4a This is worth taking a look at: https://github.com/google/go-cloud. It can potentially simplifies the code around uploading to GCS/S3.

mrow4a · 2018-07-28T09:17:52Z

@liyinan926 will check it, so far looks promising.

mrow4a · 2018-08-01T12:36:30Z

@liyinan926 I spent some time on implementing go-cloud, and it is very nice as it allows to cut a lot of code for uploading. However, because go-cloud does not support setting ACLs (public read), the code becomes a bit complicated.

Should I make common code for upload, and dedicated for setting ACLs? (I guess at some point package will have ACL support)

I will add new commit here to show-case it

mrow4a · 2018-08-01T16:22:00Z

@liyinan926 could you have a look into s3 implementation (I for now skipped GSC)

liyinan926 · 2018-08-01T16:25:26Z

@mrow4a I will take a look soon.

mrow4a · 2018-08-02T09:53:54Z

@liyinan926 I have been playing with GCS client, and this is not trivial to set ACLs (meaning that cloud-go does not support it, and two HTTPClients have to be maintained).

mrow4a · 2018-08-02T12:32:02Z

@liyinan926 I added and tested the support for S3A and GCS using go-cloud (including documentation) - please review.

Add aws sdk as dependency

mrow4a · 2018-08-06T13:03:04Z

I did rebase and updated PR Description in the top.

liyinan926 · 2018-08-06T15:37:57Z

sparkctl/cmd/create.go

@@ -31,15 +31,19 @@ import (
 	"k8s.io/apimachinery/pkg/util/yaml"
 	clientset "k8s.io/client-go/kubernetes"

+	"context"


These two imports should be in their respective groups.

liyinan926 · 2018-08-06T15:39:12Z

sparkctl/cmd/create.go

+	fileName := filepath.Base(localFilePath)
+	uploadFilePath := filepath.Join(uploadPath, fileName)
+	// Check if exists by trying to fetch metadata
+	_, err := bh.bucket.NewRangeReader(bh.ctx, uploadFilePath, 0, 0)


We need to make sure the reader gets closed.

reader, err := bh.bucket.NewRangeReader(bh.ctx, uploadFilePath, 0, 0) defer reader.Close()

liyinan926 · 2018-08-06T15:39:53Z

sparkctl/cmd/create.go

+		if err != nil {
+			return "", fmt.Errorf("Failed to read file: %s", err)
+		}
+


Need to add defer data.Close().

data is not a handle, is actual bytes (buf.Bytes())

liyinan926 · 2018-08-06T15:40:28Z

sparkctl/cmd/create.go

+			return "", fmt.Errorf("Failed to read file: %s", err)
+		}
+
+		w, err := bh.bucket.NewWriter(bh.ctx, uploadFilePath, nil)


Ditto. Need to make sure w gets closed, by adding defer w.Close() after the error check below.

liyinan926 · 2018-08-06T15:43:29Z

sparkctl/cmd/create.go

 		return nil, fmt.Errorf(
 			"unable to upload local dependencies: no upload location specified via --upload-to")
 	}

-	uploadLocationUrl, err := url.Parse(UploadTo)
+	var bh *BlobHandler


Looks like bh and ctx are not needed until the start of the switch block.

liyinan926 · 2018-08-06T15:56:03Z

sparkctl/cmd/gcs.go

+func (blob GCSBlob) setPublicACL(
+	ctx context.Context,
+	bucket string,
+	filePath string) error {
 	client, err := storage.NewClient(ctx)


So we still need to use this gcs-specific client for setting public ACL?

Yes. This already kills a lot of redundant code, but ACL are not implemented in cloud-go yet. This cloud-go seems a very young project.

OK, can we add a TODO to get rid of this gcs-specific code once cloud-go adds support for that?

puneetloya · 2018-08-07T14:55:23Z

this change would let spark use s3 as a hadoop endpoint(checkpoint), yes?

liyinan926 · 2018-08-07T15:49:49Z

sparkctl/cmd/create.go

+	uploadFilePath := filepath.Join(uploadPath, fileName)
+	// Check if exists by trying to fetch metadata
+	reader, err := bh.bucket.NewRangeReader(bh.ctx, uploadFilePath, 0, 0)
+	defer reader.Close()


It looks like you don't even need to use defer here since it's closed immediately after being created.

Not sure if I understand.

You don't need to use defer as reader is not used anywhere below, it's closed right after it's created.

liyinan926 · 2018-08-07T15:52:01Z

sparkctl/cmd/create.go

+		// Write data to bucket and close bucket writer
+		_, writeErr := w.Write(data)
+		if err := w.Close(); err != nil {
+			return "", fmt.Errorf("Failed to close bucket writer: %s", err)


We use lower case letter for the first word of the error string in this package.

liyinan926 · 2018-08-07T15:52:22Z

sparkctl/cmd/create.go

+
+		// Check if write has been successful
+		if writeErr != nil {
+			return "", fmt.Errorf("Failed to write to bucket: %s", err)


Ditto. Failed -> failed.

mrow4a · 2018-08-07T18:28:30Z

@puneetloya It will allow to do the same as for GCS. Specificaly, to upload your local dependency to your AWS S3 or other S3 compatible storage using a aws-go-sdk via s3://path. It will also add to job definition your uploaded dependency with url s3a://path. Of course you need to have your s3a access and key specified in hadoopConf or sparkConf. S3 is a objectstorage endpoint, S3A is hadoop-s3-connector.

@liyinan926 I think not to confuse people, I need to add scheme as s3, but translate it to s3a.

liyinan926 · 2018-08-07T18:55:10Z

@liyinan926 I think not to confuse people, I need to add scheme as s3, but translate it to s3a.

That's a good idea.

mrow4a · 2018-08-08T13:05:04Z

Clarified the README and adjusted to review @liyinan926

liyinan926

LGTM after the latest comments get addressed.

liyinan926 · 2018-08-08T16:34:50Z

sparkctl/cmd/create.go

+	fileName := filepath.Base(localFilePath)
+	uploadFilePath := filepath.Join(uploadPath, fileName)
+	// Check if exists by trying to fetch metadata
+	_, err := uh.b.NewRangeReader(uh.ctx, uploadFilePath, 0, 0)


Sorry if I didn't make it clear. You need to close the returned Reader, but you don't need to use defer reader.Close(). Simply do the following is fine.

reader, err := uh.b.NewRangeReader(uh.ctx, uploadFilePath, 0, 0) reader.Close()

Oh, ok - now I got your point.

liyinan926 · 2018-08-08T16:37:08Z

sparkctl/README.md

+spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
+```
+
+NOTE: In Spark 2.3.0 init-containers are used for Kubernetes as resource manager. In future versions, init-containers are removed. 


for Kubernetes as resource manager -> for downloading remote application dependencies. Also this applies to any Spark 2.3.x versions.

liyinan926 · 2018-08-08T16:40:20Z

sparkctl/cmd/create.go

@@ -267,28 +272,118 @@ func isContainerLocalFile(file string) (bool, error) {
 	return false, nil
 }

+type Blob interface {


Looks like both Blob and UploadHandler are only used inside the cmd package, so they should both start with lower-case letters.

Add documentation for staging local dependencies in s3a

liyinan926 mentioned this pull request Jun 10, 2018

Spark operator roadmap for 2018 #40

Closed

13 tasks

liyinan926 reviewed Jun 11, 2018

View reviewed changes

mrow4a force-pushed the upstream_s3_support branch 2 times, most recently from f401977 to e1bbd0d Compare June 12, 2018 16:16

mrow4a commented Jun 12, 2018

View reviewed changes

mrow4a mentioned this pull request Jun 14, 2018

Add s3 support for staging local dependencies in sparkctl cerndb/spark-on-k8s-operator#12

Merged

mrow4a force-pushed the upstream_s3_support branch from a923b09 to d480f30 Compare June 14, 2018 14:07

mrow4a force-pushed the upstream_s3_support branch from d480f30 to afb6bab Compare July 27, 2018 12:46

mrow4a force-pushed the upstream_s3_support branch from afb6bab to d0e5053 Compare July 30, 2018 15:43

mrow4a force-pushed the upstream_s3_support branch from d0e5053 to 75b7b7e Compare August 1, 2018 16:19

mrow4a changed the title ~~Add s3 support for staging local dependencies in sparkctl #144~~ [WIP] Add s3 support for staging local dependencies in sparkctl #144 Aug 1, 2018

mrow4a force-pushed the upstream_s3_support branch from 75b7b7e to 205184d Compare August 2, 2018 12:13

mrow4a changed the title ~~[WIP] Add s3 support for staging local dependencies in sparkctl #144~~ Add s3 support for staging local dependencies in sparkctl #144 Aug 2, 2018

mrow4a force-pushed the upstream_s3_support branch 2 times, most recently from 070ca2c to 3730b25 Compare August 2, 2018 12:30

mrow4a force-pushed the upstream_s3_support branch from 3730b25 to 87415c6 Compare August 2, 2018 12:39

mrow4a force-pushed the upstream_s3_support branch 4 times, most recently from df7a8c1 to 405b290 Compare August 3, 2018 09:14

Add s3a support in sparkctl for staging local dependencies

bee38ec

Add aws sdk as dependency

mrow4a force-pushed the upstream_s3_support branch from 405b290 to c352e1c Compare August 6, 2018 13:01

liyinan926 reviewed Aug 6, 2018

View reviewed changes

mrow4a force-pushed the upstream_s3_support branch from c352e1c to f2aa39b Compare August 7, 2018 09:49

liyinan926 reviewed Aug 7, 2018

View reviewed changes

mrow4a force-pushed the upstream_s3_support branch from f2aa39b to 5d4edd7 Compare August 8, 2018 13:04

liyinan926 reviewed Aug 8, 2018

View reviewed changes

Adjust upload-to to go-cloud

23f295d

Add documentation for staging local dependencies in s3a

mrow4a force-pushed the upstream_s3_support branch from 5d4edd7 to 23f295d Compare August 9, 2018 12:39

liyinan926 merged commit 8d95f53 into kubeflow:master Aug 9, 2018

mrow4a deleted the upstream_s3_support branch August 9, 2018 16:04



		uploader := &s3Uploader{
		client: s3manager.NewUploader(sess),

Add s3 support for staging local dependencies in sparkctl #144 #179

Add s3 support for staging local dependencies in sparkctl #144 #179

Conversation

mrow4a commented Jun 7, 2018 • edited Loading

Description

To bo done

How did I test

liyinan926 commented Jun 10, 2018

mrow4a commented Jun 11, 2018 • edited Loading

liyinan926 commented Jun 11, 2018 • edited Loading

liyinan926 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrow4a commented Jun 13, 2018 • edited Loading

liyinan926 commented Jul 3, 2018

mrow4a commented Jul 3, 2018

liyinan926 commented Jul 27, 2018

mrow4a commented Jul 28, 2018

mrow4a commented Aug 1, 2018 • edited Loading

mrow4a commented Aug 1, 2018

liyinan926 commented Aug 1, 2018

mrow4a commented Aug 2, 2018

mrow4a commented Aug 2, 2018

mrow4a commented Aug 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

puneetloya commented Aug 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrow4a commented Aug 7, 2018

liyinan926 commented Aug 7, 2018

mrow4a commented Aug 8, 2018

liyinan926 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrow4a commented Jun 7, 2018 •

edited

Loading

mrow4a commented Jun 11, 2018 •

edited

Loading

liyinan926 commented Jun 11, 2018 •

edited

Loading

mrow4a commented Jun 13, 2018 •

edited

Loading

mrow4a commented Aug 1, 2018 •

edited

Loading