CGCloud deploy docs #1279

Merged
merged 1 commit into from Nov 18, 2016

Conversation

Projects
None yet
3 participants
@jpdna
Member

jpdna commented Nov 17, 2016

No description provided.

@jpdna jpdna changed the title from CGCloud deply docs to CGCloud deploy docs Nov 17, 2016

@fnothaft

Couple of small nits, otherwise looks great! Thanks @jpdna!

docs/source/40_deploying_ADAM.md
+
+#### Launch a cluster
+
+Spin up a Spark cluster with one master and two slave nodes with the command:

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

Prefer leader/worker to master/slave.

Also, I would note in the documents that you're setting up a cluster where the workers are m3.large. Somewhat obvious, I concede, but it's useful to note that you can set a different leader node type. Also, doesn't this command need you to provide a cluster name?

@fnothaft

fnothaft Nov 17, 2016

Member

Prefer leader/worker to master/slave.

Also, I would note in the documents that you're setting up a cluster where the workers are m3.large. Somewhat obvious, I concede, but it's useful to note that you can set a different leader node type. Also, doesn't this command need you to provide a cluster name?

docs/source/40_deploying_ADAM.md
-export MY_KEYFILE="?????.pem"
-export MY_CLUSTER_NAME="adam_cluster"
-export MY_CLUSTER_SIZE=10
+[CGCloud](https://github.com/BD2KGenomics/cgcloud) lets you automate the creation, management and provisioning of VMs and clusters of VMs in Amazon EC2.

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

Can you wrap lines at 80 characters throughout?

@fnothaft

fnothaft Nov 17, 2016

Member

Can you wrap lines at 80 characters throughout?

docs/source/40_deploying_ADAM.md
+```
+cgcloud ssh spark-master
+```
+

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

Nit: extra whitespace.

@fnothaft

fnothaft Nov 17, 2016

Member

Nit: extra whitespace.

docs/source/40_deploying_ADAM.md
-Export the path to your `spark-ec2` script,
+To use the ADAM application on top of Spark, we need to download and install ADAM on `spark-master`
+From the command line on `spark-master` download a release from:
+https://github.com/bigdatagenomics/adam/releases

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

Nit: missing period at EOL.

@fnothaft

fnothaft Nov 17, 2016

Member

Nit: missing period at EOL.

docs/source/40_deploying_ADAM.md
-alias spark_ec2_login="$SPARK_EC2_SCRIPT -k $MY_KEYPAIR -i $MY_KEYFILE login $MY_CLUSTER_NAME"
+The typical flow of data to and from your ADAM application on EC2 will be:
+- Upload data to AWS S3
+- Use Conductor (described below) or otherwise transfer from S3 to the HDFS on your cluster

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

Can you add an anchor link {#conductor} in the section where conductor is described, and link from here (described below) -> [(described below)](#conductor). This'll make navigation a bit easier.

@fnothaft

fnothaft Nov 17, 2016

Member

Can you add an anchor link {#conductor} in the section where conductor is described, and link from here (described below) -> [(described below)](#conductor). This'll make navigation a bit easier.

docs/source/40_deploying_ADAM.md
+To transfer large amounts of data back and forth from S3, we suggest using [Conductor](https://github.com/BD2KGenomics/conductor).
+
+Its also possible to directly use AWS S3 as a distributed file system, but with some loss of performance.
+( example to be added )

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

Nit: I might drop the example to be added bit and remove the paragraph break between this paragraph and the conductor paragraph.

@fnothaft

fnothaft Nov 17, 2016

Member

Nit: I might drop the example to be added bit and remove the paragraph break between this paragraph and the conductor paragraph.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Nov 17, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1622/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1622/
Test PASSed.

@jpdna

This comment has been minimized.

Show comment
Hide comment
@jpdna

jpdna Nov 17, 2016

Member

ready for further review or merge

Member

jpdna commented Nov 17, 2016

ready for further review or merge

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Nov 17, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1623/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1623/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Nov 17, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1624/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1624/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Nov 17, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1625/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1625/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Nov 17, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1626/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1626/
Test PASSed.

@fnothaft

Few small nits, otherwise LGTM!

docs/source/40_deploying_ADAM.md
-alias spark_ec2_destroy="$SPARK_EC2_SCRIPT destroy $MY_CLUSTER_NAME"
-alias spark_ec2_login="$SPARK_EC2_SCRIPT -k $MY_KEYPAIR -i $MY_KEYFILE login $MY_CLUSTER_NAME"
+Spin up a Spark cluster named `cluster1` with one leader and two workers nodes
+of instance type `m3.large`with the command:

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

Space between words in m3.largewith

@fnothaft

fnothaft Nov 17, 2016

Member

Space between words in m3.largewith

docs/source/40_deploying_ADAM.md
+#### Install ADAM
+
+To use the ADAM application on top of Spark, we need to download and install
+ADAM on `spark-master`

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

period at EOL

@fnothaft

fnothaft Nov 17, 2016

Member

period at EOL

docs/source/40_deploying_ADAM.md
+To use the ADAM application on top of Spark, we need to download and install
+ADAM on `spark-master`
+From the command line on `spark-master` download a release from:
+https://github.com/bigdatagenomics/adam/releases

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

Punctuation at EOL? Maybe remove paragraph break.

@fnothaft

fnothaft Nov 17, 2016

Member

Punctuation at EOL? Maybe remove paragraph break.

docs/source/40_deploying_ADAM.md
+As of this writing, CGCloud supports Spark 1.6.2, not Spark 2.x, so download
+the Spark 1.x Scala2.10 release:
+```
+wget https://repo1.maven.org/maven2/org/bdgenomics/adam/\

This comment has been minimized.

@fnothaft

fnothaft Nov 17, 2016

Member

I would remove the \ed linebreak here.

@fnothaft

fnothaft Nov 17, 2016

Member

I would remove the \ed linebreak here.

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Nov 17, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1627/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1627/
Test PASSed.

Add CGCloud deploy doc
cgcloud doc edits

edits to cgcloud docs

more cgcloud edits

more cgcloud docs edits

more cgcloud docs edits

edit cgcloud docs

more cgcloud doc edits
@jpdna

This comment has been minimized.

Show comment
Hide comment
@jpdna

jpdna Nov 17, 2016

Member

ready again for more review or merge

Member

jpdna commented Nov 17, 2016

ready again for more review or merge

@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Nov 17, 2016

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1628/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1628/
Test PASSed.

@fnothaft fnothaft merged commit 20a0eb2 into bigdatagenomics:master Nov 18, 2016

1 check passed

default Merged build finished.
Details
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Nov 18, 2016

Member

Merged! Thanks @jpdna!

Member

fnothaft commented Nov 18, 2016

Merged! Thanks @jpdna!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment