README for Shunting Yard #35

abhimanyugupta07 · 2019-04-12T15:38:24Z

First attempt at an exhaustive README for Shunting Yard.

coveralls · 2019-04-12T15:42:21Z

Pull Request Test Coverage Report for Build 397

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 78.632%

Totals
Change from base Build 363:	0.0%
Covered Lines:	736
Relevant Lines:	936

💛 - Coveralls

AnanaMJ · 2019-04-12T15:49:03Z

README.md

@@ -1,61 +1,147 @@
 # Shunting Yard

-A Spring Boot app that reads serialized Hive MetaStore Events and builds a YAML file with the information provided in the event which is then passed to [Circus Train](https://github.com/HotelsDotCom/circus-train) to perform the replication.
+Shunting Yard reads serialized Hive MetaStore Events from a queue (currently supports [AWS SQS](https://aws.amazon.com/sqs/)) and replicates the data between two Data lakes. It does this by building a YAML file with the information provided in the event which is then passed to [Circus Train](https://github.com/HotelsDotCom/circus-train) to perform the replication.


"between two Data lakes": does "data" have to be Uppercase?

AnanaMJ · 2019-04-12T15:53:59Z

README.md

-    hive.metastore.event.listeners = com.hotels.shunting.yard.event.emitter.sqs.listener.SqsMetaStoreEventListener
-    com.hotels.shunting.yard.event.emitter.sqs.queue = https://sqs.<region>.amazonaws.com/<account-id>/<topic-name>-queue.fifo
-    com.hotels.shunting.yard.event.emitter.sqs.group.id = <group-id>
+Note that the paths above are correct as of when this document was last updated but may differ across EMR versions, refer to the [EMR release guide](http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html) for more up to date information if necessary.


There should be a full stop instead of comma, as there seem to be two separate sentences in the same sentence.

AnanaMJ · 2019-04-12T15:57:38Z

README.md

+          database-name: replica_database
+          table-name: test_table_1    
+
+#### Only Change the target database but the table name remains same as source


"change" with lowercase "c"

And maybe it would be good to rephrase this as "Change only the target database while keeping the same source table name" to keep the same tone of the writing (as an instruction)

what about the second half?

We talked about this and I'm ok with what's in the document now :)

AnanaMJ · 2019-04-12T15:59:58Z

README.md

+        replica-table:
+          database-name: replica_database 
+
+#### Only Change the target table name but the database remains same as source


Same thing here, but the database and table switched around :)

AnanaMJ · 2019-04-12T16:03:26Z

README.md

+|`source-catalog.hive-metastore-uris`|No|Fully qualified URI of the source cluster's Hive metastore Thrift service.|
+|`replica-catalog.name`|Yes|A name for the replica catalog for events and logging.|
+|`replica-catalog.hive-metastore-uris`|Yes|Fully qualified URI of the replica cluster's Hive metastore Thrift service.|
+|`event-receiver.configuration-properties.com.hotels.shunting.yard.event.receiver.sqs.queue`|Yes|Fully qualified URI of the [AWS SQS](https://aws.amazon.com/sqs/) Queue to read the hive events from.|


Uppercase H on "hive"

AnanaMJ · 2019-04-12T16:04:40Z

README.md

+|`replica-catalog.name`|Yes|A name for the replica catalog for events and logging.|
+|`replica-catalog.hive-metastore-uris`|Yes|Fully qualified URI of the replica cluster's Hive metastore Thrift service.|
+|`event-receiver.configuration-properties.com.hotels.shunting.yard.event.receiver.sqs.queue`|Yes|Fully qualified URI of the [AWS SQS](https://aws.amazon.com/sqs/) Queue to read the hive events from.|
+|`event-receiver.configuration-properties.com.hotels.shunting.yard.event.receiver.sqs.wait.time.seconds`|No|Wait time in seconds for which the receiver will poll the SQS queue for a batch of messages. Default is 10 seconds. Read more about long polling with AWS SQS [here](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html)|


Full stop at the end 🙂

AnanaMJ · 2019-04-12T16:05:46Z

README.md

+|`replica-catalog.hive-metastore-uris`|Yes|Fully qualified URI of the replica cluster's Hive metastore Thrift service.|
+|`event-receiver.configuration-properties.com.hotels.shunting.yard.event.receiver.sqs.queue`|Yes|Fully qualified URI of the [AWS SQS](https://aws.amazon.com/sqs/) Queue to read the hive events from.|
+|`event-receiver.configuration-properties.com.hotels.shunting.yard.event.receiver.sqs.wait.time.seconds`|No|Wait time in seconds for which the receiver will poll the SQS queue for a batch of messages. Default is 10 seconds. Read more about long polling with AWS SQS [here](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html)|
+|`source-table-filter.table-names`|No|A list of tables selected for Shunting Yard replication. Supported Format:`database_1.table_1, database_2.table_2`|


space after colon:
"format: database_1.table_1, database_2.table_2"

and lowercase f on "Format"

There's still no space after colon

It is there now :)

AnanaMJ · 2019-04-12T16:08:36Z

README.md

+|`table-replications[n].replica-table.database-name`|No|The name of the destination database in which to replicate the table. Defaults to source database name.|
+|`table-replications[n].replica-table.table-name`|No|The name of the table at the destination. Defaults to source table name.|
+
+### Configuring Graphite Metrics


I think "Metrics" should be with lowercase m. "System architecture" has 2 hashes (compared to 3 here) and doesn't have uppercase for each letter in the subtitle, so this shouldn't either.

Graphite can stay with uppercase, because it's the name of the tool, right?

AnanaMJ

I definitely wasn't looking for a missing full stop on purpose :)

abhimanyugupta07 · 2019-04-12T16:18:30Z

I definitely wasn't looking for a missing full stop on purpose :)

I think you were 🙈

massdosage

Good skeleton, just needs some fleshing out.

massdosage · 2019-04-12T18:12:16Z

README.md

@@ -1,6 +1,6 @@
 # Shunting Yard

-Shunting Yard reads serialized Hive MetaStore Events from a queue (currently supports [AWS SQS](https://aws.amazon.com/sqs/)) and replicates the data between two Data lakes. It does this by building a YAML file with the information provided in the event which is then passed to [Circus Train](https://github.com/HotelsDotCom/circus-train) to perform the replication.
+Shunting Yard reads serialized Hive MetaStore Events from a queue (currently supports [AWS SQS](https://aws.amazon.com/sqs/)) and replicates the data between two data lakes. It does this by building a YAML file with the information provided in the event which is then passed to [Circus Train](https://github.com/HotelsDotCom/circus-train) to perform the replication.


currently supports AWS SQS -> AWS SQS is currently supported

massdosage · 2019-04-12T18:15:19Z

README.md


-You can obtain Shunting Yard from Maven Central:
+1. Download the version to use from [Maven Central](https://mvnrepository.com/artifact/com.hotels/shunting-yard-binary) and uncompress it in a directory of your choosing.


It's a bit unclear here which file one should actually download, for a new user we really need to spell it out for them. I know this is what it says in the Circus Train docs but that could also be improved (once we get the wording sorted out here let's do the same for Circus Train). I think Waggle Dance is a better example of this: https://github.com/HotelsDotCom/waggle-dance/#install

massdosage · 2019-04-12T18:15:58Z

README.md


-[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.hotels/shunting-yard/badge.svg?subject=com.hotels:shunting-yard)](https://maven-badges.herokuapp.com/maven-central/com.hotels/shunting-yard) [![Build Status](https://travis-ci.org/HotelsDotCom/shunting-yard.svg?branch=master)](https://travis-ci.org/HotelsDotCom/shunting-yard) [![Coverage Status](https://coveralls.io/repos/github/HotelsDotCom/shunting-yard/badge.svg?branch=master)](https://coveralls.io/github/HotelsDotCom/shunting-yard?branch=master) ![GitHub license](https://img.shields.io/github/license/HotelsDotCom/shunting-yard.svg)
+2. Download and install the latest version of [Circus Train](http://mvnrepository.com/artifact/com.hotels/circus-train/) and set the `CIRCUS_TRAIN_HOME` environment variable:


Once we sort out the "Install" section for Circus Train let's rather link to the section in its README instead of duplicating links to Maven repos etc. here.

massdosage · 2019-04-12T18:19:11Z

README.md

-[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.hotels/shunting-yard/badge.svg?subject=com.hotels:shunting-yard)](https://maven-badges.herokuapp.com/maven-central/com.hotels/shunting-yard) [![Build Status](https://travis-ci.org/HotelsDotCom/shunting-yard.svg?branch=master)](https://travis-ci.org/HotelsDotCom/shunting-yard) [![Coverage Status](https://coveralls.io/repos/github/HotelsDotCom/shunting-yard/badge.svg?branch=master)](https://coveralls.io/github/HotelsDotCom/shunting-yard?branch=master) ![GitHub license](https://img.shields.io/github/license/HotelsDotCom/shunting-yard.svg)
+2. Download and install the latest version of [Circus Train](http://mvnrepository.com/artifact/com.hotels/circus-train/) and set the `CIRCUS_TRAIN_HOME` environment variable:
+
+         export CIRCUS_TRAIN_HOME=/home/hadoop/circus-train-<circus-train-version>


Should we use /opt as a base instead of /home/hadoop in the examples?

massdosage · 2019-04-12T18:20:01Z

README.md


-On the target cluster, download and install the latests version of Circus Train and set the `CIRCUS_TRAIN_HOME` environment variable:
+The YAML fragment below shows some common options for setting up the base source (where data is coming from), replica (where data is going to) and the SQS queue to read hive events from.


massdosage · 2019-04-12T18:30:26Z

README.md

+|Property|Required|Description|
+|:----|:----:|:----|
+|`source-catalog.name`|Yes|A name for the source catalog for events and logging.|
+|`source-catalog.hive-metastore-uris`|No|Fully qualified URI of the source cluster's Hive metastore Thrift service.|


What happens if you don't provide this? We should probably explain...

oh. That is mandatory. Changed now.

README.md

massdosage · 2019-04-12T18:33:23Z

README.md

+|`replica-catalog.hive-metastore-uris`|Yes|Fully qualified URI of the replica cluster's Hive metastore Thrift service.|
+|`event-receiver.configuration-properties.com.hotels.shunting.yard.event.receiver.sqs.queue`|Yes|Fully qualified URI of the [AWS SQS](https://aws.amazon.com/sqs/) Queue to read the Hive events from.|
+|`event-receiver.configuration-properties.com.hotels.shunting.yard.event.receiver.sqs.wait.time.seconds`|No|Wait time in seconds for which the receiver will poll the SQS queue for a batch of messages. Default is 10 seconds. Read more about long polling with AWS SQS [here](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html).|
+|`source-table-filter.table-names`|No|A list of tables selected for Shunting Yard replication. Supported format:`database_1.table_1, database_2.table_2`|


For all the values which are optional we should explain what the behaviour is if they're not set to anything.

massdosage · 2019-04-12T18:33:50Z

README.md

+|`source-table-filter.table-names`|No|A list of tables selected for Shunting Yard replication. Supported format:`database_1.table_1, database_2.table_2`|
+|`table-replications[n].source-table.database-name`|No|The name of the database in which the table you wish to replicate is located.|
+|`table-replications[n].source-table.table-name`|No|The name of the table which you wish to replicate.|
+|`table-replications[n].replica-table.database-name`|No|The name of the destination database in which to replicate the table. Defaults to source database name.|


Yes, here and the next one we explain what happens if not set.

massdosage · 2019-04-12T18:34:44Z

README.md

+|`table-replications[n].replica-table.database-name`|No|The name of the destination database in which to replicate the table. Defaults to source database name.|
+|`table-replications[n].replica-table.table-name`|No|The name of the table at the destination. Defaults to source table name.|
+
+### Configuring Graphite metrics

 Graphite configurations can be passed to Shunting Yard using an optional `--ct-config` argument which takes a YAML file and passes it directly to internal Circus Train instance. Refer to the [Circus Train README](https://github.com/HotelsDotCom/circus-train#graphite) for more details.


Make clear this is a different YAML configuration file to the one described above.

English Grammar

AnanaMJ · 2019-04-15T16:10:59Z

README.md

@@ -1,61 +1,191 @@
 # Shunting Yard

-A Spring Boot app that reads serialized Hive MetaStore Events and builds a YAML file with the information provided in the event which is then passed to [Circus Train](https://github.com/HotelsDotCom/circus-train) to perform the replication.
+Shunting Yard reads serialized Hive MetaStore Events from a queue ([AWS SQS](https://aws.amazon.com/sqs/) is currently supported) and replicates the data between two data lakes. It does this by building a YAML file with the information provided in the event which is then passed to [Circus Train](https://github.com/HotelsDotCom/circus-train) to perform the replication.


MetaStore → Metastore

AnanaMJ · 2019-04-15T16:20:05Z

README.md


-    export HIVE_LIB=/usr/lib/hive/lib/
-    export HCAT_LIB=/usr/lib/hive-hcatalog/share/hcatalog/
+Note that the paths above are correct as of when this document was last updated but may differ across EMR versions. Refer to the [EMR release guide](http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html) for more up to date information if necessary.


up to date information → up-to-date information

Up to date is without hyphens when it's used after the noun it describes: "The information is up to date."
When it's before the noun, it's with hyphens: "This is an up-to-date information."
source: https://dictionary.cambridge.org/dictionary/english/up-to-date

.... closes laptop and goes home ;)

AnanaMJ · 2019-04-15T16:22:16Z

README.md

-#### Sample ct-config.yml:
+### Specifying target database & table names
+
+Shunting Yard will by default replicate the data into the replica data lake with same replica database name and table name as the source. Sometimes a user might need to change the replica database name or table name or both. The YAML fragments below shows some common options for specifying the replica database and table name for the selected tables.


with same replica database name and table name → with the same replica database name and table name

If you want to not have that many "the", you can have
"Shunting Yard will by default replicate data into a replica data lake with the same replica database name and table name as the source."

AnanaMJ · 2019-04-15T16:24:07Z

README.md

+
+#### Change only the replica database but the table name remains same as source
+
+In this case, the replica table name is not provided in the `table-replications` and hence, it will be same as source table name.


and hence → ~~and~~ hence
It's like saying "and therefore" or "and so"

abhimanyugupta07 added 8 commits April 12, 2019 15:30

Creating a README for Shunting Yard

e4bbd85

Updating README with housekeeping

0ba4db6

Fixing formatting issues

01f6b99

fixing formatting

0cd6455

Fixing formatting

2461780

Formatting fix

679b203

Formatting fix

a667f36

full stop remove

7b5b43e

abhimanyugupta07 changed the title ~~Exh~~ README for Shunting Yard Apr 12, 2019

abhimanyugupta07 requested review from AnanaMJ, massdosage, max-jacobs, andreeapad, patduin and teabot April 12, 2019 15:44

AnanaMJ reviewed Apr 12, 2019

View reviewed changes

AnanaMJ suggested changes Apr 12, 2019

View reviewed changes

adding full stops and lower casing few things

43fddb0

massdosage suggested changes Apr 12, 2019

View reviewed changes

abhimanyugupta07 added 2 commits April 15, 2019 13:44

code review changes incorporated

d4fd989

target to replica

4b627e8

abhimanyugupta07 and others added 8 commits April 15, 2019 14:43

Formatting

91c1512

More formatting

2b94fab

Changed formatting

c32f33b

Removing duplicate line

bc7f284

better english

41caa23

Update README.md (#36)

4eefc0b

English Grammar

Source URI is mandatory and Hive metastore -> Hive Metastore

03899b6

Resolving merge conflicts

70ed58c

massdosage approved these changes Apr 15, 2019

View reviewed changes

AnanaMJ reviewed Apr 15, 2019

View reviewed changes

English Grammar

827304d

AnanaMJ approved these changes Apr 15, 2019

View reviewed changes

patduin approved these changes Apr 16, 2019

View reviewed changes

abhimanyugupta07 merged commit d4a2b2f into master Apr 16, 2019

abhimanyugupta07 deleted the readme-file branch April 16, 2019 10:06


		You can obtain Shunting Yard from Maven Central:
		1. Download the version to use from [Maven Central](https://mvnrepository.com/artifact/com.hotels/shunting-yard-binary) and uncompress it in a directory of your choosing.


		[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.hotels/shunting-yard/badge.svg?subject=com.hotels:shunting-yard)](https://maven-badges.herokuapp.com/maven-central/com.hotels/shunting-yard) [![Build Status](https://travis-ci.org/HotelsDotCom/shunting-yard.svg?branch=master)](https://travis-ci.org/HotelsDotCom/shunting-yard) [![Coverage Status](https://coveralls.io/repos/github/HotelsDotCom/shunting-yard/badge.svg?branch=master)](https://coveralls.io/github/HotelsDotCom/shunting-yard?branch=master) ![GitHub license](https://img.shields.io/github/license/HotelsDotCom/shunting-yard.svg)
		2. Download and install the latest version of [Circus Train](http://mvnrepository.com/artifact/com.hotels/circus-train/) and set the `CIRCUS_TRAIN_HOME` environment variable:


		On the target cluster, download and install the latests version of Circus Train and set the `CIRCUS_TRAIN_HOME` environment variable:
		The YAML fragment below shows some common options for setting up the base source (where data is coming from), replica (where data is going to) and the SQS queue to read hive events from.


		#### Change only the replica database but the table name remains same as source

		In this case, the replica table name is not provided in the `table-replications` and hence, it will be same as source table name.

README for Shunting Yard #35

README for Shunting Yard #35

Conversation

abhimanyugupta07 commented Apr 12, 2019 • edited

coveralls commented Apr 12, 2019 • edited

Pull Request Test Coverage Report for Build 397

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnanaMJ left a comment

Choose a reason for hiding this comment

abhimanyugupta07 commented Apr 12, 2019

massdosage left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abhimanyugupta07 commented Apr 12, 2019 •

edited

coveralls commented Apr 12, 2019 •

edited