Skip to content

Commit

Permalink
[ENV-239] Update example docs with more information (#162)
Browse files Browse the repository at this point in the history
  • Loading branch information
Jeremy Beard authored and Ian Buss committed Feb 16, 2018
1 parent 6841da4 commit 87513dc
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 2 deletions.
2 changes: 1 addition & 1 deletion examples/filesystem/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This example demonstrates a simple HDFS-based data processing pipeline.

**Run the Envelope job**

spark2-submit target/envelope-*.jar examples/filesystem/filesystem.conf
spark2-submit build/envelope/target/envelope-*.jar examples/filesystem/filesystem.conf

**Grab the results**

Expand Down
2 changes: 2 additions & 0 deletions examples/fix/README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ The configuration for this example is found link:fix.conf[here]. The messages do

## Running the example

. Modify `create_fix_tables.sql`, `fix.conf`, and `fix_generator.conf` to point to your cluster. If your cluster has secured Kafka, you will also need to modify the configuration files and below `spark2-submit` calls (see the FIX HBase example for more details) and the `kafka-topics` and `kafka-console-consumer` calls (see the test steps in the link:https://www.cloudera.com/documentation/kafka/latest/topics/kafka_security.html#concept_lcn_4mm_s5[Cloudera Kafka documentation] for more details).

. Create the required Kudu tables using the provided Apache Impala script:

impala-shell -f create_fix_tables.sql
Expand Down
4 changes: 3 additions & 1 deletion examples/traffic/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,12 @@

The traffic example is an Envelope pipeline that retrieves measurements of traffic congestion and stores an aggregated view of the traffic congestion at a point in time using the current measurement and all of those in the previous 60 seconds. Within Envelope this uses the Apache Spark Streaming window operations functionality. This example demonstrates use cases that need to do live aggregations of recently received messages prior to user querying.

A sample configuration file is provided for reference. After creating the required Apache Kudu tables using the provided Apache Impala scripts, the example can be run as:
A sample configuration file is provided for reference. After creating the required Apache Kudu tables using the provided Apache Impala scripts, and modifying the configuration file to point to your cluster, the example can be run as:

SPARK_KAFKA_VERSION=0.10 spark2-submit envelope-*.jar traffic.conf

Note that if your cluster has secured Kafka, you will also need to modify the configuration file and `spark2-submit` call -- see the FIX HBase example for more details.

An Apache Kafka producer to generate sample messages for the example, and push them in to the "traffic" topic, can be run as:

spark2-submit --class com.cloudera.labs.envelope.examples.TrafficGenerator envelope-*.jar kafkabrokerhost:9092 traffic
Expand Down

0 comments on commit 87513dc

Please sign in to comment.