Skip to content

Commit

Permalink
SPARK-3639 | Removed settings master in examples
Browse files Browse the repository at this point in the history
This patch removes setting of master as local in Kinesis examples so that users can set it using submit-job.

Author: aniketbhatnagar <aniket.bhatnagar@gmail.com>

Closes #2536 from aniketbhatnagar/Kinesis-Examples-Master-Unset and squashes the following commits:

c9723ac [aniketbhatnagar] Merge remote-tracking branch 'origin/Kinesis-Examples-Master-Unset' into Kinesis-Examples-Master-Unset
fec8ead [aniketbhatnagar] SPARK-3639 | Removed settings master in examples
31cdc59 [aniketbhatnagar] SPARK-3639 | Removed settings master in examples
  • Loading branch information
aniketbhatnagar authored and andrewor14 committed Sep 26, 2014
1 parent 1aa549b commit d16e161
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 13 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@
* org.apache.spark.examples.streaming.JavaKinesisWordCountASL mySparkStream \
* https://kinesis.us-east-1.amazonaws.com
*
* Note that number of workers/threads should be 1 more than the number of receivers.
* This leaves one thread available for actually processing the data.
*
* There is a companion helper class called KinesisWordCountProducerASL which puts dummy data
* onto the Kinesis stream.
* Usage instructions for KinesisWordCountProducerASL are provided in the class definition.
Expand Down Expand Up @@ -114,12 +117,8 @@ public static void main(String[] args) {
/* In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard */
int numStreams = numShards;

/* Must add 1 more thread than the number of receivers or the output won't show properly from the driver */
int numSparkThreads = numStreams + 1;

/* Setup the Spark config. */
SparkConf sparkConfig = new SparkConf().setAppName("KinesisWordCount").setMaster(
"local[" + numSparkThreads + "]");
SparkConf sparkConfig = new SparkConf().setAppName("KinesisWordCount");

/* Kinesis checkpoint interval. Same as batchInterval for this example. */
Duration checkpointInterval = batchInterval;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ import org.apache.log4j.Level
* org.apache.spark.examples.streaming.KinesisWordCountASL mySparkStream \
* https://kinesis.us-east-1.amazonaws.com
*
*
* Note that number of workers/threads should be 1 more than the number of receivers.
* This leaves one thread available for actually processing the data.
*
* There is a companion helper class below called KinesisWordCountProducerASL which puts
* dummy data onto the Kinesis stream.
* Usage instructions for KinesisWordCountProducerASL are provided in that class definition.
Expand Down Expand Up @@ -97,17 +101,10 @@ private object KinesisWordCountASL extends Logging {
/* In this example, we're going to create 1 Kinesis Worker/Receiver/DStream for each shard. */
val numStreams = numShards

/*
* numSparkThreads should be 1 more thread than the number of receivers.
* This leaves one thread available for actually processing the data.
*/
val numSparkThreads = numStreams + 1

/* Setup the and SparkConfig and StreamingContext */
/* Spark Streaming batch interval */
val batchInterval = Milliseconds(2000)
val batchInterval = Milliseconds(2000)
val sparkConfig = new SparkConf().setAppName("KinesisWordCount")
.setMaster(s"local[$numSparkThreads]")
val ssc = new StreamingContext(sparkConfig, batchInterval)

/* Kinesis checkpoint interval. Same as batchInterval for this example. */
Expand Down

0 comments on commit d16e161

Please sign in to comment.