Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-4381][Streaming]Add warning log when user set spark.master to local in Spark Streaming and there's no job executed #3244

Closed
wants to merge 3 commits into from
Closed

[SPARK-4381][Streaming]Add warning log when user set spark.master to local in Spark Streaming and there's no job executed #3244

wants to merge 3 commits into from

Conversation

jerryshao
Copy link
Contributor

No description provided.

@jerryshao jerryshao changed the title Add warning log when user set spark.master to local in Spark Streaming and there's no job executed [SPAKR-4381][Streaming]Add warning log when user set spark.master to local in Spark Streaming and there's no job executed Nov 13, 2014
@jerryshao jerryshao changed the title [SPAKR-4381][Streaming]Add warning log when user set spark.master to local in Spark Streaming and there's no job executed [SPARK-4381][Streaming]Add warning log when user set spark.master to local in Spark Streaming and there's no job executed Nov 13, 2014
@SparkQA
Copy link

SparkQA commented Nov 13, 2014

Test build #23309 has started for PR 3244 at commit eca428b.

  • This patch merges cleanly.

srowen
srowen reviewed Nov 13, 2014
View reviewed changes
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
@@ -121,6 +121,11 @@ class StreamingContext private[streaming] (
}
}

if (sc.conf.get("spark.master") == "local") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check for the case of local[1] too I suppose?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the note , I will add it.

@SparkQA
Copy link

SparkQA commented Nov 13, 2014

Test build #23310 has started for PR 3244 at commit d726e85.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 13, 2014

Test build #23309 timed out for PR 3244 at commit eca428b after a configured wait of 120m.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23309/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Nov 13, 2014

Test build #23310 has finished for PR 3244 at commit d726e85.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23310/
Test PASSed.

tdas
tdas reviewed Nov 19, 2014
View reviewed changes
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
@@ -121,6 +121,11 @@ class StreamingContext private[streaming] (
}
}

if (sc.conf.get("spark.master") == "local" || sc.conf.get("spark.master") == "local[1]") {
logWarning("spark.master should be set as local[n], n > 1 in local mode if you have receivers" +
" to get data, otherwise the job cannot get enough resource to execute.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"otherwise Spark jobs will not get resources to process the received data."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks TD, I will modify it :)

@tdas
Copy link
Contributor

tdas commented Nov 19, 2014

LGTM, one small point.

@tdas
Copy link
Contributor

tdas commented Nov 20, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Nov 20, 2014

Test build #23649 has started for PR 3244 at commit d2486c7.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 20, 2014

Test build #23649 has finished for PR 3244 at commit d2486c7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23649/
Test PASSed.

@tdas
Copy link
Contributor

tdas commented Nov 25, 2014

Since no unit tests cover this change, I tested it manually. It works as expected merging this to both master and 1.2

asfgit pushed a commit that referenced this pull request Nov 25, 2014
@jerryshao @tdas
[SPARK-4381][Streaming]Add warning log when user set spark.master to …
b026546 
…local in Spark Streaming and there's no job executed

Author: jerryshao <saisai.shao@intel.com>

Closes #3244 from jerryshao/SPARK-4381 and squashes the following commits:

d2486c7 [jerryshao] Improve the warning log
d726e85 [jerryshao] Add local[1] to the filter condition
eca428b [jerryshao] Add warning log

(cherry picked from commit fef27b2)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@asfgit asfgit closed this in fef27b2 Nov 25, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@jerryshao @SparkQA @AmplabJenkins @tdas @srowen