add a util method for changing the log level while running #2433

holdenk · 2014-09-17T18:55:52Z

No description provided.

SparkQA · 2014-09-17T18:59:21Z

QA tests have started for PR 2433 at commit 44d6577.

This patch merges cleanly.

SparkQA · 2014-09-17T19:50:25Z

QA tests have finished for PR 2433 at commit 44d6577.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2014-09-17T21:41:00Z

Is the idea here for this to be called by spark users or by spark internal components? If the former, this won't be visible because it's in a private object. It could be nice to expose this in SparkContext, but if we do it there I would expose a string instead of taking a logging level object, and we do the conversion on our side. It will make it easier in python and also not bind our public API to log4j.

pwendell · 2014-09-17T21:41:40Z

One case where this could be useful is in the Spark shell:

scala> sc.setLoggingLevel("WARN")

holdenk · 2014-09-17T21:43:34Z

Sounds good, how about I have the spark context do the conversion and call the utils method?

pwendell · 2014-09-17T21:58:49Z

yeah, that works

SparkQA · 2014-09-23T04:29:24Z

QA tests have started for PR 2433 at commit cdb3bfc.

This patch merges cleanly.

SparkQA · 2014-09-23T05:36:30Z

QA tests have finished for PR 2433 at commit cdb3bfc.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-23T05:36:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20687/

pwendell · 2014-09-23T20:33:47Z

core/src/main/scala/org/apache/spark/util/Utils.scala

@@ -1453,6 +1453,14 @@ private[spark] object Utils extends Logging {
  }

  /**
+   * configure a new log4j level
+   */
+  def setLogLevel(l: org.apache.log4j.Level) {


can we call this setLoggingLevel so it's consistent with the other one?

pwendell · 2014-09-23T20:34:31Z

Minor comments, otherwise looks good!

…ark context for changing the log level and add a note about overriding user-defined log settings.

SparkQA · 2014-09-23T23:14:30Z

QA tests have started for PR 2433 at commit 03ed4f9.

This patch merges cleanly.

SparkQA · 2014-09-24T00:22:22Z

QA tests have finished for PR 2433 at commit 03ed4f9.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- logInfo("Interrupting user class to stop.")

SparkQA · 2014-09-24T00:22:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20732/

pwendell · 2014-09-25T05:51:53Z

Hey @holdenk forgot to mention - can you add this in Java and Python also? Guessing this will be really useful for people working on the Spark shell.

holdenk · 2014-09-25T06:49:49Z

Sure I will do that tomorrow :)

SparkQA · 2014-10-22T16:39:47Z

QA tests have started for PR 2433 at commit e6ae26a.

This patch merges cleanly.

SparkQA · 2014-10-22T16:45:27Z

QA tests have finished for PR 2433 at commit e6ae26a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-22T16:45:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22038/
Test FAILed.

…text

SparkQA · 2014-10-22T17:14:43Z

QA tests have started for PR 2433 at commit 3429353.

This patch merges cleanly.

SparkQA · 2014-10-22T18:26:06Z

QA tests have finished for PR 2433 at commit 3429353.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-22T18:26:10Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22039/
Test PASSed.

davies · 2014-10-23T22:59:39Z

@holdenk Could you add some examples about how the logging levels should be? All list all the valid names in docstring.

@tdas We could use this in the Streaming examples!

tdas · 2014-10-24T00:10:35Z

Great! But how does this interact in situations where the downstream applications have explicitly removed Log4j from their path because they use other logging libraries (they do)? Will this throw an error?
And a more general question is that should we expose functionality is something as core as SparkContext that is specific to a particular logging library? @pwendell

holdenk · 2014-10-27T20:46:13Z

@tdas If we switch to a different logging library it would just be a matter of calling the new log libraries change log level function (and possibly translating some string names).

pwendell · 2014-11-19T08:23:47Z

@tdas if people are doing major dependency re-writes of Spark to replace it's logging subsystem, IMO, they should be willing to tolerate a runtime error if they try to call this. I've only seen this in the case of embedded Spark applications where the user controls all the code anyways.

Holden what about enumerating the logging levels per @davies suggestion? This seems like a good idea. Modulo that suggestion, this looks good to me.

pwendell · 2015-01-19T09:36:16Z

Let's close this issue pending an update from @holdenk.

@pwendel

…ontext. Based on an earlier PR, apache#2433 includes CR feedback from @pwendel & @davies

@pwendel

Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, #2433 includes CR feedback from pwendel & davies Author: Holden Karau <holden@pigscanfly.ca> Closes #5791 from holdenk/SPARK-3444-provide-an-easy-way-to-change-log-level-r2 and squashes the following commits: 3bf3be9 [Holden Karau] fix exception 42ba873 [Holden Karau] fix exception 9117244 [Holden Karau] Only allow valid log levels, throw exception if invalid log level. 338d7bf [Holden Karau] rename setLoggingLevel to setLogLevel fac14a0 [Holden Karau] Fix style errors d9d03f3 [Holden Karau] Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, #2433 includes CR feedback from @pwendel & @davies

@pwendel

Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from pwendel & davies Author: Holden Karau <holden@pigscanfly.ca> Closes apache#5791 from holdenk/SPARK-3444-provide-an-easy-way-to-change-log-level-r2 and squashes the following commits: 3bf3be9 [Holden Karau] fix exception 42ba873 [Holden Karau] fix exception 9117244 [Holden Karau] Only allow valid log levels, throw exception if invalid log level. 338d7bf [Holden Karau] rename setLoggingLevel to setLogLevel fac14a0 [Holden Karau] Fix style errors d9d03f3 [Holden Karau] Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from @pwendel & @davies

@pwendel

Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from pwendel & davies Author: Holden Karau <holden@pigscanfly.ca> Closes apache#5791 from holdenk/SPARK-3444-provide-an-easy-way-to-change-log-level-r2 and squashes the following commits: 3bf3be9 [Holden Karau] fix exception 42ba873 [Holden Karau] fix exception 9117244 [Holden Karau] Only allow valid log levels, throw exception if invalid log level. 338d7bf [Holden Karau] rename setLoggingLevel to setLogLevel fac14a0 [Holden Karau] Fix style errors d9d03f3 [Holden Karau] Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from @pwendel & @davies

@pwendel

Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from pwendel & davies Author: Holden Karau <holden@pigscanfly.ca> Closes apache#5791 from holdenk/SPARK-3444-provide-an-easy-way-to-change-log-level-r2 and squashes the following commits: 3bf3be9 [Holden Karau] fix exception 42ba873 [Holden Karau] fix exception 9117244 [Holden Karau] Only allow valid log levels, throw exception if invalid log level. 338d7bf [Holden Karau] rename setLoggingLevel to setLogLevel fac14a0 [Holden Karau] Fix style errors d9d03f3 [Holden Karau] Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, apache#2433 includes CR feedback from @pwendel & @davies

add a util method for changing the log level while running

44d6577

add setLoggingLevel to the SparkContext

cdb3bfc

pwendell reviewed Sep 23, 2014
View reviewed changes

CR feedback, make the name of the utils function match that in the sp…

03ed4f9

…ark context for changing the log level and add a note about overriding user-defined log settings.

holdenk added 2 commits October 22, 2014 09:31

merge in master

e1c1cd0

Add Java and Python APIs

e6ae26a

just pass it through from Java Spark Context to the regular Spark Con…

3429353

…text

asfgit closed this in 1ac1c1d Jan 19, 2015

holdenk added a commit to holdenk/spark that referenced this pull request Apr 29, 2015

Add support for changing the log level at run time through the SparkC…

d9d03f3

…ontext. Based on an earlier PR, apache#2433 includes CR feedback from @pwendel & @davies

holdenk mentioned this pull request Apr 29, 2015

[SPARK-3444] Provide an easy way to change log level #5791

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add a util method for changing the log level while running #2433

add a util method for changing the log level while running #2433

holdenk commented Sep 17, 2014

SparkQA commented Sep 17, 2014

SparkQA commented Sep 17, 2014

pwendell commented Sep 17, 2014

pwendell commented Sep 17, 2014

holdenk commented Sep 17, 2014

pwendell commented Sep 17, 2014

SparkQA commented Sep 23, 2014

SparkQA commented Sep 23, 2014

SparkQA commented Sep 23, 2014

pwendell Sep 23, 2014

pwendell commented Sep 23, 2014

SparkQA commented Sep 23, 2014

SparkQA commented Sep 24, 2014

SparkQA commented Sep 24, 2014

pwendell commented Sep 25, 2014

holdenk commented Sep 25, 2014

SparkQA commented Oct 22, 2014

SparkQA commented Oct 22, 2014

AmplabJenkins commented Oct 22, 2014

SparkQA commented Oct 22, 2014

SparkQA commented Oct 22, 2014

AmplabJenkins commented Oct 22, 2014

davies commented Oct 23, 2014

tdas commented Oct 24, 2014

holdenk commented Oct 27, 2014

pwendell commented Nov 19, 2014

pwendell commented Jan 19, 2015

add a util method for changing the log level while running #2433

add a util method for changing the log level while running #2433

Conversation

holdenk commented Sep 17, 2014

SparkQA commented Sep 17, 2014

SparkQA commented Sep 17, 2014

pwendell commented Sep 17, 2014

pwendell commented Sep 17, 2014

holdenk commented Sep 17, 2014

pwendell commented Sep 17, 2014

SparkQA commented Sep 23, 2014

SparkQA commented Sep 23, 2014

SparkQA commented Sep 23, 2014

pwendell Sep 23, 2014

Choose a reason for hiding this comment

pwendell commented Sep 23, 2014

SparkQA commented Sep 23, 2014

SparkQA commented Sep 24, 2014

SparkQA commented Sep 24, 2014

pwendell commented Sep 25, 2014

holdenk commented Sep 25, 2014

SparkQA commented Oct 22, 2014

SparkQA commented Oct 22, 2014

AmplabJenkins commented Oct 22, 2014

SparkQA commented Oct 22, 2014

SparkQA commented Oct 22, 2014

AmplabJenkins commented Oct 22, 2014

davies commented Oct 23, 2014

tdas commented Oct 24, 2014

holdenk commented Oct 27, 2014

pwendell commented Nov 19, 2014

pwendell commented Jan 19, 2015