[SPARK-21775][Core]Dynamic Log Level Settings for executors

[lvdongr]

What changes were proposed in this pull request?

Sometimes we want to change the log level of executor when our application has already deployed, to see detail infomation or decrease the log items. Changing the log4j configure file is not convenient,so We add the ability to set log level settings for a running executor.
https://issues.apache.org/jira/browse/SPARK-21775

How was this patch tested?

manual tests

Please review http://spark.apache.org/contributing.html before opening a pull request.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

This seems like a ton of complexity. I don't think this functionality is worth nearly this.

[ajbozarth]

I agree with @srowen and I'm not a fan out the UI either

[jerryshao]

This seems like a super specific requirement, AFAIK there's no Hadoop related project support dynamically changing log level, do we really need this feature is Spark, do we have other workaround without changing the code? From my aspect, changing a mount of the code to support such specific requirement usually hard to persuade the community.

[lvdongr]

The log level setting is a very useful function.Our team is doing a spark application and when we want to see the debug log, we have to restart the application every time. So we develop this function.
The complexity lays on the ui display and set the log level. But we can choose not to show the setting Button on UI, and just give a api or restful interface for user to access this function.Then the change on spark is no much.
And Storm (http://storm.apache.org/) has the same function,too. @srowen @jerryshao @ajbozarth

[srowen]

We won't merge this, it's too much overhead for little gain. You can close this.

[lvdongr]

ok. Thank you all the same for your review @srowen @jerryshao @ajbozarth .

[sunzhusz]

spark not to supported the feature that is foolish