-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-32772][SQL] Reduce log messages for spark-sql CLI #29619
Conversation
Test build #128163 has finished for PR 29619 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this match how spark-shell is configured? seems fine then.
@@ -54,6 +54,8 @@ import org.apache.spark.util.ShutdownHookManager | |||
* has dropped its support. | |||
*/ | |||
private[hive] object SparkSQLCLIDriver extends Logging { | |||
initializeLogIfNecessary(true) | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably best to move this to line 63
I think so. spark-shell, pyspark and sparkr CLI seem to be configured with the same manner. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
…ark-sql CLI ### What changes were proposed in this pull request? Remove legacy silent support mode for spark-sql CLI. ### Why are the changes needed? #29619 add new silent mode. We can remove legacy silent support mode. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual test: ``` spark-sql> LM-SHC-16508156:spark yumwang$ bin/spark-sql -S NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly. 20/09/03 09:06:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 20/09/03 09:06:16 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist 20/09/03 09:06:16 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist 20/09/03 09:06:19 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0 20/09/03 09:06:19 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore yumwang10.226.196.190 spark-sql> select * from test1; 1 spark-sql> select * from test1; 1 ``` Closes #29631 from wangyum/SPARK-32772. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Yuming Wang <yumwang@ebay.com>
What changes were proposed in this pull request?
This PR reduces log messages for spark-sql CLI like spark-shell and pyspark CLI.
Why are the changes needed?
When we launch spark-sql CLI, too many log messages are shown and it's sometimes difficult to find the result of query.
Does this PR introduce any user-facing change?
Yes. Log messages are reduced for spark-sql CLI like as follows.
How was this patch tested?
Launched spark-sql CLI and confirmed that log messages are reduced as I paste above.