Kylin 3726: KylinSession should load spark properties from spark-defaults.conf

[lshmouse]

When testing parquet storage, the spark session job failed to be submit for no JAVA_HOME in executor env. This config is set in the spark default property file: spark-defaults.conf.

2018-12-18,15:13:15,466 ERROR org.apache.spark.deploy.yarn.YarnAllocator: Failed to launch executor 6 on container container_e823_1541646991414_1025309_01_000007
java.util.NoSuchElementException: key not found: JAVA_HOME
	at scala.collection.MapLike$class.default(MapLike.scala:228)
	at scala.collection.AbstractMap.default(Map.scala:59)
	at scala.collection.mutable.HashMap.apply(HashMap.scala:65)
	at org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:286)
	at org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:275)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:275)
	at org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:274)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.deploy.yarn.ExecutorRunnable.prepareEnvironment(ExecutorRunnable.scala:274)
	at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:92)
	at org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:69)
	at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$runAllocatedContainers$1$$anon$1.run(YarnAllocator.scala:556)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

[asfgit]

Can one of the admins verify this patch?

[shaofengshi]

hello, the commit message need to be started with JIRA ID like "KYLIN-3726", so that github can link the PR with the JIRA automatically. Could you please modify the commit message? "git rebase -i HEAD~1" can do that. Thanks!

[shaofengshi]

Excellent, thank you Shaohui!