[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI #26378

shahidki31 · 2019-11-03T09:44:02Z

What changes were proposed in this pull request?

Support JDBC/ODBC tab for HistoryServer WebUI. Currently from Historyserver we can't access the JDBC/ODBC tab for thrift server applications. In this PR, I am doing 2 main changes

Refactor existing thrift server listener to support kvstore
Add history server plugin for thrift server listener and tab.

Why are the changes needed?

Users can access Thriftserver tab from History server for both running and finished applications,

Does this PR introduce any user-facing change?

Support for JDBC/ODBC tab for the WEBUI from History server

How was this patch tested?

Add UT and Manual tests

Start Thriftserver and Historyserver

sbin/stop-thriftserver.sh
sbin/stop-historyserver.sh
sbin/start-thriftserver.sh
sbin/start-historyserver.sh

Launch beeline
bin/beeline -u jdbc:hive2://localhost:10000
Run queries

Go to the JDBC/ODBC page of the WebUI from History server

SparkQA · 2019-11-03T09:46:18Z

Test build #113164 has finished for PR 26378 at commit f4a16a5.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-11-03T09:54:44Z

Test build #113165 has finished for PR 26378 at commit bff90de.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-11-03T10:59:48Z

Test build #113170 has finished for PR 26378 at commit 5490c8e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-03T11:44:35Z

Test build #113166 has finished for PR 26378 at commit dfd1ff6.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-11-03T11:47:54Z

Test build #113171 has finished for PR 26378 at commit 24dde37.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-03T11:49:56Z

retest this please

SparkQA · 2019-11-03T12:02:39Z

Test build #113172 has finished for PR 26378 at commit 905e635.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

SparkQA · 2019-11-03T14:02:07Z

Test build #113173 has finished for PR 26378 at commit 4eab137.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-04T09:44:38Z

retest this please

SparkQA · 2019-11-04T11:48:48Z

Test build #113204 has finished for PR 26378 at commit 4eab137.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-04T14:59:25Z

Jenkins, retest this please

shahidki31 · 2019-11-04T16:50:28Z

Retest this please

shahidki31

cc @vanzin , @cloud-fan , @wangym , @srowen , @AngersZhuuuu
Kindly review

shahidki31 · 2019-11-04T17:11:56Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala

@@ -121,179 +129,10 @@ object HiveThriftServer2 extends Logging {
    }
  }

-  private[thriftserver] class SessionInfo(
-      val sessionId: String,


The code here and below are moved to HiveThriftServer2Listener.scala class

shahidki31 · 2019-11-04T17:12:35Z

...e-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala

-   * An inner sparkListener called in sc.stop to clean up the HiveThriftServer2
-   */
-  private[thriftserver] class HiveThriftServer2Listener(
-      val server: HiveServer2,


Moved to a separate file (HiveThriftserver2Listener.scala)

shahidki31 · 2019-11-04T17:14:13Z

...server/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Listener.scala

+    ip: String,
+    sessionId: String,
+    userName: String,
+    startTime: Long) extends SparkListenerEvent


New events added for Thriftserver applications. Eventually these events will be catch by the listeners EventLoggingListner as well as HiveThriftServer2Listener`

shahidki31 · 2019-11-04T17:16:23Z

...r/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala

@@ -77,7 +77,8 @@ private[hive] class SparkExecuteStatementOperation(
    // RDDs will be cleaned automatically upon garbage collection.
    logInfo(s"Close statement with $statementId")
    cleanup(OperationState.CLOSED)
-    HiveThriftServer2.listener.onOperationClosed(statementId)
+    HiveThriftServer2.listener.postLiveListenerBus(SparkListenerOperationClosed(statementId,
+      System.currentTimeMillis()))


Changes here are below files are,

posting the Thriftserver specific events to LiveListener Bus, so that it can get by both EvenLoggingListener (For HistoryServer) as well as HiveThriftServer2Listener (For live UI)

shahidki31 · 2019-11-04T17:17:41Z

...c/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2AppStatusStore.scala

+class HiveThriftServer2AppStatusStore(
+    store: KVStore,
+    val listener: Option[HiveThriftServer2Listener] = None) {
+


Both Live and History UI will read from this store for populating the WEBUI

SparkQA · 2019-11-04T18:37:14Z

Test build #113216 has finished for PR 26378 at commit 4eab137.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

SparkQA · 2019-11-04T18:59:48Z

Test build #113217 has finished for PR 26378 at commit 540faf0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-04T19:49:36Z

retest this please

SparkQA · 2019-11-04T21:29:25Z

Test build #113224 has finished for PR 26378 at commit e965c5f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

shahidki31 · 2019-11-04T21:34:56Z

Not sure which test is failing. seems all UTs are passing as per the Jenkins report.

SparkQA · 2019-11-04T21:44:29Z

Test build #113225 has finished for PR 26378 at commit e965c5f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HiveThriftServer2HistoryServerPlugin extends AppHistoryServerPlugin
class HiveThriftServer2AppStatusStore(

SparkQA · 2019-11-05T21:25:21Z

Test build #113284 has finished for PR 26378 at commit 5c2932e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-27T01:25:54Z

Test build #114490 has finished for PR 26378 at commit eaea6e3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

...ver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala

...src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2EventManager.scala

SparkQA · 2019-11-27T01:44:55Z

Test build #114497 has finished for PR 26378 at commit db4269c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-27T03:20:01Z

Test build #114492 has finished for PR 26378 at commit a55927e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-11-27T03:46:14Z

Test build #114495 has finished for PR 26378 at commit af65eed.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

juliuszsompolski

LGTM pending scalastyle.

...tserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPageSuite.scala

SparkQA · 2019-11-27T16:48:58Z

Test build #114538 has finished for PR 26378 at commit f7fcc75.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang

LGTM, thanks for the work!

gengliangwang · 2019-11-28T00:08:51Z

@shahidki31 one more comment, I have a good idea about fixing the UI tab order.
We can add a new API displayOrder in the trait AppHistoryServerPlugin(with default value as Int.MaxValue).
And then we can set the displayOrder of SQL tab as 0, and the thriftserver tab as 1

I will merge this one once this is updated.

shahidki31 · 2019-11-28T01:54:06Z

@shahidki31 one more comment, I have a good idea about fixing the UI tab order.
We can add a new API displayOrder in the trait AppHistoryServerPlugin(with default value as Int.MaxValue).
And then we can set the displayOrder of SQL tab as 0, and the thriftserver tab as 1

Yeah, that is a nice idea. Thanks @gengliangwang . I will update the code

…nto ThriftKVStore

SparkQA · 2019-11-28T05:12:30Z

Test build #114558 has finished for PR 26378 at commit 2bc86c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

core/src/main/scala/org/apache/spark/status/AppHistoryServerPlugin.scala

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLHistoryServerPlugin.scala

SparkQA · 2019-11-28T12:05:42Z

Test build #114569 has finished for PR 26378 at commit 39ddc89.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

shahidki31 · 2019-11-29T06:16:32Z

retest this please

SparkQA · 2019-11-29T08:05:02Z

Test build #114603 has finished for PR 26378 at commit 39ddc89.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

shahidki31 · 2019-11-29T08:37:13Z

retest this please

SparkQA · 2019-11-29T11:18:31Z

Test build #114625 has finished for PR 26378 at commit 39ddc89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2019-11-30T03:43:18Z

Thanks, merging to master

shahidki31 · 2019-11-30T05:05:22Z

Thanks a lot @gengliangwang @juliuszsompolski @AngersZhuuuu for the reviews

…oryServer WebUI ### What changes were proposed in this pull request? Support JDBC/ODBC tab for HistoryServer WebUI. Currently from Historyserver we can't access the JDBC/ODBC tab for thrift server applications. In this PR, I am doing 2 main changes 1. Refactor existing thrift server listener to support kvstore 2. Add history server plugin for thrift server listener and tab. ### Why are the changes needed? Users can access Thriftserver tab from History server for both running and finished applications, ### Does this PR introduce any user-facing change? Support for JDBC/ODBC tab for the WEBUI from History server ### How was this patch tested? Add UT and Manual tests 1. Start Thriftserver and Historyserver ``` sbin/stop-thriftserver.sh sbin/stop-historyserver.sh sbin/start-thriftserver.sh sbin/start-historyserver.sh ``` 2. Launch beeline `bin/beeline -u jdbc:hive2://localhost:10000` 3. Run queries Go to the JDBC/ODBC page of the WebUI from History server ![image](https://user-images.githubusercontent.com/23054875/68365501-cf013700-0156-11ea-84b4-fda8008c92c4.png) Closes apache#26378 from shahidki31/ThriftKVStore. Authored-by: shahid <shahidki31@gmail.com> Signed-off-by: Gengliang Wang <gengliang.wang@databricks.com>

shahidki31 force-pushed the ThriftKVStore branch from dfd1ff6 to 7015ac6 Compare November 3, 2019 10:49

shahidki31 force-pushed the ThriftKVStore branch from 5490c8e to 24dde37 Compare November 3, 2019 11:46

shahidki31 force-pushed the ThriftKVStore branch from 24dde37 to 905e635 Compare November 3, 2019 11:49

shahidki31 mentioned this pull request Nov 3, 2019

[SPARK-29726][WEBUI]Support KVStore for HiveThriftServer2Listener #26372

Closed

shahidki31 force-pushed the ThriftKVStore branch from 905e635 to 4eab137 Compare November 3, 2019 12:13

dongjoon-hyun added the WEB UI label Nov 3, 2019

shahidki31 force-pushed the ThriftKVStore branch 2 times, most recently from 841f8a5 to 540faf0 Compare November 4, 2019 17:10

shahidki31 commented Nov 4, 2019

View reviewed changes

kvstore thrift

e965c5f

shahidki31 force-pushed the ThriftKVStore branch from 540faf0 to e965c5f Compare November 4, 2019 19:48

gengliangwang reviewed Nov 27, 2019

View reviewed changes

...ver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala Show resolved Hide resolved

...src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2EventManager.scala Outdated Show resolved Hide resolved

address comment

db4269c

juliuszsompolski approved these changes Nov 27, 2019

View reviewed changes

...tserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPageSuite.scala Outdated Show resolved Hide resolved

scalastyle

f7fcc75

gengliangwang approved these changes Nov 27, 2019

View reviewed changes

shahidki31 added 2 commits November 28, 2019 08:19

add display order. Minor refactor test.

5db0667

Merge branch 'ThriftKVStore' of https://github.com/shahidki31/spark i…

2bc86c5

…nto ThriftKVStore

gengliangwang reviewed Nov 28, 2019

View reviewed changes

core/src/main/scala/org/apache/spark/status/AppHistoryServerPlugin.scala Outdated Show resolved Hide resolved

gengliangwang reviewed Nov 28, 2019

View reviewed changes

core/src/main/scala/org/apache/spark/status/AppHistoryServerPlugin.scala Outdated Show resolved Hide resolved

gengliangwang reviewed Nov 28, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLHistoryServerPlugin.scala Outdated Show resolved Hide resolved

address comments

39ddc89

gengliangwang closed this in b182ed8 Nov 30, 2019

shahidki31 deleted the ThriftKVStore branch November 30, 2019 05:50

[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI #26378

[SPARK-29724][SPARK-29726][WEBUI][SQL] Support JDBC/ODBC tab for HistoryServer WebUI #26378

Conversation

shahidki31 commented Nov 3, 2019 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Nov 3, 2019

SparkQA commented Nov 3, 2019

SparkQA commented Nov 3, 2019

SparkQA commented Nov 3, 2019

SparkQA commented Nov 3, 2019

shahidki31 commented Nov 3, 2019

SparkQA commented Nov 3, 2019

SparkQA commented Nov 3, 2019

shahidki31 commented Nov 4, 2019

SparkQA commented Nov 4, 2019

shahidki31 commented Nov 4, 2019 • edited Loading

shahidki31 commented Nov 4, 2019

shahidki31 left a comment

Choose a reason for hiding this comment

shahidki31 Nov 4, 2019

Choose a reason for hiding this comment

shahidki31 Nov 4, 2019

Choose a reason for hiding this comment

shahidki31 Nov 4, 2019

Choose a reason for hiding this comment

shahidki31 Nov 4, 2019

Choose a reason for hiding this comment

shahidki31 Nov 4, 2019 • edited Loading

Choose a reason for hiding this comment

SparkQA commented Nov 4, 2019

SparkQA commented Nov 4, 2019

shahidki31 commented Nov 4, 2019

SparkQA commented Nov 4, 2019

shahidki31 commented Nov 4, 2019

SparkQA commented Nov 4, 2019

SparkQA commented Nov 5, 2019

SparkQA commented Nov 27, 2019

SparkQA commented Nov 27, 2019

SparkQA commented Nov 27, 2019

SparkQA commented Nov 27, 2019

juliuszsompolski left a comment

Choose a reason for hiding this comment

SparkQA commented Nov 27, 2019

gengliangwang left a comment

Choose a reason for hiding this comment

gengliangwang commented Nov 28, 2019

shahidki31 commented Nov 28, 2019

SparkQA commented Nov 28, 2019

SparkQA commented Nov 28, 2019

shahidki31 commented Nov 29, 2019

SparkQA commented Nov 29, 2019

shahidki31 commented Nov 29, 2019

SparkQA commented Nov 29, 2019

gengliangwang commented Nov 30, 2019

shahidki31 commented Nov 30, 2019

shahidki31 commented Nov 3, 2019 •

edited

Loading

shahidki31 commented Nov 4, 2019 •

edited

Loading

shahidki31 Nov 4, 2019 •

edited

Loading