[SPARK-16635] [WEBUI] [SQL] [WIP] Provide Session support in the Spark UI

[nblintao]

What changes were proposed in this pull request?

Add a Session Tab and the related Session Page in the Spark UI.

The page could show all properties configured for the session. It could be updated to the latest configuration after refreshing, compared to the SparkContext configurations showed in the Environments Page which cannot be updated during runtime.

How was this patch tested?

Manually tested and viewed, and unit tests.

When a spark-shell is opened, the Session page could be found at the related web UI (e.g. http://192.168.1.104:4040/session/), showing all the properties of the session.

First input lines below into the shell, a new row about "Property A" could be found at the Session page after refreshing.

import org.apache.spark.sql.SparkSession
SparkSession.builder.config("Property A", "the value of Property A").getOrCreate()

Then input the line below, another row is shown.

spark.conf.set("Property B", "the value of Property B")

Screenshot:

[SparkQA]

Test build #62731 has finished for PR 14319 at commit f7741a0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class SparkListenerSessionUpdate(sessionDetails: Map[String, String])
  • class SessionListener extends SparkListener

[ajbozarth]

I've read through your code and didn't catch any issues, I also checked it out and it looks good. I think this is a nice feature to add, my only qualm is it add yet another tab to the Web UI. If everyone is ok adding another tab I'm ok with it, but could it make sense to just put the table in the env tab? Opinions? @srowen @tgravescs
Also I think you missed something in your updates to the tests but I couldn't see it on first look

[nblintao]

Thanks, @ajbozarth.
Yes, I think the configuration alone is not enough for a new tab. @yhuai and I actually plan to do more on this tab. As mentioned in JIRA:

In Spark 2.0, SparkSession will be the entry point of spark. We can think about how to display per-session info (like configurations) and jobs in a good way. We can start to experiment a Session tab and see if we can come up with a way to just show a single session's info in a nice way. Then, we can see if this new tab can replace any existing tab.

So this is not final and please make more suggestions. I've just put WIP in the title to avoid misleading.
Also, I will try to fix the error recently.

[nblintao]

Doing more study to figure out the requirements on providing per-session information to users. So, I decided to close the PR for now. The work will be continued after figuring out how this page should be designed.
Thank @yhuai, @ajbozarth, and others involved.