[SPARK-27715][SQL][UI] SQL query details in UI does not show in correct format.

[uncleGen]

What changes were proposed in this pull request?

before pr:

after pr:

How was this patch tested?

manual test

[uncleGen]

cc @srowen

[SparkQA]

Test build #105399 has finished for PR 24609 at commit 956f61f.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[uncleGen]

retest this please

[SparkQA]

Test build #105407 has finished for PR 24609 at commit 956f61f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #105445 has finished for PR 24609 at commit 1f6c366.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shahidki31]

It seems jobDescription trims the execution.description. But for long queries we shouldn't trim the query and need to display in the description.

[uncleGen]

Sorry, do not understand what you mean. makeDescription returns HTML rendering of a job or stage description. It will try to parse the string as HTML and make sure that it only contains anchors with root-relative links. Otherwise, the whole string will rendered as a simple escaped text.

[shahidki31]

In the details, before this change it used to show entire query if the query is large. But after the change, it seems not showing the entire sql query.

After change:

Before change:

[uncleGen]

@shahidki31 Got it, let me confirm this problem.

[uncleGen]

@shahidki31 Could you please test the pr again? It is fixed in my local test.

[shahidki31]

Thanks @uncleGen . Should we fix here too?

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala

Lines 204 to 212 in 58989cd

	val description = paths.size match {
	case 0 =>
	s"Listing leaf files and directories 0 paths"
	case 1 =>
	s"Listing leaf files and directories for 1 path:<br/>${paths(0)}"
	case s =>
	s"Listing leaf files and directories for $s paths:<br/>${paths(0)}, ..."
	}
	sparkContext.setJobDescription(description)

[srowen]

@uncleGen I'd just look at this possible change too.

[SparkQA]

Test build #105479 has finished for PR 24609 at commit ca1a1f7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

That's safer, but, I wonder if we should just remove the <br/> from StreamExecution.getBatchDescriptionString. It doesn't seem essential, nor does it seem appropriate to add HTML there.

[gaborgsomogyi]

s/UI dose not/UI does not

[uncleGen]

@gaborgsomogyi Sorry, what do you mean?

[srowen]

(He's just suggesting you fix the typo in the title here)

[uncleGen]

@srowen I removed the html tag from the StreamExecution.getBatchDescriptionString, and get correct format. Take a review please.

[srowen]

Yes, pending tests, that looks like a better fix.

[SparkQA]

Test build #111346 has finished for PR 24609 at commit 47eb0f4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gaborgsomogyi]

@uncleGen the title still contains the typo and it's not allowed to fix by reviewers. Can you please take care of it? Otherwise looks good.

[srowen]

(I fixed the title.)

[uncleGen]

Thanks @srowen and @gaborgsomogyi

[srowen]

Merged to master

[srowen]

(Currently having trouble with JIRA, even after updating to LDAP password; I'll update the issue when it works or else anyone else can resolve it)