[SPARK-8702][WebUI]Avoid massive concating strings in Javascript #7082
Conversation
|
/cc @sarutak |
|
Merged build triggered. |
|
Merged build started. |
|
Test build #35980 has started for PR 7082 at commit |
|
@sarutak I also find another issue in the console. Because
|
|
Test build #979 has started for PR 7082 at commit |
|
Oh, I see. could you file another one as a separate issue? |
| |'start': new Date($launchTime), | ||
| |'end': new Date($finishTime) | ||
| |} | ||
| |""".stripMargin.replaceAll("\n", " ") |
There was a problem hiding this comment.
We don't need stripMargin right?
There was a problem hiding this comment.
It's necessary. stripMargin removes the prefix " |".
There was a problem hiding this comment.
I misunderstood the effect of "|" and stripMargin. Yes it's needed.
There was a problem hiding this comment.
The \n isn't working on Windows if Git decides to check out the file using \r\n line-endings. May want to change it to [\r\n] instead.
There was a problem hiding this comment.
I see. Did you see it does not work? I think, it's better to replace it with a system property line.separator?
There was a problem hiding this comment.
Right. It complained ILLEGAL character and that string is still shown as multiline in developer console. I think the line.separator wouldn't work at least for my case where I'm compiling on Windows but deploying on Linux. Don't ask me why :)
There was a problem hiding this comment.
Thanks I opened a new PR #7133 for the issue you mentioned.
|
Test build #979 has finished for PR 7082 at commit
|
|
retest this please |
|
Merged build triggered. |
|
Merged build started. |
|
Test build #35985 has started for PR 7082 at commit |
|
Test build #35980 has finished for PR 7082 at commit
|
|
Merged build finished. Test PASSed. |
Opened a JIRA for this issue: https://issues.apache.org/jira/browse/SPARK-8705 |
|
Test build #35985 has finished for PR 7082 at commit
|
|
Merged build finished. Test PASSed. |
When there are massive tasks, such as
sc.parallelize(1 to 100000, 10000).count(), the generated JS codes have a lot of string concatenations in the stage page, nearly 40 string concatenations for one task.We can generate the whole string for a task instead of execution string concatenations in the browser.
Before this patch, the load time of the page is about 21 seconds.
After this patch, it reduces to about 17 seconds.
One disadvantage is that the generated JS codes become hard to read.