[SPARK-6796][Streaming][WebUI] Add "Active Batches" and "Completed Batches" lists to StreamingPage #5434
Conversation
|
cc @tdas and @JoshRosen |
|
Test build #29922 has finished for PR 5434 at commit
|
|
This is great start. But we should be adding unit tests for this. Recently we have started writing unit tests for UI using selenium. Take a look at how this is written. Try adding unit test to that do some basic testing of this feature. |
| <h4 id="active">Active Batches ({runningBatches.size + waitingBatches.size})</h4> ++ | ||
| new ActiveBatchTable(runningBatches, waitingBatches).toNodeSeq | ||
|
|
||
| val completedBatches = listener.retainedCompletedBatches. |
There was a problem hiding this comment.
Move this to the top of the function, i think more consistent.
|
I added some tests to verify the title titles and column names. |
|
retest this please |
|
Test build #29986 has finished for PR 5434 at commit
|
|
LGTM. Though, let me try running this locally and seeing the UI myself. |
|
Rebased on master. |
|
Test build #30060 has finished for PR 5434 at commit
|
| /** | ||
| * Return HTML for all rows of this table. | ||
| */ | ||
| protected def renderRows: Seq[Node] = Nil |
There was a problem hiding this comment.
You could make this class abstract and leave this method undefined, then override it in the concrete classes.
There was a problem hiding this comment.
Fixed. Made BatchTableBase abstract.
|
Test build #30125 has finished for PR 5434 at commit
|
| val formattedProcessingTime = processingTime.map(UIUtils.formatDuration).getOrElse("-") | ||
|
|
||
| <td sorttable_customkey={batchTime.toString}>{formattedBatchTime}</td> | ||
| <td sorttable_customkey={eventCount.toString}>{eventCount.toString} events</td> |
There was a problem hiding this comment.
One small nit that I noticed when i ran it myself. This says events where as else where in the UI it says records. Lets change it to events to keep things consistent.
|
Another points I noticed in the UI, in the |
|
Also, could you make the stats at the top of the page "Waiting batches" and "Processed batches" links to the corresponding sections? And the names should be consistent, so please rename them here, keep them as "Active Batches" and "Completed Batches". |
Conflicts: streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala
|
@tdas I fixed as per your suggestion. I also made "startTime" more readable. Here is the new screenshot:
|
|
Test build #30172 has finished for PR 5434 at commit
|
|
Test build #30174 has finished for PR 5434 at commit
|
|
I am merging this. Thanks! |
This PR adds two lists,
Active BatchesandCompleted Batches. Here is the screenshot:Due to SPARK-6766, I need to merge #5414 in my local machine to get the above screenshot.