[SPARK-8701][Streaming][WebUI] Add input metadata in the batch page #7081

zsxwing · 2015-06-29T08:18:03Z

This PR adds metadata to InputInfo. InputDStream can report its metadata for a batch and it will be shown in the batch page.

For example,

FileInputDStream will display the new files for a batch, and DirectKafkaInputDStream will display its offset ranges.

SparkQA · 2015-06-29T11:28:46Z

Test build #35977 has finished for PR 7081 at commit d496ae9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-06-30T22:40:11Z

streaming/src/main/scala/org/apache/spark/streaming/scheduler/BatchInfo.scala

@@ -33,7 +33,7 @@ import org.apache.spark.streaming.Time
 @DeveloperApi
 case class BatchInfo(
    batchTime: Time,
-    streamIdToNumRecords: Map[Int, Long],


This is a problem. We cannot expose InputInfo class through BatchInfo as the first one is private[streaming] whereas the latter in public.

I wondering what is the right way here. Any ideas?

Use two parameters: streamIdToNumRecords and streamIdToMetadata?

Okay I chatted with @pwendell , and the decision in as follows.

Name = StreamInputInfo,

It will have the field info: Map[String, Object]. Rather than a string, its most future-proof is to use a Map[String, Object], as we may want to not just strings but actual objects. For example, for files, one of the keys in the map be "files" and the value is a list of files.

Deprecate streamIdToNumRecords and introduce streamIdToInputInfo: Map[Int, StreamInputInfo]

object StreamInputInfo can have well known key names like for number of records, etc. They are used to store the corresponding data in the map.

Maybe case class StreamInputInfo(numRecords: Int, metadata: Map[String, Object]) is fine.

And the metadata map will have at least one standard field named "Description" which will map to the string that will be shown in the UI.

Also, please make the string prettier by introducing newlines and tabs, and making the HTML preserve newlines and tabs

…ng, Any]

zsxwing · 2015-07-04T11:27:22Z

New screenshot

zsxwing · 2015-07-04T11:28:01Z

streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala

+  private def metadataDescriptionToHTML(metadataDescription: String): Seq[Node] = {
+    // tab to 4 spaces and "\n" to "<br/>"
+    Unparsed(StringEscapeUtils.escapeHtml4(metadataDescription).
+      replaceAllLiterally("\t", "&nbsp;&nbsp;&nbsp;&nbsp;").replaceAllLiterally("\n", "<br/>"))


Because HTML doesn't support tab, I use 4 spaces instead.

SparkQA · 2015-07-04T13:17:05Z

Test build #36532 has finished for PR 7081 at commit 1d94582.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-07-07T02:23:26Z

external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala

+    }
+    val description = offsetRanges.map { offsetRange =>
+      s"topic: ${offsetRange.topic}\tpartition: ${offsetRange.partition}\t" +
+        s"range: [${offsetRange.fromOffset}, ${offsetRange.untilOffset})"


Can you use the format X --> Y (where Y = unitOffset - 1)
BTW, if the range is from: 5 until: 5 (that is not data), then you should ignore that range in the UI.

Or "offsets: X to Y"

zsxwing · 2015-07-07T11:44:26Z

New screenshot:

And screenshot for no files and no Kafka data:

SparkQA · 2015-07-07T13:08:41Z

Test build #36679 has started for PR 7081 at commit 8730dec.

SparkQA · 2015-07-07T13:24:11Z

Test build #36679 has finished for PR 7081 at commit 8730dec.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class StreamInputInfo(

AmplabJenkins · 2015-07-07T13:24:18Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-07-07T14:03:12Z

Merged build triggered.

AmplabJenkins · 2015-07-07T14:03:18Z

Merged build started.

SparkQA · 2015-07-07T14:06:46Z

Test build #36682 has started for PR 7081 at commit 74762da.

SparkQA · 2015-07-07T16:17:34Z

Test build #36682 has finished for PR 7081 at commit 74762da.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class StreamInputInfo(

AmplabJenkins · 2015-07-07T16:18:09Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-07-09T08:43:11Z

Merged build triggered.

AmplabJenkins · 2015-07-09T08:43:17Z

Merged build started.

SparkQA · 2015-07-09T08:44:27Z

Test build #36915 has started for PR 7081 at commit d906209.

AmplabJenkins · 2015-07-09T08:53:11Z

Merged build triggered.

AmplabJenkins · 2015-07-09T08:53:17Z

Merged build started.

SparkQA · 2015-07-09T08:54:36Z

Test build #36918 has started for PR 7081 at commit f7abd9b.

SparkQA · 2015-07-09T09:09:18Z

Test build #36918 has finished for PR 7081 at commit f7abd9b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class StreamInputInfo(

AmplabJenkins · 2015-07-09T09:09:26Z

Merged build finished. Test FAILed.

tdas · 2015-07-09T09:22:23Z

Jenkins, test this please.

AmplabJenkins · 2015-07-09T09:23:11Z

Merged build triggered.

AmplabJenkins · 2015-07-09T09:23:18Z

Merged build started.

SparkQA · 2015-07-09T09:26:54Z

Test build #36920 has started for PR 7081 at commit f7abd9b.

SparkQA · 2015-07-09T10:43:51Z

Test build #36915 has finished for PR 7081 at commit d906209.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class StreamInputInfo(

AmplabJenkins · 2015-07-09T10:44:18Z

Merged build finished. Test FAILed.

tdas · 2015-07-09T10:50:13Z

Jenkins, test this please.

AmplabJenkins · 2015-07-09T10:53:10Z

Merged build triggered.

AmplabJenkins · 2015-07-09T10:53:19Z

Merged build started.

SparkQA · 2015-07-09T10:57:00Z

Test build #36925 has started for PR 7081 at commit f7abd9b.

SparkQA · 2015-07-09T11:32:10Z

Test build #36920 has finished for PR 7081 at commit f7abd9b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class StreamInputInfo(

AmplabJenkins · 2015-07-09T11:32:25Z

Merged build finished. Test FAILed.

zsxwing · 2015-07-09T13:40:08Z

retest this please

AmplabJenkins · 2015-07-09T13:43:11Z

Merged build triggered.

SparkQA · 2015-07-09T13:52:02Z

Test build #36925 timed out for PR 7081 at commit f7abd9b after a configured wait of 175m.

AmplabJenkins · 2015-07-09T13:52:11Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-07-09T13:53:19Z

Merged build started.

SparkQA · 2015-07-09T13:57:03Z

Test build #36932 has started for PR 7081 at commit f7abd9b.

SparkQA · 2015-07-09T16:32:43Z

Test build #36932 has finished for PR 7081 at commit f7abd9b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class StreamInputInfo(

AmplabJenkins · 2015-07-09T16:33:16Z

Merged build finished. Test PASSed.

tdas · 2015-07-09T20:47:58Z

Merging this to master! Thanks @zsxwing

Add input metadata in the batch page

d496ae9

tdas reviewed Jun 30, 2015
View reviewed changes

Raname InputInfo to StreamInputInfo and change "metadata" to Map[Stri…

1d94582

…ng, Any]

zsxwing reviewed Jul 4, 2015
View reviewed changes

tdas reviewed Jul 7, 2015
View reviewed changes

zsxwing added 2 commits July 7, 2015 19:29

Address comments

450a46c

Merge branch 'master' into input-metadata

7903e33

Fix MiMa tests

74762da

Merge branch 'master' into input-metadata

d906209

Revert the space changes in project/MimaExcludes.scala

f7abd9b

asfgit closed this in 1f6b0b1 Jul 9, 2015

zsxwing deleted the input-metadata branch July 10, 2015 00:29

[SPARK-8701][Streaming][WebUI] Add input metadata in the batch page #7081

[SPARK-8701][Streaming][WebUI] Add input metadata in the batch page #7081

Conversation

zsxwing commented Jun 29, 2015

SparkQA commented Jun 29, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zsxwing commented Jul 4, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 4, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zsxwing commented Jul 7, 2015

SparkQA commented Jul 7, 2015

SparkQA commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

SparkQA commented Jul 7, 2015

SparkQA commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

AmplabJenkins commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

SparkQA commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

SparkQA commented Jul 9, 2015

SparkQA commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

tdas commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

SparkQA commented Jul 9, 2015

SparkQA commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

tdas commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

SparkQA commented Jul 9, 2015

SparkQA commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

zsxwing commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

SparkQA commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

SparkQA commented Jul 9, 2015

SparkQA commented Jul 9, 2015

AmplabJenkins commented Jul 9, 2015

tdas commented Jul 9, 2015