Skip to content

Commit

Permalink
[SPARK-35311][SS][UI][DOCS] Structured Streaming Web UI state informa…
Browse files Browse the repository at this point in the history
…tion documentation

### What changes were proposed in this pull request?
In this PR I'm adding Structured Streaming Web UI state information documentation.

### Why are the changes needed?
Missing documentation.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
```
cd docs/
SKIP_API=1 bundle exec jekyll build
```
Manual webpage check.

Closes #32433 from gaborgsomogyi/SPARK-35311.

Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
  • Loading branch information
gaborgsomogyi authored and HeartSaVioR committed May 14, 2021
1 parent 8fa739f commit b6a0a7e
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 0 deletions.
Binary file added docs/img/webui-structured-streaming-detail2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 6 additions & 0 deletions docs/web-ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -415,6 +415,7 @@ click a "run id" in the tables.

<p style="text-align: center;">
<img src="img/webui-structured-streaming-detail.png" title="Structured Streaming Query Statistics" alt="Structured Streaming Query Statistics">
<img src="img/webui-structured-streaming-detail2.png">
</p>

The statistics page displays some useful metrics for insight into the status of your streaming
Expand All @@ -431,6 +432,11 @@ The tracked operations are listed as follows.
* latestOffset & getOffset: Time taken to query the maximum available offset for this source.
* queryPlanning: Time taken to generates the execution plan.
* walCommit: Time taken to write the offsets to the metadata log.
* **Global Watermark Gap.** The gap between batch timestamp and global watermark for the batch.
* **Aggregated Number Of Total State Rows.** The aggregated number of total state rows.
* **Aggregated Number Of Updated State Rows.** The aggregated number of updated state rows.
* **Aggregated State Memory Used In Bytes.** The aggregated state memory used in bytes.
* **Aggregated Number Of State Rows Dropped By Watermark.** The aggregated number of state rows dropped by watermark.

As an early-release version, the statistics page is still under development and will be improved in
future releases.
Expand Down

0 comments on commit b6a0a7e

Please sign in to comment.