SUMMARY.adoc

Summary

1. Introduction

2. Spark Structured Streaming — Streaming Datasets

Using Structured Streaming

1. DataStreamReader — Loading Datasets from Streaming Data Sources

2. Streaming Source

   1. FileStreamSource

   2. KafkaSource

      1. KafkaRelation

      2. KafkaSourceRDD

      3. CachedKafkaConsumer

      4. KafkaOffsetReader

      5. ConsumerStrategy Contract for KafkaConsumer Providers

      6. KafkaSourceOffset

   3. MemoryStream

   4. RateStreamSource

   5. TextSocketSource

3. Streaming Operators / Streaming Dataset API

   1. dropDuplicates Operator — Streaming Deduplication

   2. explain Operator — Explaining Query Plan

   3. groupBy Operator — Streaming Aggregation (with Implicit State Logic)

4. KeyValueGroupedDataset — Streaming Aggregation

   1. mapGroupsWithState Operator — Stateful Streaming Aggregation (with Explicit State Logic)

   2. flatMapGroupsWithState Operator — Arbitrary Stateful Streaming Aggregation (with Explicit State Logic)

5. DataStreamWriter — Writing Datasets To Streaming Data Sinks

   1. ForeachWriter

   2. OutputMode

   3. Trigger — How Frequently to Check Sources For New Data

6. Streaming Sink

   1. ConsoleSink for Showing DataFrames to Console

   2. FileStreamSink

   3. ForeachSink

   4. KafkaSink

   5. MemorySink

7. StreamingQuery

8. StreamingQueryManager

   1. UnsupportedOperationChecker

9. Configuration Properties

Monitoring

1. StreamingQueryListener

   1. QueryStartedEvent

   2. QueryProgressEvent

   3. QueryTerminatedEvent

2. Web UI

3. Logging

Extending Structured Streaming

1. StreamSourceProvider — Streaming Data Source Provider

   1. KafkaSourceProvider — Data Source Provider for Apache Kafka

   2. RateSourceProvider

   3. TextSocketSourceProvider

2. StreamSinkProvider

   1. ConsoleSinkProvider

Query Planning and Execution

1. StreamExecution — Execution Environment of Streaming Dataset

   1. ProgressReporter

2. TriggerExecutor

3. IncrementalExecution — QueryExecution of Streaming Datasets

4. StreamingQueryListenerBus

Logical Operators

1. EventTimeWatermark Unary Logical Operator

2. FlatMapGroupsWithState Unary Logical Operator

3. Deduplicate Unary Logical Operator

4. MemoryPlan Logical Query Plan

5. StreamingRelation Leaf Logical Operator

6. StreamingExecutionRelation Leaf Logical Operator to Represent Source At Execution

Physical Operators

1. EventTimeWatermarkExec Unary Physical Operator for Accumulating Event Time Watermark

   1. EventTimeStatsAccum Accumulator

2. FlatMapGroupsWithStateExec Unary Physical Operator

3. StateStoreRestoreExec Unary Physical Operator for Restoring Streaming Aggregates

4. StateStoreSaveExec Unary Physical Operator for Saving State of Streaming Aggregates

5. StreamingDeduplicateExec Unary Physical Operator for Streaming Deduplication

6. StreamingRelationExec Leaf Physical Operator

7. WatermarkSupport Contract for Streaming Watermark in Unary Physical Operators

Execution Planning Strategies

1. FlatMapGroupsWithStateStrategy Execution Planning Strategy for FlatMapGroupsWithState Logical Operator

2. StatefulAggregationStrategy Execution Planning Strategy for EventTimeWatermark and Aggregate Logical Operators

3. StreamingDeduplicationStrategy Execution Planning Strategy for Deduplicate Logical Operator

4. StreamingRelationStrategy Execution Planning Strategy for StreamingRelation and StreamingExecutionRelation Logical Operators

Offsets and Checkpointing

1. Offset

2. MetadataLog — Contract for Metadata Storage

3. HDFSMetadataLog — MetadataLog with Hadoop HDFS for Storage

   1. BatchCommitLog — HDFSMetadataLog for Batch Completion Log

   2. CompactibleFileStreamLog

   3. OffsetSeqLog — HDFSMetadataLog with OffsetSeq Metadata

4. OffsetSeqMetadata

State in Stateful Streaming Aggregations

1. GroupState — State Per Group in Stateful Streaming Aggregation

   1. GroupStateImpl

   2. GroupStateTimeout

2. StateStore

   1. StateStoreOps — Implicits Methods for Creating StateStoreRDD

   2. StateStoreProvider

   3. StateStoreUpdater

   4. StateStoreWriter — Recording Metrics For Writing to StateStore

3. HDFSBackedStateStore

   1. HDFSBackedStateStoreProvider

4. StateStoreRDD — RDD for Updating State (in StateStore)

5. StateStoreCoordinator — Tracking Locations of StateStores for StateStoreRDD

   1. StateStoreCoordinatorRef Interface for Communication with StateStoreCoordinator

Varia

1. StreamProgress Custom Scala Map