Skip to content

Latest commit

 

History

History
130 lines (102 loc) · 2.96 KB

pyspark.streaming.rst

File metadata and controls

130 lines (102 loc) · 2.96 KB
Raw

Spark Streaming

Core Classes

pyspark.streaming

StreamingContext DStream

Streaming Management

pyspark.streaming

StreamingContext.addStreamingListener StreamingContext.awaitTermination StreamingContext.awaitTerminationOrTimeout StreamingContext.checkpoint StreamingContext.getActive StreamingContext.getActiveOrCreate StreamingContext.getOrCreate StreamingContext.remember StreamingContext.sparkContext StreamingContext.start StreamingContext.stop StreamingContext.transform StreamingContext.union

Input and Output

StreamingContext.binaryRecordsStream StreamingContext.queueStream StreamingContext.socketTextStream StreamingContext.textFileStream DStream.pprint DStream.saveAsTextFiles

Transformations and Actions

pyspark.streaming

DStream.cache DStream.checkpoint DStream.cogroup DStream.combineByKey DStream.context DStream.count DStream.countByValue DStream.countByValueAndWindow DStream.countByWindow DStream.filter DStream.flatMap DStream.flatMapValues DStream.foreachRDD DStream.fullOuterJoin DStream.glom DStream.groupByKey DStream.groupByKeyAndWindow DStream.join DStream.leftOuterJoin DStream.map DStream.mapPartitions DStream.mapPartitionsWithIndex DStream.mapValues DStream.partitionBy DStream.persist DStream.reduce DStream.reduceByKey DStream.reduceByKeyAndWindow DStream.reduceByWindow DStream.repartition DStream.rightOuterJoin DStream.slice DStream.transform DStream.transformWith DStream.union DStream.updateStateByKey DStream.window

Kinesis

pyspark.streaming.kinesis

KinesisUtils.createStream InitialPositionInStream.LATEST InitialPositionInStream.TRIM_HORIZON