Skip to content

Latest commit

 

History

History
27 lines (20 loc) · 772 Bytes

spark-sql-streaming-FileStreamSink.adoc

File metadata and controls

27 lines (20 loc) · 772 Bytes
Raw

FileStreamSink

FileStreamSink is the streaming sink for the parquet format.

Caution
 FIXME 
import scala.concurrent.duration._
import org.apache.spark.sql.streaming.{OutputMode, Trigger}
val out = in.
  writeStream.
  format("parquet").
  option("path", "parquet-output-dir").
  option("checkpointLocation", "checkpoint-dir").
  trigger(Trigger.ProcessingTime(10.seconds)).
  outputMode(OutputMode.Append).
  start

FileStreamSink supports Append output mode only.

It uses spark.sql.streaming.fileSink.log.deletion (as isDeletingExpiredLog)

addBatch Method

Caution
 FIXME