[SPARK-14078] Streaming Parquet Based FileSink #11897
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val message: String, | ||
| val cause: Throwable, | ||
| val startOffset: Option[Offset] = None, | ||
| val endOffset: Option[Offset] = None | ||
| ) extends Exception(message, cause) { | ||
| ) extends Exception(message, cause) with Serializable { |
There was a problem hiding this comment.
nit: Exception has already extended Serializable. Not need to add it again.
|
Test build #53807 has finished for PR 11897 at commit
|
| path: String, | ||
| fileFormat: FileFormat) extends Sink with Logging { | ||
|
|
||
| val basePath = new Path(path) |
| .startStream(outputDir) | ||
|
|
||
| inputData.addData(1, 2, 3) | ||
| failAfter(streamingTimeout) { query.processAllAvailable() } |
There was a problem hiding this comment.
There is a race condition here: noNewData may become true before processAllAvailable.
There was a problem hiding this comment.
Never mind. I just realized it will continue to set noNewData to true
|
Looks pretty good. Just some nits. |
|
Test build #53841 has finished for PR 11897 at commit
|
|
Test build #53847 has finished for PR 11897 at commit
|
| } | ||
|
|
||
| override def getOffset: Option[Offset] = Some(fetchMaxOffset()).filterNot(_.offset == -1) | ||
|
|
||
| override def toString: String = s"FileSink[$path]" |
There was a problem hiding this comment.
Haha, yes. I'll fix this in a follow up.
kiku-jw
pushed a commit
to kiku-jw/spark
that referenced
this pull request
Jun 26, 2019
…eaming Sink ## What changes were proposed in this pull request? File source V1 supports reading output of FileStreamSink as batch. apache#11897 We should support this in file source V2 as well. When reading with paths, we first check if there is metadata log of FileStreamSink. If yes, we use `MetadataLogFileIndex` for listing files; Otherwise, we use `InMemoryFileIndex`. ## How was this patch tested? Unit test Closes apache#24900 from gengliangwang/FileStreamV2. Authored-by: Gengliang Wang <gengliang.wang@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new
Sinkimplementation that writes out Parquet files. In order to correctly handle partial failures while maintaining exactly once semantics, the files for each batch are written out to a unique directory and then atomically appended to a metadata log. When a parquet basedDataSourceis initialized for reading, we first check for this log directory and use it instead of file listing when present.Unit tests are added, as well as a stress test that checks the answer after non-deterministic injected failures.