Please sign in to comment.
[SPARK-23288][SS] Fix output metrics with parquet sink
## What changes were proposed in this pull request? Output metrics were not filled when parquet sink used. This PR fixes this problem by passing a `BasicWriteJobStatsTracker` in `FileStreamSink`. ## How was this patch tested? Additional unit test added. Author: Gabor Somogyi <email@example.com> Closes #20745 from gaborgsomogyi/SPARK-23288. (cherry picked from commit 918c7e9) Signed-off-by: Wenchen Fan <firstname.lastname@example.org>
- Loading branch information...
Showing with 82 additions and 16 deletions.
- +1 −10 sql/core/src/main/scala/org/apache/spark/sql/execution/command/DataWritingCommand.scala
- +21 −4 sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
- +8 −2 sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
- +52 −0 sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala