Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-23288][SS] Fix output metrics with parquet sink #20639

Closed
wants to merge 2 commits into from

Conversation

gaborgsomogyi
Copy link
Contributor

What changes were proposed in this pull request?

Output metrics were not filled when parquet sink used.

This PR fixes this problem by passing a BasicWriteJobStatsTracker in FileStreamSink.

How was this patch tested?

Additional unit test added.

@gaborgsomogyi
Copy link
Contributor Author

cc @tdas @viirya @vanzin

@tdas
Copy link
Contributor

tdas commented Feb 28, 2018

@zsxwing as well.

@tdas
Copy link
Contributor

tdas commented Feb 28, 2018

this is ok to test

@SparkQA
Copy link

SparkQA commented Feb 28, 2018

Test build #4137 has finished for PR 20639 at commit 22e6ca1.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@vanzin
Copy link
Contributor

vanzin commented Mar 1, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Mar 2, 2018

Test build #87856 has finished for PR 20639 at commit 55aa8bc.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class CatalogColumnStat(
  • case class LocalRelation(
  • case class StreamingDataSourceV2Relation(

@gaborgsomogyi
Copy link
Contributor Author

Executed these tests manually again but working fine. Seems like flaky.

@vanzin
Copy link
Contributor

vanzin commented Mar 5, 2018

retest this please

@gaborgsomogyi
Copy link
Contributor Author

God, seems like stuck somehow. I'll re-create the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants