Skip to content

[Spark] Delete dataFile on iteration level in SparkParquetWritersFlatDataBenchmark#4149

Merged
rdblue merged 1 commit intoapache:masterfrom
yaooqinn:minor
Feb 21, 2022
Merged

[Spark] Delete dataFile on iteration level in SparkParquetWritersFlatDataBenchmark#4149
rdblue merged 1 commit intoapache:masterfrom
yaooqinn:minor

Conversation

@yaooqinn
Copy link
Member

This PR fixes a minor issue for a jmh test - SparkParquetWritersFlatDataBenchmark

The dataFile holds mocked outputs now cleaned at Level.Trial which will be executed after the set of benchmark iterations. Then, it cause errors like below.

# Warmup Iteration   1: 7.251 s/op
# Warmup Iteration   2: <failure>

org.apache.iceberg.exceptions.AlreadyExistsException: File already exists: /Users/kentyao/iceberg/spark/v3.2/spark/build/tmp/jmh/parquet-flat-data-benchmark3385433263675547096.parquet
	at org.apache.iceberg.Files$LocalOutputFile.create(Files.java:58)
	at org.apache.iceberg.parquet.ParquetIO$ParquetOutputFile.create(ParquetIO.java:148)

In this PR, I set the TearDown level to Iteration to make the dataFile be deleted after each iteration.

@github-actions github-actions bot added the spark label Feb 17, 2022
@rdblue
Copy link
Contributor

rdblue commented Feb 20, 2022

@nastra or @RussellSpitzer, can you review this one? Thank you!

Copy link
Contributor

@nastra nastra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nastra
Copy link
Contributor

nastra commented Feb 21, 2022

@rdblue or @RussellSpitzer when you merge this one, please also merge #3910 as it fixes the same issue for Spark 3.2 benchmarks

@rdblue rdblue merged commit 888e599 into apache:master Feb 21, 2022
@rdblue
Copy link
Contributor

rdblue commented Feb 21, 2022

Thanks, @yaooqinn!

arminnajafi pushed a commit to arminnajafi/iceberg that referenced this pull request Feb 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants