New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-25510][TEST] Create new trait replace BenchmarkWithCodegen #22522
Conversation
Test build #96452 has finished for PR 22522 at commit
|
* Common base trait for micro benchmarks that are supposed to run standalone (i.e. not together | ||
* with other benchmarks). | ||
*/ | ||
private[benchmark] trait RunBenchmarkWithCodegen { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shall this extend BenchmarkBase
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd remove private[benchmark]
to be consistent with other benchmark classes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
extends BenchmarkBase
and add getSparkSession
function thus subclass can build their own SparkSession
:
spark/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/RunBenchmarkWithCodegen.scala
Lines 28 to 58 in 42230b6
trait RunBenchmarkWithCodegen extends BenchmarkBase { | |
val spark: SparkSession = getSparkSession | |
/** Subclass can override this function to build their own SparkSession */ | |
def getSparkSession: SparkSession = { | |
SparkSession.builder() | |
.master("local[1]") | |
.appName(this.getClass.getCanonicalName) | |
.config(SQLConf.SHUFFLE_PARTITIONS.key, 1) | |
.config(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key, 1) | |
.getOrCreate() | |
} | |
/** Runs function `f` with whole stage codegen on and off. */ | |
def runBenchmark(name: String, cardinality: Long)(f: => Unit): Unit = { | |
val benchmark = new Benchmark(name, cardinality, output = output) | |
benchmark.addCase(s"$name wholestage off", numIters = 2) { iter => | |
spark.sqlContext.conf.setConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED, value = false) | |
f | |
} | |
benchmark.addCase(s"$name wholestage on", numIters = 5) { iter => | |
spark.sqlContext.conf.setConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED, value = true) | |
f | |
} | |
benchmark.run() | |
} | |
} |
I think this change is necessary, but I'd like to migrate one benchmark to use this new trait as an example. We can migrate others in follow up PRs. |
Thanks @cloud-fan I have migrate |
Test build #96783 has finished for PR 22522 at commit
|
What changes were proposed in this pull request?
We need create a new trait to replace
BenchmarkWithCodegen
asBenchmarkWithCodegen
extends fromSparkFunSuite
.For example. when doing
AggregateBenchmark
refactor.Before this change:
After this change:
All these benchmarks will use this trait:
How was this patch tested?
manual tests