[SPARK-25658][SQL][TEST] Refactor HashByteArrayBenchmark to use main method #22652

wangyum · 2018-10-06T07:53:08Z

What changes were proposed in this pull request?

Refactor HashByteArrayBenchmark to use main method.

use spark-submit:

bin/spark-submit --class  org.apache.spark.sql.HashByteArrayBenchmark --jars ./core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar ./sql/catalyst/target/spark-catalyst_2.11-3.0.0-SNAPSHOT-tests.jar

Generate benchmark result:

SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "catalyst/test:runMain org.apache.spark.sql.HashByteArrayBenchmark"

How was this patch tested?

manual tests

SparkQA · 2018-10-06T09:37:18Z

Test build #97040 has finished for PR 22652 at commit 3e6a058.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-10-06T09:44:41Z

retest this please

SparkQA · 2018-10-06T11:35:05Z

Test build #97043 has finished for PR 22652 at commit 3e6a058.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-10-06T11:50:16Z

retest this please

SparkQA · 2018-10-06T15:44:49Z

Test build #97047 has finished for PR 22652 at commit 3e6a058.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-10-06T16:20:48Z

sql/catalyst/src/test/scala/org/apache/spark/sql/HashByteArrayBenchmark.scala

+ * {{{
+ *   1. without sbt: bin/spark-submit --class <this class> <spark sql test jar>
+ *   2. build/sbt "sql/test:runMain <this class>"
+ *   3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"


sql/test -> catalyst/test.
If we use sql/test, the result will be generated in sql module instead of catalyst module.

It's for both line 32 and 33.

You are right. Thanks

Also, please check line 31, too.

dongjoon-hyun · 2018-10-06T16:43:55Z

Could you review and merge wangyum#17 ?

SparkQA · 2018-10-06T20:34:06Z

Test build #97059 has finished for PR 22652 at commit bdb0549.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-06T20:52:32Z

Test build #97060 has finished for PR 22652 at commit b5190d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-06T22:04:59Z

Test build #97066 has finished for PR 22652 at commit cc268ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-10-06T22:20:45Z

sql/catalyst/src/test/scala/org/apache/spark/sql/HashByteArrayBenchmark.scala

 import org.apache.spark.sql.catalyst.expressions.{HiveHasher, XXH64}
 import org.apache.spark.unsafe.Platform
 import org.apache.spark.unsafe.hash.Murmur3_x86_32

 /**
 * Synthetic benchmark for MurMurHash 3 and xxHash64.
+ * To run this benchmark:
+ * {{{
+ *   1. without sbt: bin/spark-submit --class <this class> <spark catalyst test jar>


Is this a correct guide? BenchmarkBase is in a different jar file, isn't it?

It seems that we missed this because we thought this is a legacy guide which has been worked before.

Yes, you are right:

LM-SHC-16502798:spark yumwang$ bin/spark-submit --class org.apache.spark.sql.HashByteArrayBenchmark ./sql/catalyst/target/spark-catalyst_2.11-3.0.0-SNAPSHOT-tests.jar18/10/07 07:35:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/benchmark/BenchmarkBase at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ......

The correct usage should be:

bin/spark-submit --class org.apache.spark.sql.HashByteArrayBenchmark --jars ./core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar ./sql/catalyst/target/spark-catalyst_2.11-3.0.0-SNAPSHOT-tests.jar

SparkQA · 2018-10-07T07:05:01Z

Test build #97075 has finished for PR 22652 at commit 0a7741a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-07T07:05:01Z

Test build #97074 has finished for PR 22652 at commit 11f2bbe.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2018-10-07T07:07:51Z

retest this please

SparkQA · 2018-10-07T11:03:13Z

Test build #97077 has finished for PR 22652 at commit 0a7741a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

Thank you, @wangyum .

+1, LGTM. Merged to master.

…method ## What changes were proposed in this pull request? Refactor `HashByteArrayBenchmark` to use main method. 1. use `spark-submit`: ```console bin/spark-submit --class org.apache.spark.sql.HashByteArrayBenchmark --jars ./core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar ./sql/catalyst/target/spark-catalyst_2.11-3.0.0-SNAPSHOT-tests.jar ``` 2. Generate benchmark result: ```console SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "catalyst/test:runMain org.apache.spark.sql.HashByteArrayBenchmark" ``` ## How was this patch tested? manual tests Closes apache#22652 from wangyum/SPARK-25658. Lead-authored-by: Yuming Wang <wgyumg@gmail.com> Co-authored-by: Yuming Wang <yumwang@ebay.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

refactor HashByteArrayBenchmark

3e6a058

dongjoon-hyun reviewed Oct 6, 2018

View reviewed changes

Fix scala doc

bdb0549

dongjoon-hyun and others added 2 commits October 6, 2018 17:45

Update result (#17)

b5190d4

Fix scala doc

cc268ca

dongjoon-hyun reviewed Oct 6, 2018

View reviewed changes

wangyum added 2 commits October 7, 2018 07:45

Fix scala doc

11f2bbe

Fix scala doc

0a7741a

dongjoon-hyun approved these changes Oct 7, 2018

View reviewed changes

asfgit closed this in b1328cc Oct 7, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-25658][SQL][TEST] Refactor HashByteArrayBenchmark to use main method #22652

[SPARK-25658][SQL][TEST] Refactor HashByteArrayBenchmark to use main method #22652

wangyum commented Oct 6, 2018 •

edited

SparkQA commented Oct 6, 2018

wangyum commented Oct 6, 2018

SparkQA commented Oct 6, 2018

wangyum commented Oct 6, 2018

SparkQA commented Oct 6, 2018

dongjoon-hyun Oct 6, 2018

dongjoon-hyun Oct 6, 2018

wangyum Oct 6, 2018

dongjoon-hyun Oct 6, 2018

dongjoon-hyun commented Oct 6, 2018

SparkQA commented Oct 6, 2018

SparkQA commented Oct 6, 2018

SparkQA commented Oct 6, 2018

dongjoon-hyun Oct 6, 2018 •

edited

dongjoon-hyun Oct 6, 2018

wangyum Oct 7, 2018

SparkQA commented Oct 7, 2018

SparkQA commented Oct 7, 2018

wangyum commented Oct 7, 2018

SparkQA commented Oct 7, 2018

dongjoon-hyun left a comment

[SPARK-25658][SQL][TEST] Refactor HashByteArrayBenchmark to use main method #22652

[SPARK-25658][SQL][TEST] Refactor HashByteArrayBenchmark to use main method #22652

Conversation

wangyum commented Oct 6, 2018 • edited

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Oct 6, 2018

wangyum commented Oct 6, 2018

SparkQA commented Oct 6, 2018

wangyum commented Oct 6, 2018

SparkQA commented Oct 6, 2018

dongjoon-hyun Oct 6, 2018

Choose a reason for hiding this comment

dongjoon-hyun Oct 6, 2018

Choose a reason for hiding this comment

wangyum Oct 6, 2018

Choose a reason for hiding this comment

dongjoon-hyun Oct 6, 2018

Choose a reason for hiding this comment

dongjoon-hyun commented Oct 6, 2018

SparkQA commented Oct 6, 2018

SparkQA commented Oct 6, 2018

SparkQA commented Oct 6, 2018

dongjoon-hyun Oct 6, 2018 • edited

Choose a reason for hiding this comment

dongjoon-hyun Oct 6, 2018

Choose a reason for hiding this comment

wangyum Oct 7, 2018

Choose a reason for hiding this comment

SparkQA commented Oct 7, 2018

SparkQA commented Oct 7, 2018

wangyum commented Oct 7, 2018

SparkQA commented Oct 7, 2018

dongjoon-hyun left a comment

Choose a reason for hiding this comment

wangyum commented Oct 6, 2018 •

edited

dongjoon-hyun Oct 6, 2018 •

edited