Skip to content

[SPARK-25657][SQL][TEST] Refactor HashBenchmark to use main method#22651

Closed
wangyum wants to merge 4 commits intoapache:masterfrom
wangyum:SPARK-25657
Closed

[SPARK-25657][SQL][TEST] Refactor HashBenchmark to use main method#22651
wangyum wants to merge 4 commits intoapache:masterfrom
wangyum:SPARK-25657

Conversation

@wangyum
Copy link
Member

@wangyum wangyum commented Oct 6, 2018 •
edited
Loading

What changes were proposed in this pull request?

Refactor HashBenchmark to use main method.

  1. use spark-submit:
bin/spark-submit --class  org.apache.spark.sql.HashBenchmark --jars ./core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar ./sql/catalyst/target/spark-catalyst_2.11-3.0.0-SNAPSHOT-tests.jar
  1. Generate benchmark result:
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "catalyst/test:runMain org.apache.spark.sql.HashBenchmark"

How was this patch tested?

manual tests

@wangyum wangyum changed the title [SPARK-25488][SQL][TEST] Refactor HashBenchmark to use main method [SPARK-25657][SQL][TEST] Refactor HashBenchmark to use main method Oct 6, 2018
@SparkQA
Copy link

SparkQA commented Oct 6, 2018

Test build #97039 has finished for PR 22651 at commit 53e3b7b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun reviewed Oct 6, 2018
View reviewed changes
sql/catalyst/src/test/scala/org/apache/spark/sql/HashBenchmark.scala Outdated
* To run this benchmark:
* {{{
* 1. without sbt: bin/spark-submit --class <this class> <spark sql test jar>
* 2. build/sbt "sql/test:runMain <this class>"
Copy link
Member

@dongjoon-hyun dongjoon-hyun Oct 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sql/test -> catalyst/test

dongjoon-hyun
dongjoon-hyun reviewed Oct 6, 2018
View reviewed changes
sql/catalyst/src/test/scala/org/apache/spark/sql/HashBenchmark.scala Outdated
* {{{
* 1. without sbt: bin/spark-submit --class <this class> <spark sql test jar>
* 2. build/sbt "sql/test:runMain <this class>"
* 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
Copy link
Member

@dongjoon-hyun dongjoon-hyun Oct 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto.

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Oct 6, 2018

Also, please review and merge wangyum#16 .

@SparkQA
Copy link

SparkQA commented Oct 6, 2018

Test build #97057 has finished for PR 22651 at commit 871b1a9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 6, 2018

Test build #97067 has finished for PR 22651 at commit 2a512ce.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Oct 6, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Oct 7, 2018

Test build #97069 has finished for PR 22651 at commit 2a512ce.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 7, 2018

Test build #97076 has finished for PR 22651 at commit 7ec7237.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Oct 7, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Oct 7, 2018

Test build #97078 has finished for PR 22651 at commit 7ec7237.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member Author

wangyum commented Oct 7, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Oct 7, 2018

Test build #97079 has finished for PR 22651 at commit 7ec7237.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun approved these changes Oct 7, 2018
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @wangyum .

+1, LGTM. Merged to master.

@asfgit asfgit closed this in 669ade3 Oct 7, 2018
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
@wangyum @dongjoon-hyun
[SPARK-25657][SQL][TEST] Refactor HashBenchmark to use main method
997fbe5 
## What changes were proposed in this pull request?

Refactor `HashBenchmark` to use main method.
1. use `spark-submit`:
```console
bin/spark-submit --class  org.apache.spark.sql.HashBenchmark --jars ./core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar ./sql/catalyst/target/spark-catalyst_2.11-3.0.0-SNAPSHOT-tests.jar
```

2. Generate benchmark result:
```console
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "catalyst/test:runMain org.apache.spark.sql.HashBenchmark"
```

## How was this patch tested?
manual tests

Closes apache#22651 from wangyum/SPARK-25657.

Lead-authored-by: Yuming Wang <wgyumg@gmail.com>
Co-authored-by: Yuming Wang <yumwang@ebay.com>
Co-authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wangyum @SparkQA @dongjoon-hyun