Skip to content

Commit

Permalink
[KYUUBI #3539] [FEATURE][TPCDS] Add white list help run the specified…
Browse files Browse the repository at this point in the history
… queries

### _Why are the changes needed?_

Add `white-list` help run the specified queries.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

  `--white-list q4,q15,q19,q25,q42,q49,q60,q66,q68,q72,q76,q78,q79,q89,q97`

![popo_2022-09-22  15-01-05](https://user-images.githubusercontent.com/52876270/191679906-76639468-a67d-408f-8b83-a18b34ef80e7.jpg)

Closes #3539 from zwangsheng/feature/tpcds_white_list.

Closes #3539

7465795 [zwangsheng] fix
26ade91 [Binjie Yang] Update dev/kyuubi-tpcds/src/main/scala/org/apache/kyuubi/tpcds/benchmark/RunBenchmark.scala
11a5b5c [Binjie Yang] Update dev/kyuubi-tpcds/src/main/scala/org/apache/kyuubi/tpcds/benchmark/RunBenchmark.scala
780410a [zwangsheng] fix
1901893 [zwangsheng] naming
1a4d1a2 [zwangsheng] naming
e1bb006 [zwangsheng] fix
1be2bcb [zwangsheng] white

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Binjie Yang <52876270+zwangsheng@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 17289b1)
Signed-off-by: Cheng Pan <chengpan@apache.org>
  • Loading branch information
2 people authored and pan3793 committed Sep 27, 2022
1 parent e41a906 commit b82737c
Showing 1 changed file with 16 additions and 3 deletions.
Expand Up @@ -29,7 +29,8 @@ case class RunConfig(
filter: Option[String] = None,
iterations: Int = 3,
breakdown: Boolean = false,
resultsDir: String = "/spark/sql/performance")
resultsDir: String = "/spark/sql/performance",
queries: Set[String] = Set.empty)

// scalastyle:off
/**
Expand Down Expand Up @@ -65,6 +66,11 @@ object RunBenchmark {
opt[String]('r', "results-dir")
.action((x, c) => c.copy(resultsDir = x))
.text("dir to store benchmark results, e.g. hdfs://hdfs-nn:9870/pref")
opt[String]('q', "queries")
.action { case (x, c) =>
c.copy(queries = x.split(",").map(_.trim).filter(_.nonEmpty).toSet)
}
.text("name of the queries to run, use , split multiple name")
help("help")
.text("prints this usage text")
}
Expand Down Expand Up @@ -96,11 +102,18 @@ object RunBenchmark {
benchmark.tpcds2_4Queries
}

val runQueries =
if (config.queries.nonEmpty) {
allQueries.filter(q => config.queries.contains(q.name.split('-')(0)))
} else {
allQueries
}

println("== QUERY LIST ==")
allQueries.foreach(q => println(q.name))
runQueries.foreach(q => println(q.name))

val experiment = benchmark.runExperiment(
executionsToRun = allQueries,
executionsToRun = runQueries,
includeBreakdown = config.breakdown,
iterations = config.iterations,
tags = Map("host" -> InetAddress.getLocalHost.getHostName))
Expand Down

0 comments on commit b82737c

Please sign in to comment.