[CARBONDATA-1899] Add CarbonData concurrency test case #1670

xubo245 · 2017-12-15T14:44:25Z

Add CarbonData concurrency test case

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed?
No
Any backward compatibility impacted?
No
Document update required?
No
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
No

CarbonDataQA · 2017-12-15T16:45:08Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2014/

CarbonDataQA · 2017-12-15T19:22:14Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/794/

ravipesala · 2017-12-15T22:24:53Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2337/

jackylk · 2017-12-17T03:23:22Z

...est/src/test/scala/org/apache/carbondata/spark/testsuite/dataretention/ConcurrencyTest.scala

+ * limitations under the License.
+ */
+
+package org.apache.carbondata.spark.testsuite.dataretention


You do not need to add in testcase, I think you can add it in a separated benchmark module

Do you mean add it into example module?

CarbonDataQA · 2017-12-17T13:54:40Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2074/

ravipesala · 2017-12-17T14:09:07Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2375/

CarbonDataQA · 2017-12-17T14:09:42Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/849/

CarbonDataQA · 2017-12-17T14:40:18Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/851/

CarbonDataQA · 2017-12-17T14:51:19Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2076/

ravipesala · 2017-12-17T15:24:38Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2377/

ravipesala · 2017-12-17T16:02:25Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2378/

CarbonDataQA · 2017-12-18T09:11:23Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/862/

CarbonDataQA · 2017-12-18T09:13:52Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2087/

ravipesala · 2017-12-18T09:55:31Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2385/

xubo245 · 2017-12-18T11:03:07Z

retest this please

ManoharVanam · 2017-12-18T12:35:36Z

examples/spark2/src/main/scala/org/apache/carbondata/examples/ConcurrencyTest.scala

+      val executorService = Executors.newFixedThreadPool(ThreadNum)
+      val results = new util.ArrayList[Future[Results]]()
+      for (num <- (1 to TaskNum).par) {
+        results.add(executorService.submit(new QueryTask(spark, sqlText)))


For better concurrency, I think invokeAll is better option instead of submitting task every time

Ok, I am testing the invokeALl in local

CarbonDataQA · 2017-12-18T13:21:20Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/872/

CarbonDataQA · 2017-12-18T13:29:38Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2097/

CarbonDataQA · 2017-12-19T15:36:23Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2135/

CarbonDataQA · 2017-12-19T15:40:33Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/911/

ravipesala · 2017-12-19T18:46:34Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2424/

manishgupta88 · 2017-12-20T04:58:11Z

retest sdv please

manishgupta88 · 2017-12-20T06:35:41Z

examples/spark2/src/main/scala/org/apache/carbondata/examples/ConcurrencyTest.scala

+      val sql = s"query ${index + 1}: $sqlText "
+      printResult(results, sql)
+      executorService.shutdown()
+      executorService.awaitTermination(600, TimeUnit.SECONDS)


why printResult is getting called before shutting down executorService?

Because printResult invoke get() method(java.util.concurrent.Future#get()), it waits if necessary for the computation to complete, and then retrieves its result. So the tasks have finished when shutting down executorService

manishgupta88 · 2017-12-20T06:46:10Z

examples/spark2/src/main/scala/org/apache/carbondata/examples/ConcurrencyTest.scala

+  val r = new Random()
+  val tmpId = r.nextInt(cardinalityId) % totalNum
+  val tmpCity = "city" + (r.nextInt(cardinalityCity) % totalNum)
+  val queries: Array[Query] = Array(


I think it is better to give an option to read queries, table name and database name from external file...so that any number of queries can be added without modifying the code.....This will also help to standalone run this class and specifying the query file path as an argument to it.
If not now, you can enhance the framework in another PR

OK, I will enhance the framework.

manishgupta88 · 2017-12-20T06:50:25Z

examples/spark2/src/main/scala/org/apache/carbondata/examples/ConcurrencyTest.scala

+      .getOrCreateCarbonSession(storeLocation)
+    spark.sparkContext.setLogLevel("warn")
+
+    doParameter(args)


Change the method name to initParameters

manishgupta88 · 2017-12-20T06:51:35Z

examples/spark2/src/main/scala/org/apache/carbondata/examples/ConcurrencyTest.scala

+    val sortTimeArray = timeArray.sorted
+
+    val time90 = ((sortTimeArray.length) * 0.9).toInt - 1
+    val time99 = ((sortTimeArray.length) * 0.99).toInt - 1


Please add a detailed comment for your printing logic and explain what is time90 and time99

ravipesala · 2017-12-20T07:06:27Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2445/

CarbonDataQA · 2017-12-20T08:55:46Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/938/

CarbonDataQA · 2017-12-20T08:57:44Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2167/

manishgupta88 · 2017-12-20T13:20:30Z

LGTM...once build runs I will merge

xubo245 · 2017-12-21T01:40:43Z

retest this please

CarbonDataQA · 2017-12-21T02:51:35Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/973/

CarbonDataQA · 2017-12-21T02:52:06Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2196/

Add CarbonData concurrency test case move test to excample module add doParameter fix scala style change the code by review comment

CarbonDataQA · 2017-12-21T04:41:49Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/977/

CarbonDataQA · 2017-12-21T05:15:28Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2201/

manishgupta88 · 2017-12-21T06:40:01Z

LGTM

Add CarbonData concurrency test case This closes apache#1670

jackylk reviewed Dec 17, 2017

View reviewed changes

xubo245 force-pushed the concurrentTest branch 2 times, most recently from 7de9899 to 8618c3e Compare December 17, 2017 13:48

xubo245 force-pushed the concurrentTest branch from 8618c3e to 08c885f Compare December 17, 2017 13:56

xubo245 force-pushed the concurrentTest branch from 08c885f to 054a623 Compare December 18, 2017 08:43

ManoharVanam reviewed Dec 18, 2017

View reviewed changes

xubo245 force-pushed the concurrentTest branch from 054a623 to 213c964 Compare December 19, 2017 13:17

manishgupta88 suggested changes Dec 20, 2017

View reviewed changes

xubo245 force-pushed the concurrentTest branch from 75f5432 to 8116b3a Compare December 20, 2017 11:37

[CARBONDATA-1899] Add CarbonData concurrency test case

dd4f3dc

Add CarbonData concurrency test case move test to excample module add doParameter fix scala style change the code by review comment

xubo245 force-pushed the concurrentTest branch from 8116b3a to dd4f3dc Compare December 21, 2017 03:13

asfgit closed this in 71959da Dec 21, 2017

SangeetaGulia pushed a commit to SangeetaGulia/incubator-carbondata that referenced this pull request Jan 3, 2018

[CARBONDATA-1899] Add CarbonData concurrency test case

48720b0

Add CarbonData concurrency test case This closes apache#1670

jatin9896 pushed a commit to jatin9896/incubator-carbondata that referenced this pull request Jan 5, 2018

[CARBONDATA-1899] Add CarbonData concurrency test case

b7ccf28

Add CarbonData concurrency test case This closes apache#1670

anubhav100 pushed a commit to anubhav100/incubator-carbondata that referenced this pull request Jun 22, 2018

[CARBONDATA-1899] Add CarbonData concurrency test case

8e2d15d

Add CarbonData concurrency test case This closes apache#1670

[CARBONDATA-1899] Add CarbonData concurrency test case #1670

[CARBONDATA-1899] Add CarbonData concurrency test case #1670

Conversation

xubo245 commented Dec 15, 2017

CarbonDataQA commented Dec 15, 2017

CarbonDataQA commented Dec 15, 2017

ravipesala commented Dec 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Dec 17, 2017

ravipesala commented Dec 17, 2017

CarbonDataQA commented Dec 17, 2017

CarbonDataQA commented Dec 17, 2017

CarbonDataQA commented Dec 17, 2017

ravipesala commented Dec 17, 2017

ravipesala commented Dec 17, 2017

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 18, 2017

ravipesala commented Dec 18, 2017

xubo245 commented Dec 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 19, 2017

CarbonDataQA commented Dec 19, 2017

ravipesala commented Dec 19, 2017

manishgupta88 commented Dec 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravipesala commented Dec 20, 2017

CarbonDataQA commented Dec 20, 2017

CarbonDataQA commented Dec 20, 2017

manishgupta88 commented Dec 20, 2017

xubo245 commented Dec 21, 2017

CarbonDataQA commented Dec 21, 2017

CarbonDataQA commented Dec 21, 2017

CarbonDataQA commented Dec 21, 2017

CarbonDataQA commented Dec 21, 2017

manishgupta88 commented Dec 21, 2017