Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP][CARBONDATA-3079] Improve the C++ SDK read performance by merging column in JNI #2912

Closed
wants to merge 1 commit into from

Conversation

xubo245
Copy link
Contributor

@xubo245 xubo245 commented Nov 9, 2018

supprt readNextStringRow and readNextBatchStringRow

The performance between CSDK and SDK(40 million recodes, print num is 100000,6 columns)
This table show: don' t get column, only get row(Object array).

Batch size SDK       CSDK1(don’t get column)       CSDK(don’t get clomun) CSDK2(get column)        
  1 2 3 avg 1 2 3 avg SDK/CSDK1 1 2 3 avg CSDK2/CSDK1
100000 1523573 1474790 1449569 1482644 1380077 1380772 1348124 1369658 1.082 279265 282142 288975 283461 0.207
32000 1547971 1510519 1516607 1525032 1382393 1392961 1391146 1388833 1.098 279020 285982 284344 283115 0.204
10000 1604480 1538599 1574839 1572639 1392247 1413682 1426892 1410940 1.115 293650 289586 278339 287192 0.204
1000 1553553 1555251 1601713 1570172 1425066 1406788 1400285 1410713 1.113 289310 288588 289952 289283 0.205
100 1545616 1577918 1559186 1560907 1376255 1382642 1386771 1381889 1.130 280351 280218 282870 281146 0.203
10 1509486 1511429 1493751 1504889 1209469 1236183 1208908 1218187 1.235 250760 277940 267380 265360 0.218
1 1161568 1128954 1184135 1158219 650563 620164 617285 629337 1.840 228316 215732 227424 223824 0.356
                             
  SDK(get column)                          
  1540277 1509318 1554487 1534694                    
        1.006                    

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

  • Any interfaces changed?

  • Any backward compatibility impacted?

  • Document update required?

  • Testing done
    Please provide details on
    - Whether new unit test cases have been added or why no new tests are required?
    - How it is tested? Please attach test report.
    - Is it a performance related change? Please attach the performance test report.
    - Any additional information to help reviewers in testing this change.

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1349/

@CarbonDataQA
Copy link

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9607/

@CarbonDataQA
Copy link

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1559/

…umn in JNI

supprt readNextStringRow and readNextBatchStringRow

test
@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1537/

@CarbonDataQA
Copy link

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1748/

@CarbonDataQA
Copy link

Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9796/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1979/

@CarbonDataQA
Copy link

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10231/

@CarbonDataQA
Copy link

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2252/

@CarbonDataQA
Copy link

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/53/

@CarbonDataQA
Copy link

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/52/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/441/

@CarbonDataQA
Copy link

Build Success with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/709/

@CarbonDataQA1
Copy link

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/54/

@xubo245 xubo245 closed this Mar 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants