Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CARBONDATA-1086]Added documentation for BATCH SORT SUPPORT FOR DATA LOADING #943

Closed
wants to merge 2 commits into from
Closed

[CARBONDATA-1086]Added documentation for BATCH SORT SUPPORT FOR DATA LOADING #943

wants to merge 2 commits into from

Conversation

vandana7
Copy link
Contributor

No description provided.

@asfbot
Copy link

asfbot commented May 24, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/99/

@asfbot
Copy link

asfbot commented May 24, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/101/

Failed Tests: 2

carbondata-pr-spark-2.1/org.apache.carbondata:carbondata-core: 2

@asfbot
Copy link

asfbot commented May 28, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/31/

Failed Tests: 0


Test FAILed.

@asfbot
Copy link

asfbot commented May 28, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/223/

Build result: ABORTED

[...truncated 662.51 KB...] at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoFailureException: Timed out after 0 seconds waiting for forked process to complete. at org.scalatest.tools.maven.AbstractScalaTestMojo.runForkingOnce(AbstractScalaTestMojo.java:319) at org.scalatest.tools.maven.AbstractScalaTestMojo.runScalaTest(AbstractScalaTestMojo.java:242) at org.scalatest.tools.maven.TestMojo.execute(TestMojo.java:106) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) ... 31 more[ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn -rf :carbondata-spark-common-testBuild was abortedAborted by chenliang613channel stoppedSetting status of 5f02367 to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-2.1/223/ and message: 'Build finished. 'Using context: Jenkins (Spark 2.1): Maven clean install
Test FAILed.

@CarbonDataQA
Copy link

Can one of the admins verify this patch?

2 similar comments
@asfbot
Copy link

asfbot commented May 31, 2017

Can one of the admins verify this patch?

@CarbonDataQA
Copy link

Can one of the admins verify this patch?

@asfbot
Copy link

asfbot commented Jun 1, 2017

Tests Failed for Spark2.1

@asfbot
Copy link

asfbot commented Jun 1, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/20/

@asfgit
Copy link

asfgit commented Jun 14, 2017

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/342/

Failed Tests: 1

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 1

sgururajshetty
sgururajshetty suggested changes Jun 14, 2017
View reviewed changes
Copy link
Contributor

@sgururajshetty sgururajshetty left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. change mb to MB

  2. Re confirm this parameter name carbon.load.batch.sort.size.inmb i think it should be carbon.batch.sort.size.inmb

  3. Change carbon.load.batch.sort.size.inmb description to :
    Specifies the size of data to be considered for batch sorting during data load. Recommended value is less than 45% of the total sort data (sort.inmemory.size.inmb), otherwise it may start spilling the intermediate temp data to disk, so it may slow down the loading process.

  4. BATCH_SORT i think it has changed. Please refere the latest implementation for all the document update. Below are the details for reference:
    SORT_SCOPE: This option specifies the scope of the sort during data load. There are two types of sort scope, BATCH_SORT and LOCAL_SORT. BATCH_SORT will increase the load performance but decreases the query performance. By default the sort scope is LOCAL_SORT.
    OPTIONS ('SORT_SCOPE'='BATCH_SORT')
    You can also specify the sort size option for sort scope.
    OPTIONS ('SORT_SCOPE'='BATCH_SORT', 'batch_sort_size_inmb'='7')

@CarbonDataQA
Copy link

Can one of the admins verify this patch?

@vandana7
Copy link
Contributor Author

  1. I have confirmed that "carbon.load.batch.sort.size.inmb" property is present CarbonCommonConstants class. I have also updated the description for the same.
  2. I have also added "carbon.load.sort.scope" property in the documentation.
    Please review

@CarbonDataQA
Copy link

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/366/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2953/

@sgururajshetty
Copy link
Contributor

@vandana7 there is a space missing after a . in a sentence.
Also under DML operation we will have the main section as sort_scope and not batch_sort. Then we will explain about the different types of SORT_SCOPE.

@CarbonDataQA
Copy link

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/411/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3192/

@CarbonDataQA
Copy link

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/597/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3204/

@CarbonDataQA
Copy link

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/609/

@vandana7 vandana7 closed this Jul 26, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgururajshetty sgururajshetty sgururajshetty requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@vandana7 @asfbot @CarbonDataQA @asfgit @sgururajshetty @manishgupta88