Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CARBONDATA-1282]Choose BatchedDatasource scan only if schema fits for codegen #1148

Closed
wants to merge 1 commit into from

Conversation

ashokblend
Copy link
Contributor

@ashokblend ashokblend commented Jul 8, 2017

Problem
When table is having large no of column say 2000, then spark gives code generation issue during full scan query as size of generated code exceeds 64KB.
Solution
As in code, we have two BatchedDataSourceScan and RowDataSourceScan to scan query. As per implementation BatchedDataSourceScan is used when code generation is supported else RowDataSourceScan . Spark checks the configuration spark.sql.codegen.wholeStage is enabled and also column size should not exceeds its configuration i.e spark.sql.codegen.maxFields. Hence we need to add one more check for spark.sql.codegen.maxFields.
Testing
Tested manually.

@asfgit
Copy link

asfgit commented Jul 8, 2017

Can one of the admins verify this patch?

1 similar comment
@asfgit
Copy link

asfgit commented Jul 8, 2017

Can one of the admins verify this patch?

@CarbonDataQA
Copy link

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/372/

@CarbonDataQA
Copy link

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2960/

@ashokblend ashokblend changed the title [WIP]Choose BatchedDatasource scan only if schema fits for codegen [CARBONDATA-1282]Choose BatchedDatasource scan only if schema fits for codegen Jul 10, 2017
@gvramana
Copy link
Contributor

LGTM

@asfgit asfgit closed this in 619f1f9 Jul 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants