[CARBONDATA-2908]the option of sort_scope don't effects while creating table by dataframe #2684
Conversation
…g table by data frame
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8232/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/161/ |
| @@ -81,6 +81,7 @@ class CarbonDataFrameWriter(sqlContext: SQLContext, val dataFrame: DataFrame) { | |||
| private def makeCreateTableString(schema: StructType, options: CarbonOption): String = { | |||
| val property = Map( | |||
| "SORT_COLUMNS" -> options.sortColumns, | |||
| "SORT_SCOPE" -> options.sortScope, | |||
There was a problem hiding this comment.
From this PR, it seems that carbon will not output error message even if there is some table properties it can not recognized during create table by DataFrame API. I think it is better to add a check for it.
There was a problem hiding this comment.
I can't list all of the properties of table, any class record this ?
There was a problem hiding this comment.
I think you can refer to CREATE TABLE section in http://carbondata.apache.org/data-management-on-carbondata.html
|
@jackylk properties of table can be add to TableProperty in schema.thrift |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8306/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/236/ |
# Conflicts: # integration/spark2/src/main/scala/org/apache/spark/sql/CarbonDataFrameWriter.scala
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/8/ |
|
@jackylk revert the change of thrift, only solve sortScope |
|
retest this please |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/266/ |
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8335/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/97/ |
|
retest this please |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/101/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/269/ |
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8339/ |
|
@qiuchenjian please update the pr's title, doesn't display completely. |
qiuchenjian
changed the title
qiuchenjian
changed the title
|
LGTM |
|
merged, thanks for working on this |
…g table by data frame
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.