[CARBONDATA-2912] Support CSV table load csv data with spark2.2 #2691
Conversation
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8288/ |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/219/ |
|
@xubo245 What does 'CSV table' mean in the title? Can you explain why the version upgrading for common-langs is needed? Does it improve something? (Just curious) |
|
Carbon need upgrade commons-lang3 'vision' -- > 'version' |
|
@xuchuanyin CSV table is "create table ... using csv options...". In branch-1.3, CSV table cann't load csv data with spark2.2 because: If using low commons-lang version, the default for the timestampFormat is yyyy-MM-dd'T'HH:mm:ss.SSSXXX which is an illegal argument and can not be recognized after upgrade spark from 2.1 to 2.2. It needs to be set when you are writing the dataframe out. |
|
retest this please |
|
Oh, I didn't know this grammar before. Is CSV table a carbon table or spark table? |
|
spark，it support using parquet too |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8313/ |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/243/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/3/ |
|
retest this please |
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8340/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/102/ |
|
retest this please |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/270/ |
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8343/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/105/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/273/ |
|
@jackylk Please review it. master has updated, but branch-1.3 didn't fix it. |
| @@ -0,0 +1,4 @@ | |||
| name,age | |||
There was a problem hiding this comment.
can you reuse the current csv file, no need to add new one.
1.upgrade commons-lang3 version
2.add some test case
xubo245
force-pushed
the
CARBONDATA-2912_twoInsert1.3.2
branch
from
ae4fa80
to
8445a94
Compare
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.3/8512/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/273/ |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/442/ |
|
retest this please |
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8536/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/291/ |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/466/ |
|
retest this please |
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8539/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/294/ |
|
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/469/ |
|
retest this please |
|
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8545/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/300/ |
|
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/475/ |
|
Branch-1.3 don't support spark2.3, so CI will not work. Please review it. @chenliang613 @xuchuanyin @jackylk |
|
LGTM |
1.upgrade commons-lang3 version
2.add some test case
This closes #2691
|
merged to branch-1.3 |
In branch-1.3, CSV table cann't load csv data with spark2.2 because: If using low commons-lang version, the default for the timestampFormat is yyyy-MM-dd'T'HH:mm:ss.SSSXXX which is an illegal argument and can not be recognized after upgrade spark from 2.1 to 2.2. It needs to be set when you are writing the dataframe out.
Carbon need upgrade commons-lang3 version
CSV table is "create table ... using csv options..."
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
No
No
No
add some test case
No