[CARBONDATA-3980] Load fails with aborted exception when Bad records action is unspecified #3919
Conversation
|
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2298/ |
|
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4037/ |
ShreelekhyaG
changed the title
ShreelekhyaG
force-pushed
the
badrecordError
branch
from
5bee86b
to
982ee77
Compare
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2306/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4044/ |
ShreelekhyaG
changed the title
...src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
Outdated
Show resolved
Hide resolved
...spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
Show resolved
Hide resolved
ShreelekhyaG
force-pushed
the
badrecordError
branch
from
982ee77
to
1d9a8c6
Compare
| "DataLoad failure: Data load failed due to bad record: The value with column name " + | ||
| "projectjoindate and column data type TIMESTAMP is not a valid TIMESTAMP type.Please " + | ||
| "enable bad record logger to know the detail reason.")) | ||
| } |
There was a problem hiding this comment.
please drop the table here
| "Timestamp, projectenddate Date,attendance int,utilization int,salary int) PARTITIONED BY " + | ||
| "(deptname String,doj Timestamp,projectcode int) STORED AS carbondata ") | ||
| val csvFilePath = s"$resourcesPath/data.csv" | ||
| val ex = intercept[Exception] { |
There was a problem hiding this comment.
please intercept RuntimeException only
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2334/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4073/ |
ShreelekhyaG
force-pushed
the
badrecordError
branch
from
1d9a8c6
to
4b11d73
Compare
rework test case changes
ShreelekhyaG
force-pushed
the
badrecordError
branch
from
4b11d73
to
47acefa
Compare
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4080/ |
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2340/ |
|
LGTM |
Why is this PR needed?
Load fails with aborted exception when Bad records action is unspecified.
When the partition column is loaded with a bad record value, load fails with 'Job aborted' message in cluster. However in complete stack trace we can see the actual error message. (Like, 'Data load failed due to bad record: The value with column name projectjoindate and column data type TIMESTAMP is not a valid TIMESTAMP type')
What changes were proposed in this PR?
Fix bad record error message for the partition column. Added the error message to
operationContextmap and if its not null throwing exception witherrorMessagefromCarbonLoadDataCommand.Does this PR introduce any user interface change?
Is any new testcase added?