[HUDI-1662] Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType #2634
Conversation
|
cc @garyli1019 , could you take a look |
There was a problem hiding this comment.
@xiarixiaoyao thanks for your contribution, LGTM.
Looks like the CI is not triggered, would you try to force push again?
|
Thank you @garyli1019 |
|
@xiarixiaoyao your force push already triggered the CI. Do you mean JIRA contributor access? If so, would you send an email to the dev mailing list with your JIRA ID? That's how we usually grant access to new contributors. Thanks |
Codecov Report
@@ Coverage Diff @@
## master #2634 +/- ##
=============================================
+ Coverage 51.58% 69.48% +17.90%
+ Complexity 3285 363 -2922
=============================================
Files 446 53 -393
Lines 20409 1963 -18446
Branches 2116 235 -1881
=============================================
- Hits 10528 1364 -9164
+ Misses 9003 465 -8538
+ Partials 878 134 -744
Flags with carried forward coverage won't be shown. Click here to find out more. |
…i mor table contains date type
|
cc @garyli1019 . Sorry for the late reply。 now the ci is pass, could you check and merge |
Tips
What is the purpose of the pull request
fixed the bug that: Failed to query real-time view use hive/spark-sql when hudi mor table contains dateType
test step:
step1: prepare raw DataFrame with DateType, and insert it to HudiMorTable
df_raw.withColumn("date", lit(Date.valueOf("2020-11-10")))
merge(df_raw, "bulk_insert", "huditest.bulkinsert_mor_10g")
step2: prepare update DataFrame with DateType, and upsert into HudiMorTable
df_update = sql("select * from huditest.bulkinsert_mor_10g_rt").withColumn("date", lit(Date.valueOf("2020-11-11")))
merge(df_update, "upsert", "huditest.bulkinsert_mor_10g")
step3: use hive-beeeline/ spark-sql query mor_rt table
use beeline/spark-sql execute statement select * from huditest.bulkinsert_mor_10g_rt where primary_key = 10000000;
then the follow error will occur:
java.lang.ClassCastExceoption: org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.hive.serde2.io.DateWritableV2
Brief change log
when hudi read log file and convert avro INT type record to writable，logicalType is not respected which lead the dateType will cast to IntWritable。so cast avro INT type to writable, logicalType must be considered
Verify this pull request
Existing UT tests
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.