Hoodie 0.4.7: Error upserting bucketType UPDATE for partition #, No value present #764
Assignees
Comments
jackwang2
changed the title
|
Looks similar to https://issues.apache.org/jira/browse/HUDI-116 , although 0.4.7 should have the fix already.. Seems like a tagging issue again? are you able to trim the data down and give me a reproducible case? And for context, you upgraded to 0.4.7 and it started failing right away? were things working fine in prior version (can you share that?) |
|
@vinothchandar Thanks for looking this. Yes, seems the version of 0.4.7 is not compatible with 0.4.6, and it throws exception like below, but I use the new version 0.4.7 and write data to a new location to skip the |
|
@jackwang2 there should not be any compatibility issues between 0.4.6 and 0.4.7.
|
|
I am facing similar issue while creating the MOR tables.Please take a look. ERROR Log : |
|
@jackwang2 @amaranathv there are 3 issues here with different stack traces. @jackwang2 first issue you reported . is that gone? or you waiting for 0.4.7 to be fixed with issue below to try that? @jackwang2 second issue (probably same as what you mentioned on slack?) . are you able to reproduce it? https://github.com/apache/incubator-hudi/blob/hoodie-0.4.7/hoodie-client/src/main/java/com/uber/hoodie/io/HoodieCommitArchiveLog.java#L279 seems its related to archiving CLEAN action. The schema for this has not changed in years. it looks like some corruption. are you able to print out @amaranathv your stack trace is different.. can you print out partitionPath below? It seems like its null .. Please check if your input has valid non-null partition paths. |
|
@vinothchandar for the issue |
|
I am creating table without partition column .
I can try to rerun the process again and check.
Will delta streamer support MOR without partition column?
I have tested the copy on write without partition column.
Now I am testing Merge on Read without partition column.
That is when I got this issue.
…Sent from my iPhone
On Jul 10, 2019, at 8:29 PM, jackwang2 ***@***.***> wrote:
@vinothchandar for the issue java.util.NoSuchElementException: No value present, It was not reproduced after I changing to another column for partition. Anyway, I would try to print the info like you suggested and update you.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
I am getting same error. |
|
@amaranathv just to confirm you are using |
|
yes |
|
DataSourceWriteOptions.KEYGENERATOR_CLASS_OPT_KEY-> "com.uber.hoodie.NonpartitionedKeyGenerator" |
|
@amaranathv again the issue from this line |
|
With PR 775 this issue seems to have been fixed. I was able to reproduce this error before the fix. After applying PR 775 could not reproduce it anymore. @amaranathv can you test this PR for empty path exception? |
@jackwang2 : Are you still seeing this issue ? |
|
I am still working on performance side of the copy of on write.Will do the testing again after the performance test complete. |
|
It looks like the "Not an Avro data file" exception is thrown when there is a 0 byte stream read into the datafilereader as can be seen here : https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileReader.java#L55 and here : https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileConstants.java#L29 From the stack trace (by tracing the line numbers), it looks like the CLEAN file is failing to be archived. I looked at the clean logic and we do create clean files even when we don't have anything to clean but that does not result in a 0 bytes file, it still has some valid avro data. Although we need to fix not creating a clean file when there is nothing to clean, this still doesn't result into the error. I'm wondering if this has anything to do with any sort of race condition leading to archiving running when clean is a 0 sized file. @jackwang2 How are you running the cleaner and the archival process ? Are you explicitly doing anything there ? |
|
@n3nash No, I didn't. The main logic is for just global deduplication, and
code is pasted as below:
df.dropDuplicates(recordKey)
.write
.format("com.uber.hoodie")
.mode(SaveMode.Append)
.option(HoodieWriteConfig.TABLE_NAME, tableName)
.option(HoodieIndexConfig.INDEX_TYPE_PROP,
HoodieIndex.IndexType.GLOBAL_BLOOM.name)
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, recordKey)
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY,
partitionCol)
.option(DataSourceWriteOptions.OPERATION_OPT_KEY,
DataSourceWriteOptions.INSERT_OPERATION_OPT_VAL)
.option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY, storageType)
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, preCombineCol)
.option("hoodie.consistency.check.enabled", "true")
.option("hoodie.parquet.small.file.limit", 1024 * 1024 * 128)
.save(tgtFilePath)
Thanks,
Jack
…On Thu, Aug 1, 2019 at 9:01 AM n3nash ***@***.***> wrote:
It looks like the "Not an Avro data file" exception is thrown when there
is a 0 byte stream read into the datafilereader as can be seen here :
https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileReader.java#L55
and here :
https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DataFileConstants.java#L29
From the stack trace (by tracing the line numbers), it looks like the
CLEAN file is failing to be archived. I looked at the clean logic and we do
create clean files even when we don't have anything to clean but that does
not result in a 0 bytes file, it still has some valid avro data. I'm
wondering if this has anything to do with any sort of race condition
leading to archiving running when clean is a 0 sized file.
@jackwang2 <https://github.com/jackwang2> How are you running the cleaner
and the archival process ? Are you explicitly doing anything there ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#764>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGJTUULNELHUUHGR5OAMMBDQCIYW5ANCNFSM4H3Z6GBQ>
.
--
[image: vshapesaqua11553186012.gif] <https://vungle.com/> *Jianbin Wang*
Sr. Engineer II, Data
+86 18633600964
[image: in1552694272.png] <https://www.linkedin.com/company/vungle> [image:
fb1552694203.png] <https://facebook.com/vungle> [image:
tw1552694330.png] <https://twitter.com/vungle> [image:
ig1552694392.png] <https://www.instagram.com/vungle>
Units 3801, 3804, 38F, C Block, Beijing Yintai Center, Beijing, China
|
|
@jackwang2 biggest issue here is that we could not reproduce the error ourselves.. are you able to provide some reproducible case? is it easy to repro? |
|
I can confirm that I have been hit by this error. The occurrence of this error is in 2 tables that do not have any similarity in the pattern of data ingestion or sizes of the data they hold. As @n3nash has mentioned there are clean file of size 0 bytes. Deleting the file manually clears the problem to the table (albeit with the first attempt failing and things working from the 2nd attempt onwards). I have tried to reproduce the issue without any luck so far. I still continue to pursue to replicate the issue. If anyone else especially @jackwang2 knows what caused this and if he has fixed it in his pipeline, I would be very grateful. Any insights are hugely welcome. The stack trace is as below:
|
|
@smdahmed : Looked at the code to see how this can happen. Not clear how this can happen. Assuming you are using S3, Have you tried setting the consistency guard ( https://hudi.apache.org/configurations.html#withConsistencyCheckEnabled) ? |
|
Balaji - sincere apologies for not getting back earlier. The fact is that this happened only once and affected only 2 tables. I have reset the tables and since then, this issue has never resurfaced. I did see S3 logs etc to see if there was any abnormal activity at the platform infra level but there is nothing that we find. I shall wait to see and in the next deployment, I shall push the consistency guard option that you recommend. Thanks for all the help. |
|
Closing due to inactivity.. |
|
experiencing exact same issue from using Hudi + Glue on EMR.. |
|
Here's my config |
|
@mingujotemp : This is a old ticket possibly using different version of hudi. Can you kindly open new ticket with hudi version, symptom, steps to repro after looking at other open tickets. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I use the AWS S3 as storage, and the piece of code likes below
Highly appreciate it if you show any ideas?
The text was updated successfully, but these errors were encountered: