-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SUPPORT]Sync hive lost some partitions when submit multiple commits at the same time #7570
Comments
#6662 this should be fixed by this pr |
Thanks for your reply. And could you explain the specific reason? Is it because some commits are archived so cannot be synced to hive. |
the sync logic is: check last_update_time in hive table properties, get all commits from that time, then update last_update_time,this is not working for multiple writers |
Is that means, when 20221227042855832.commit goes to sync hive, if the last_update_time in hive table properties is 20221227042906103, then the commit of 20221227042855832 will not be synced to hive. |
yes |
Thanks! |
Tips before filing an issue
Have you gone through our FAQs?
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
Issue:
Lost some partitions when sync hive
Background:
We have a data ingest pipeline, which ingest about 500 partitions per day. And the pipeline will submit multiple commits at the same time to insert different partitions. The sync hive function is enabled for each commit.
And after all of commits succeed, we found that some partitions are missing in the hive table.
The following is the analysis of log and hoodie files:
For the hoodie files, shows six of the commits. Then it was found that only 20221227042858342 & 20221227042906103 two commits were synced to hive, and the rest of the partitions did not appear in hive table.
I think the root cause is because of the mechanism of sync hive. When hudi sync hive after the commit is succeed, it will first get the latest synced commit, and then use the timestamp of this commit as a benchmark to check whether the new column and partition are added to the commit behind it, and if so, it will sync to hive.
So if a commit A is submmitted before this latest synced commit B, but succeeds after commit B, so it will not be synced hive. Because of commit A's timestamp < commit B's timestamp, it won't be detected.
Here is the log of commit 20221227042859357, we can see it get latest synced commit is 20221227042906103, which commit after 20221227042859357 itself. So the partition inserted by 20221227042859357 commit has not been detected, and the partition that needs to be synced is 0.
To Reproduce
Steps to reproduce the behavior:
1.Submit multiple commits at the same time to insert different partitions. The sync hive function is enabled for each commit.
2.The order in which all commits succeed is inconsistent with the order in which they were submitted.
3.Check whether the hive table has parititon for all inserts
Expected behavior
A clear and concise description of what you expected to happen.
Each commit can specify a synchronized partition as the currently inserted parition.
Environment Description
Hudi version :0.11.1
Spark version :3.2.1
Hive version :XXX
Hadoop version :3.3.2
Storage (HDFS/S3/GCS..) :S3
Running on Docker? (yes/no) :no
Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.
The text was updated successfully, but these errors were encountered: