-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SUPPORT] hoodie commit time format change #6907
Comments
@KnightChess My issue is not with table upgrade but timestamp format. Is there a property to configure _hoodie_commit_type timestamp format? |
@sknukala as @KnightChess mentioned you don't have to worry about the timestamp format as the new millisecond instant time is designed to be backward compatible. You don’t have to do any special handling. Do you have any specific reason for enforcing timestamp format? |
@yihua A lot of downstreams use this column to incrementally pull data and change in format impacts all of them. If the format can be controlled, it will be easy. Also, as we upgraded table, old data is in legacy format while new loads have ms leading to inconsistency. |
@sknukala if you are using Hudi incremental query, the instant timestamp format (second vs millisecond granularity) should not matter, because internally Hudi treats the instant time as a String and uses the predicate based on the instant time with String comparison for filtering records, so millisecond-level instant time is still backward compatible with second-level instant time. Could you clarify how the incremental pull is impacted? |
@sknukala : let us know if you see any inconsistencies. and may be provide a reproducible script if feasible. |
@yihua @nsivabalan : As you pointed, hudi is handling changes as it uses string comparison. However, this change is affecting places where _hudi_commit_column is casted to timestamp format. Ex: scripts loading incremental data to database using _hoodie_commit_time |
hmmm, I see. sorry, I can't think of easier route here. Also, we usually don't have any backporting fixes. i.e. even if we solve the issue, we can't port it back to 0.10.1 and other versions > 0.10.1. @xushiyan @yihua : can you folks think of any other approach here. |
@nsivabalan : Adding a config to current hudi version 0.12 and future versions would help. Please let me know |
@sknukala : is it not possible to fix the consumers to detech whether its sec or ms granularity before casting. bcoz, then you can never upgrade hudi to ms granularity and are essentially stuck. |
@nsivabalan : the granularity of seconds just works fine for us. Having a config can help users control the timestamp format or if need be to adapt ms, we can plan this activity. Currently, having a default to ms is blocking migration to hudi 12(as we need to update all consumers) and we are loosing all performance improvements implemented in recent hudi versions. with the config, upgrade would be seamless. |
I discussed w/ few other hudi experts. We feel this has to be addressed at app layer where commit times are casted to timestamp. we don't have plans to support sec level granularity. sorry about that. |
Describe the problem you faced
We are trying to migrate hudi table from 0.8 to 0.12 and noticed that _hoodie_commit_time format has changed to include milliseconds.
Below is example of sample data in different version:
hudi 0.8: 20220920044733
hudi 0.12: 20220923141615400
Is there a property to configure timestamp format? We need this to ensure backward compatibility and also reduce changes to data while migrating
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description
Hudi version : 0.12
Spark version : 3.1
EMR version : EMR 6.3
Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : no
The text was updated successfully, but these errors were encountered: