ADS layer malfunctioned. After restarting the job, the data statistics were incorrect

[jack1234smith]

Describe the problem you faced

Based on Flink and Hudi's real-time data warehouse, the ADS layer job has malfunctioned, while the ODS and DWD layers have normal job. Restart the ADS layer job one day after the failure. At this point, there are two types of data anomalies. One is that the new data overwrites the old data. For example, the newly calculated March monthly report data will overwrite the old March monthly report; The second type is that new tasks only retrieve data from the most recent commit and do not count the entire amount of data, resulting in new monthly report data being less than the actual value.
How did everyone solve it?

Environment Description

Hudi version : 0.14.1
Flink version : 1.17
Storage (HDFS/S3/GCS..) : HDFS
Running on Yarn? (yes/no) : yes

[danny0405]

One is that the new data overwrites the old data

Not sure if you are using the upsert operation by using the index for updating. If you are using the Flink_STATE index, you need to recover the job from the latest checkpoint. Another choice is using the hashing index by specifying the option index.type as BUCKET, whereas there is no need for state recovery.

The second type is that new tasks only retrieve data from the most recent commit and do not count the entire amount of data

Are you saying the state acc of Flink? did you recover from the latest checkpoint state?

[jack1234smith]

I sure I use the upsert operation and recover the job from the latest checkpoint.