Receive an unexpected event instant time[SUPPORT]

[zwj0110]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

1. use flink to write hudi table

The exception continue to occur

2. checkpoint
   the checkpoint is not always failed, theck checkpoint can success

Expected behavior

Flink job can run normally

Environment Description

• Hudi version : 0.10

• Flink version : 1.13.1

• Hive version : 3.1.2-amzn-4

• Hadoop version : 3.2.1-amzn-3

• Storage (HDFS/S3/GCS..) : S3

• Running on Docker? (yes/no) : no

[danny0405]

Do you have multiple writers writing into the same table at the same time ?

[zwj0110]

Do you have multiple writers writing into the same table at the same time ?

No, Only one writer

[zwj0110]

I find the data can be writed,when the checkpoint is success,but after running some time , the data will duplicate

[danny0405]

your checkpoint size is huge, did you use the rocks-db statebackend instead ? And the 0.11.1 supports bucket index, you can try that.

[zwj0110]

your checkpoint size is huge, did you use the rocks-db statebackend instead ? And the 0.11.1 supports bucket index, you can try that.
I use the rocks-db statebackend, bucket index it only can reduce the checkpoint size ?

[danny0405]

your checkpoint size is huge, did you use the rocks-db statebackend instead ? And the 0.11.1 supports bucket index, you can try that.
I use the rocks-db statebackend, bucket index it only can reduce the checkpoint size ?

Did you open incremental checkpoint, the bucket index does not use any state-backend at all.

[zwj0110]

your checkpoint size is huge, did you use the rocks-db statebackend instead ? And the 0.11.1 supports bucket index, you can try that.
I use the rocks-db statebackend, bucket index it only can reduce the checkpoint size ?

Did you open incremental checkpoint, the bucket index does not use any state-backend at all.

No, I write a lot table to hudi, there are ok , but I don't know the reason why it's broken

[danny0405]

The exception usually happens when it receives an event that does not belong to the current checkpoint, for example, multiple writers write into the table at the same time.

[zwj0110]

The exception usually happens when it receives an event that does not belong to the current checkpoint, for example, multiple writers write into the table at the same time.
Okay, I will check it , you can close the issue