Question about CommitLog::recoverAbnormally

[BeiKeJieDeLiuLangMao]

As i know, logic of recoverAbnormally is as follow:

1. Find first valid CommitLog file from last one
2. Check all message in last valid CommitLog file
3. For these valid messages in last valid CommitLog file, rebuild ConsumeQueue and IndexFile
4. Fix all kinds of pointers, and clear ConsumeQueue redundant data

Now consider follow situation:

1. A few messages been written to CommitLog file L2(last 2), and these message are last messages of this CommitLog file
2. Create a new CommitLog file L1(last 1),then write a message to L1
3. At this time, L2's last few messages and L1's first message are not been written to ConsumeQueue and IndexFile yet, then RocketMQ crash
4. When RocketMQ restart, it will invoke recoverAbnormally, and only rebuild L1's message, L2's last few messages are lost.

If my analysis is right, i think follow solution could resolve it:

1. Find first valid CommitLog file from last one
2. Check all message in last valid CommitLog file
3. Fix all kinds of pointers, and clear ConsumeQueue redundant data
4. In step3, we could get valid ConsumeQueue max physics offset, besides we could also get IndexFile's max physics offset from file header
5. In the light of ConsumeQueue max physics offset and IndexFile max physics offset, we could rebuild lost content respectively

This solution not only could resolve data lost issue what i described before, but also could reduce duplicate data of ConsumeQueue and IndexFile :).

[Gump518]

Theoretically, I agree that this is the case.Doesn't everybody have that question?Strangely enough, 19 years of asking questions and nobody paid attention

[github-actions]

This issue is stale because it has been open for 365 days with no activity. It will be closed in 3 days if no further activity occurs.

[github-actions]

This issue was closed because it has been inactive for 3 days since being marked as stale.