[log] Follower fetch offset need truncate to leaderEndOffsetSnapshot if we found duplicated batch or out of sequence batch

[swuferhong]

Purpose

Linked issue: close #704

This pr is aims to address the issue where a DuplicateSequenceException or OutOfOrderSequenceException may occur when the follower fetches data from the leader for sync. The root cause of this problem lies in our current approach of updating the follower's HW (HighWatermark) before updating the leader's HW. This update mechanism may lead to the issues illustrated in the diagram below:

As in case 1, for example. In this scenario, when the leader, follower1, and follower2 are all offline, follower1 has written batch1 (WriterId=1, batchSequence=10), but the leader did not write this batch. When the tabletServer recover, follower2 is elected as the new leader. At this point, another writer, such as writer2 or writer3, quickly writes new batches and closes the gap with follower1. If the client then writes batch1, when follower1 pulls the updates, a DuplicateSequenceException will occur.

To fix this error, we introduce a variable named LeaderEndOffsetSnapshot to remember the endOffset snapshot when the replica become leader. If follower encounter DuplicateSequenceException, it need to truncate to leader's LeaderEndOffsetSnapshot, by this way, it can be continue to fetch data from leader.

Brief change log

Tests

API and Format

Documentation