[DocDB] Retryable requests file can be ahead of committed_op_id during tablet bootstrap

[Huqicheng]

Jira Link: DB-7402

Description

On a leader, there's a scenario where the retryable requests file has some op ids that are greater than the committed_op_id stored in the last write batch in WAL:

Majority replicated N write ops (1.1 ~ 1.N+1). And those writes are not applied and committed on leader
Kill both followers
Before leader lease is expired, the leader receives a new write N+1 (1.N+2)
Leader starts to apply pending writes and can apply up to the Nth write and leave N+1 as pending. The retryable requests file also has marked the first Nth writes as replicated and flush it to disk.
Kill the leader and restart it.
The last committed id read from WALs is still 1.1 and treat all write ops as orphaned pending replicates. Since the retryable requests file has stored the first Nth writes, when registering the orphaned pending replicates, it will hit duplicate write error.

If an op id is covered by the retryable requests file, it must be committed. So we can do the following fix:

last_committed_op_id = std::max(
          std::max(last_committed_op_id, read_result.committed_op_id),
          last_op_id_in_retryable_requests);

cc @bogdan

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[rthallamko3]

This can result in bootstrap failure on the follower. Mitigation is to delete the retryable requests file.