[Not yet upstream] Serial replication can get stuck on empty ranges

[sidkhillon]

This is a long PR description, but the summary is that serial replication can get stuck if a server crashes twice without having edits in between. Basically, if the situation is:

• Write edits
• Server crashes
• Server crashes (no new edits)
• Write edits <-- these will be stuck forever

Fix serial replication stuck after consecutive RegionServer crashes

Problem

Serial replication can get permanently stuck after an RS is restarted (gracefully or via SIGKILL) multiple times without user writes between restarts. Once stuck, the affected RS's WAL reader blocks indefinitely in waitUntilCanPush(), causing head-of-line blocking for all regions sharing that WAL — including unrelated regions moved onto the stuck RS after the fact.

Mamba CDC outage

On an hbase cluster, a CDC peer was created on Jul 13 while a rolling Hadoop datanode upgrade was in progress. The upgrade uses a drain/return pattern: displacing all ~95 regions from each RS to temporary RSs, rolling WALs, stopping the RS, restarting, then returning regions. Each upgrade creates two barriers per region (one on drain, one on return).

Over Jul 14-16, the upgrade job restarted 33 RSs. Each drain/return cycle created intermediate barrier ranges with no user writes between them. WAL reader threads became stuck in SerialReplicationChecker.waitUntilCanPush() on REGION_OPEN entries for these intermediate ranges, with the log message:

Previous range for <hubspot table>/.../862704391 has not been finished yet, give up

Thread dumps showed WAL readers in TIMED_WAITING with waited counts of 65K-104K (looping for days). The stuck entries blocked all subsequent WAL entries via head-of-line blocking, cascading from 4 stuck RSs on Jul 14 to all 33 RSs by Jul 15 22:00. The only resolution was deleting and re-creating the CDC peer.

Example from a host:

barriers=[862629162, 862681720, 862703835, 862704390, 862758946, 862759989, 862760093]

Stuck range [862703835, 862704390) = 555 seqIds spanning 13 minutes (Jul 15 10:59-11:12), created by a drain/return bounce during the upgrade.

Root cause

When an RS restarts and opens a region, a new replication barrier (openSeqNum) is written to hbase:meta, and a REGION_OPEN marker is appended to the WAL. The marker's seqId is always openSeqNum + 1 because mvcc.advanceTo(openSeqNum) sets the base, then mvcc.begin() increments before the write.

If the RS restarts again before any user data is written, the next RS computes its openSeqNum from recovery, which lands at previousOpenSeqNum + 1. This creates two consecutive barriers that differ by exactly 1.

The range between these barriers (e.g., [38, 39)) contains only seqId 38, the openSeqNum itself. No WAL entry is ever written at the openSeqNum — it's set via mvcc.advanceTo(), not mvcc.begin() — so this range is empty.

Meanwhile, the REGION_OPEN marker from the previous restart has seqId openSeqNum + 1, which exactly matches the new barrier.

In SerialReplicationChecker.canPush(), Arrays.binarySearch(barriers, seqId) finds this exact match. The code does index++ to convert from 0-based to 1-based range indexing, causing isRangeFinished(barriers[index-1]) to check the empty range:

isRangeFinished(39) → pushedSeqId >= 38

No WAL entry at seqId 38 exists, so pushedSeqId never reaches 38. The check fails permanently, and waitUntilCanPush spins indefinitely.

This blocks the WAL reader thread entirely. Since SerialReplicationSourceWALReader is single-threaded and processes entries sequentially, any entries after the stuck one — including entries for unrelated regions — are head-of-line blocked.

See HBASE-29499 for further details and independent reproduction by the Apache HBase community.

Fix

In SerialReplicationChecker.canPush(): when seqId exactly matches a barrier at position index > 0 and barriers[index] - barriers[index-1] == 1, don't increment the index. This makes isRangeFinished check the range before the empty gap-1 range instead of the empty range itself.

The gap == 1 condition precisely identifies empty ranges:

• The only seqId in [X, X+1) is X, which is the openSeqNum
• openSeqNum never has a WAL entry (mvcc.advanceTo doesn't write; mvcc.begin increments past it)
• A data entry's seqId can never equal a later barrier because barrier = maxPreviousSeqId + 1 and the data entry is one of those previous seqIds, so barrier > dataSeqId

For gap > 1, the range contains at least the REGION_OPEN marker, which gets processed by the WAL reader and advances pushedSeqId — so the normal index++ path works correctly.

Why skipping the empty range is safe

The fix skips the isRangeFinished check for gap-1 ranges. This cannot cause out-of-order replication or dropped messages because:

The range is provably empty. A gap-1 range [X, X+1) contains only seqId X, which is the openSeqNum. The openSeqNum is set via mvcc.advanceTo(X) — this advances the MVCC write point without calling mvcc.begin(), so no WAL entry is written at that seqId. The first actual WAL entry is the REGION_OPEN marker at X+1 (from mvcc.begin() which increments the write point). Since the range contains zero WAL entries, there is nothing to replicate and nothing to skip.

Prior data is still checked. The fix doesn't remove the ordering check — it shifts it one range back. With the un-incremented index, isRangeFinished(barriers[index-1]) verifies that all data from the range before the empty range has been fully replicated. If that prior range still has unreplicated entries, canPush returns false and the WAL reader waits, preserving serial ordering.

Entries within the same WAL are ordered. The stuck entry (the REGION_OPEN marker at seqId == barrier) and any preceding data entries from the same RS incarnation are in the same WAL file. The SerialReplicationSourceWALReader processes WAL entries sequentially, so by the time it reaches the REGION_OPEN marker, all earlier entries from that WAL have already been read into the batch. They are shipped to the peer in order before the marker's seqId is recorded.

Cross-WAL ordering is enforced by the prior range check. Data from earlier RS incarnations lives in different WALs, processed by separate claimed-queue readers. The isRangeFinished check that the fix preserves (against the range before the empty one) ensures that all data from those earlier incarnations has been replicated before the boundary entry is allowed through. The empty gap-1 range sits between the completed prior data and the current entry — skipping its check doesn't change the fact that the prior data must finish first.

[cathturner]

overall approach lgtm!

[cathturner]

nit: looks like we don't use tableName here

[cathturner]

nit: go -> goes?

[cathturner]

✨

[hgromer]

Looks great thank you!