Skip to content

Rolled-back follower writes can enter a later leader's replication batch #8293

Description

Describe the bug

A transaction already rolled back locally can be queued by Store::commit() while the node is a follower, survive a subsequent election, and then be included in a fresh leader's replication batch.

This was reproduced against current CCF ee34e7b2e0d3ae7f7a43a913d8ecfe35f7d74a9e using the real Store + AFT + Merkle history fixture from #8238. After an ordinary fresh commit, the Store was at 4.2 while both history and AFT had advanced to seqno 3. The old write was absent from the local map. The stale transaction also incorrectly returned SUCCESS after its local write had been rolled back.

The relevant ordering is:

  • Store::commit() lines 1001-1021 rejects a stale term only when is_primary() is true, then inserts the pending transaction.
  • Lines 1033-1055 leave an entry queued behind a hole and return SUCCESS.
  • Store::rollback() lines 687-710 returns without clearing pending_txs when the rollback target equals the current version.
  • A subsequent fresh commit drains both its own entry and the stale pending entry into the new-view batch.

To Reproduce

Use #8238 at 344e89c59adac38087267bd9189b8e7b855bb328 as a harness overlay on the current revision above, rather than checking out the PR's older base. Add the following source to commit_concurrency_test alongside its existing threaded scenarios. This uses the PR's checkpoint to pause ordinary transactions after local application/serialization but before Store::commit().

#include "commit_concurrency/threaded/fixture.h"

#define DOCTEST_CONFIG_NO_SHORT_MACRO_NAMES
#include <doctest/doctest.h>
#include <thread>

using namespace ccf::kv::test;
using namespace std::chrono_literals;

class PausedWrite
{
  Checkpoint checkpoint;
  std::jthread worker;

public:
  ccf::kv::CommitResult result = ccf::kv::CommitResult::FAIL_CONFLICT;

  PausedWrite(CommitConcurrencyFixture& fixture, size_t key)
  {
    worker = std::jthread([this, &fixture, key]() {
      auto tx = fixture.store->create_tx();
      tx.rw(fixture.table)->put(key, key);
      result = tx.commit(
        ccf::empty_claims(), checkpoint_write_set_observer(checkpoint));
    });
    checkpoint.wait_until_paused();
  }

  ~PausedWrite()
  {
    resume();
  }

  void resume()
  {
    checkpoint.release();
    if (worker.joinable())
    {
      worker.join();
    }
  }
};

DOCTEST_TEST_CASE("Stale follower pending write survives election")
{
  CommitConcurrencyFixture fixture;
  const auto baseline = fixture.commit_signature();
  PausedWrite first(fixture, 10);
  PausedWrite second(fixture, 20);
  DOCTEST_REQUIRE(
    fixture.store->current_txid().seqno == baseline.seqno + 2);

  fixture.step_down();
  first.resume();
  DOCTEST_REQUIRE(
    first.result == ccf::kv::CommitResult::FAIL_NO_REPLICATE);
  DOCTEST_REQUIRE(fixture.store->current_txid() == baseline);

  second.resume();
  DOCTEST_CHECK(
    second.result == ccf::kv::CommitResult::FAIL_NO_REPLICATE);
  DOCTEST_REQUIRE(fixture.store->current_txid() == baseline);
  DOCTEST_REQUIRE_FALSE(read_value(*fixture.store, fixture.table, 20));

  fixture.raft->periodic(200ms);
  DOCTEST_REQUIRE(fixture.raft->is_primary());
  DOCTEST_REQUIRE(fixture.store->current_txid() == baseline);

  auto fresh = fixture.store->create_tx();
  fresh.rw(fixture.table)->put(30, 30);
  DOCTEST_REQUIRE(fresh.commit() == ccf::kv::CommitResult::SUCCESS);
  const auto local = fixture.store->current_txid();
  DOCTEST_CHECK(fixture.raft->get_last_idx() == local.seqno);
  DOCTEST_CHECK(fixture.history_txid().seqno == local.seqno);
  DOCTEST_CHECK_FALSE(read_value(*fixture.store, fixture.table, 20));
}

Pinned sequence:

  1. Commit the baseline signature at 2.1.
  2. Pause two ordinary writers after their writes receive seqnos 2 and 3.
  3. Lose leadership.
  4. Resume the first writer. AFT refuses replication as a follower and rolls the Store back to the baseline, discarding both local writes.
  5. Resume the second writer while still a follower. Its old-view entry at seqno 3 is queued behind missing seqno 2, and the call returns SUCCESS.
  6. Run the normal election path with periodic(200ms). The singleton fixture wins election; the rollback target equals the Store version, so the stale queue entry survives.
  7. Commit one fresh write at 4.2. The pending batch also includes the discarded seqno 3 entry.

Observed output from the instrumented test:

baseline=2.1, stale result=SUCCESS, fresh=4.2, store=4.2,
history seqno=3, raft last_idx=3, rolled-back key present=false

replicated == local.seqno: 3 == 2 [fails]
history.seqno == local.seqno: 3 == 2 [fails]

Expected behavior

A transaction discarded by rollback must not be accepted into a later replication batch. Fresh writes after an election must leave the local Store, history, and AFT's replicated index consistent. The stale transaction should fail rather than return success for a discarded write.

Environment information

Additional context

This is distinct from the stale-view-before-local-application case fixed by #8242: both transactions here receive their versions before leadership changes. The original #8238 threaded and scheduled suites passed at this revision; the new scenario keeps the node a follower long enough for an already-discarded second commit to arrive.

The delayed writer corresponds to preemption in an ordinary frontend commit. The fixture regains leadership via periodic(), not force_become_primary(). A full multi-node end-to-end reproduction has not been run. This issue concerns concurrency correctness; no production fix is included.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions