Describe the bug
A transaction already rolled back locally can be queued by Store::commit() while the node is a follower, survive a subsequent election, and then be included in a fresh leader's replication batch.
This was reproduced against current CCF ee34e7b2e0d3ae7f7a43a913d8ecfe35f7d74a9e using the real Store + AFT + Merkle history fixture from #8238. After an ordinary fresh commit, the Store was at 4.2 while both history and AFT had advanced to seqno 3. The old write was absent from the local map. The stale transaction also incorrectly returned SUCCESS after its local write had been rolled back.
The relevant ordering is:
Store::commit() lines 1001-1021 rejects a stale term only when is_primary() is true, then inserts the pending transaction.
- Lines 1033-1055 leave an entry queued behind a hole and return
SUCCESS.
Store::rollback() lines 687-710 returns without clearing pending_txs when the rollback target equals the current version.
- A subsequent fresh commit drains both its own entry and the stale pending entry into the new-view batch.
To Reproduce
Use #8238 at 344e89c59adac38087267bd9189b8e7b855bb328 as a harness overlay on the current revision above, rather than checking out the PR's older base. Add the following source to commit_concurrency_test alongside its existing threaded scenarios. This uses the PR's checkpoint to pause ordinary transactions after local application/serialization but before Store::commit().
#include "commit_concurrency/threaded/fixture.h"
#define DOCTEST_CONFIG_NO_SHORT_MACRO_NAMES
#include <doctest/doctest.h>
#include <thread>
using namespace ccf::kv::test;
using namespace std::chrono_literals;
class PausedWrite
{
Checkpoint checkpoint;
std::jthread worker;
public:
ccf::kv::CommitResult result = ccf::kv::CommitResult::FAIL_CONFLICT;
PausedWrite(CommitConcurrencyFixture& fixture, size_t key)
{
worker = std::jthread([this, &fixture, key]() {
auto tx = fixture.store->create_tx();
tx.rw(fixture.table)->put(key, key);
result = tx.commit(
ccf::empty_claims(), checkpoint_write_set_observer(checkpoint));
});
checkpoint.wait_until_paused();
}
~PausedWrite()
{
resume();
}
void resume()
{
checkpoint.release();
if (worker.joinable())
{
worker.join();
}
}
};
DOCTEST_TEST_CASE("Stale follower pending write survives election")
{
CommitConcurrencyFixture fixture;
const auto baseline = fixture.commit_signature();
PausedWrite first(fixture, 10);
PausedWrite second(fixture, 20);
DOCTEST_REQUIRE(
fixture.store->current_txid().seqno == baseline.seqno + 2);
fixture.step_down();
first.resume();
DOCTEST_REQUIRE(
first.result == ccf::kv::CommitResult::FAIL_NO_REPLICATE);
DOCTEST_REQUIRE(fixture.store->current_txid() == baseline);
second.resume();
DOCTEST_CHECK(
second.result == ccf::kv::CommitResult::FAIL_NO_REPLICATE);
DOCTEST_REQUIRE(fixture.store->current_txid() == baseline);
DOCTEST_REQUIRE_FALSE(read_value(*fixture.store, fixture.table, 20));
fixture.raft->periodic(200ms);
DOCTEST_REQUIRE(fixture.raft->is_primary());
DOCTEST_REQUIRE(fixture.store->current_txid() == baseline);
auto fresh = fixture.store->create_tx();
fresh.rw(fixture.table)->put(30, 30);
DOCTEST_REQUIRE(fresh.commit() == ccf::kv::CommitResult::SUCCESS);
const auto local = fixture.store->current_txid();
DOCTEST_CHECK(fixture.raft->get_last_idx() == local.seqno);
DOCTEST_CHECK(fixture.history_txid().seqno == local.seqno);
DOCTEST_CHECK_FALSE(read_value(*fixture.store, fixture.table, 20));
}
Pinned sequence:
- Commit the baseline signature at
2.1.
- Pause two ordinary writers after their writes receive seqnos 2 and 3.
- Lose leadership.
- Resume the first writer. AFT refuses replication as a follower and rolls the Store back to the baseline, discarding both local writes.
- Resume the second writer while still a follower. Its old-view entry at seqno 3 is queued behind missing seqno 2, and the call returns
SUCCESS.
- Run the normal election path with
periodic(200ms). The singleton fixture wins election; the rollback target equals the Store version, so the stale queue entry survives.
- Commit one fresh write at
4.2. The pending batch also includes the discarded seqno 3 entry.
Observed output from the instrumented test:
baseline=2.1, stale result=SUCCESS, fresh=4.2, store=4.2,
history seqno=3, raft last_idx=3, rolled-back key present=false
replicated == local.seqno: 3 == 2 [fails]
history.seqno == local.seqno: 3 == 2 [fails]
Expected behavior
A transaction discarded by rollback must not be accepted into a later replication batch. Fresh writes after an election must leave the local Store, history, and AFT's replicated index consistent. The stale transaction should fail rather than return success for a discarded write.
Environment information
Additional context
This is distinct from the stale-view-before-local-application case fixed by #8242: both transactions here receive their versions before leadership changes. The original #8238 threaded and scheduled suites passed at this revision; the new scenario keeps the node a follower long enough for an already-discarded second commit to arrive.
The delayed writer corresponds to preemption in an ordinary frontend commit. The fixture regains leadership via periodic(), not force_become_primary(). A full multi-node end-to-end reproduction has not been run. This issue concerns concurrency correctness; no production fix is included.
Describe the bug
A transaction already rolled back locally can be queued by
Store::commit()while the node is a follower, survive a subsequent election, and then be included in a fresh leader's replication batch.This was reproduced against current CCF
ee34e7b2e0d3ae7f7a43a913d8ecfe35f7d74a9eusing the real Store + AFT + Merkle history fixture from #8238. After an ordinary fresh commit, the Store was at4.2while both history and AFT had advanced to seqno3. The old write was absent from the local map. The stale transaction also incorrectly returnedSUCCESSafter its local write had been rolled back.The relevant ordering is:
Store::commit()lines 1001-1021 rejects a stale term only whenis_primary()is true, then inserts the pending transaction.SUCCESS.Store::rollback()lines 687-710 returns without clearingpending_txswhen the rollback target equals the current version.To Reproduce
Use #8238 at
344e89c59adac38087267bd9189b8e7b855bb328as a harness overlay on the current revision above, rather than checking out the PR's older base. Add the following source tocommit_concurrency_testalongside its existing threaded scenarios. This uses the PR's checkpoint to pause ordinary transactions after local application/serialization but beforeStore::commit().Pinned sequence:
2.1.SUCCESS.periodic(200ms). The singleton fixture wins election; the rollback target equals the Store version, so the stale queue entry survives.4.2. The pending batch also includes the discarded seqno 3 entry.Observed output from the instrumented test:
Expected behavior
A transaction discarded by rollback must not be accepted into a later replication batch. Fresh writes after an election must leave the local Store, history, and AFT's replicated index consistent. The stale transaction should fail rather than return success for a discarded write.
Environment information
ee34e7b2e0d3ae7f7a43a913d8ecfe35f7d74a9e(ccf-7.0.13-38-gee34e7b2e).344e89c59adac38087267bd9189b8e7b855bb328.USE_SNMALLOC=OFF.Additional context
This is distinct from the stale-view-before-local-application case fixed by #8242: both transactions here receive their versions before leadership changes. The original #8238 threaded and scheduled suites passed at this revision; the new scenario keeps the node a follower long enough for an already-discarded second commit to arrive.
The delayed writer corresponds to preemption in an ordinary frontend commit. The fixture regains leadership via
periodic(), notforce_become_primary(). A full multi-node end-to-end reproduction has not been run. This issue concerns concurrency correctness; no production fix is included.