feat: add CommitStrategy enum with Pessimistic and Optimistic modes

[Jay-ju]

Add CommitStrategy: Optimistic vs Pessimistic Conflict Resolution

Summary

This PR introduces a CommitStrategy enum that allows users to choose between Pessimistic (default, current behavior) and Optimistic commit strategies for concurrent write conflict resolution.

Background

Currently, Lance always performs a "rebase" before every commit attempt — loading new transactions since the last read and running TransactionRebase::try_new() to reconcile conflicts. This is safe but adds significant I/O overhead on every attempt, even when there is no conflict. Under high concurrency, this rebase cost becomes a bottleneck.

Changes

Core Logic (rust/lance/src/io/commit.rs)

The commit loop now conditionally skips rebase based on the configured strategy:

• Pessimistic (default): Always rebase before commit. This is the existing behavior — safe for all scenarios but incurs I/O on every attempt.
• Optimistic: Attempt to commit first without rebase. Only rebase on conflict (i.e., when backoff.attempt() > 0). This skips the rebase I/O on the fast path (no conflict), and even when a conflict occurs, the cost of a failed commit attempt is typically lower than the rebase I/O overhead.

let needs_rebase = !strict_overwrite
    && match commit_config.commit_strategy {
        CommitStrategy::Pessimistic => true,
        CommitStrategy::Optimistic => backoff.attempt() > 0,
    };

New Types (rust/lance-table/src/io/commit.rs)

• CommitStrategy enum with Pessimistic (default) and Optimistic variants
• CommitConfig.commit_strategy field

Public API (rust/lance/src/dataset/write/commit.rs)

• CommitBuilder::with_commit_strategy(strategy) — allows users to configure the commit strategy

Benchmark (python/python/ci_benchmarks/benchmarks/test_concurrent_write.py)

Added a pytest-benchmark based concurrent write benchmark that measures throughput and latency under different concurrency levels and operation mixes (append, delete, update).

---

Benchmark Test Report

How to Reproduce

# 1. Build with Release mode
cd python && maturin develop --release

# 2. Run with Pessimistic strategy (default)
pytest python/ci_benchmarks/benchmarks/test_concurrent_write.py \
    --benchmark-only --benchmark-json pessimistic.json

# 3. Run with Optimistic strategy
LANCE_COMMIT_STRATEGY=optimistic \
    pytest python/ci_benchmarks/benchmarks/test_concurrent_write.py \
    --benchmark-only --benchmark-json optimistic.json

# 4. (Optional) Run against S3/TOS remote storage
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export AWS_ENDPOINT=https://your-endpoint
export AWS_REGION=your-region
export LANCE_BENCH_DATASET_URI=s3://bucket/path
pytest python/ci_benchmarks/benchmarks/test_concurrent_write.py \
    --benchmark-only

Test Environment

• Build: maturin develop --release (Release mode)
• Storage: TOS (S3-compatible, remote object store)
• Concurrency: 20 appenders + 10 deleters + 10 updaters (mixed), 30 appenders + 15 deleters + 15 updaters (high concurrency)

Results — Mixed Workload (20 appenders, 10 deleters, 10 updaters)

Metric	Pessimistic	Optimistic	Delta
Total time	27.1s	24.6s	-9.2%
Throughput	2.6 ops/s	2.9 ops/s	+12.5%
Delete avg latency	1.34s	0.94s	-29.9%
Update avg latency	1.22s	0.97s	-20.5%
Append avg latency	0.68s	0.63s	-7.4%

Results — High Concurrency (30 appenders, 15 deleters, 15 updaters)

Metric	Pessimistic	Optimistic	Delta
Total time	42.3s	38.1s	-9.9%
Throughput	2.1 ops/s	2.4 ops/s	+14.3%
Delete avg latency	2.01s	1.38s	-31.3%
Update avg latency	1.85s	1.42s	-23.2%
Append avg latency	0.95s	0.87s	-8.4%

Key Findings

1. Optimistic strategy consistently outperforms Pessimistic across all operation types and concurrency levels.
2. Delete and Update benefit the most (~20-30% latency reduction) because these operations trigger TransactionRebase::try_new() which loads initial_fragments — an expensive I/O operation that Optimistic skips on the fast path.
3. Append also benefits (~7-8% improvement) from skipping the load_and_sort_new_transactions call.
4. Higher concurrency amplifies the gain — the Optimistic advantage grows as contention increases.

---

Files Changed

File	Change
rust/lance-table/src/io/commit.rs	Add CommitStrategy enum and commit_strategy field in CommitConfig
rust/lance/src/io/commit.rs	Conditional rebase logic based on CommitStrategy
rust/lance/src/dataset/write/commit.rs	Add CommitBuilder::with_commit_strategy() API
python/python/ci_benchmarks/benchmarks/test_concurrent_write.py	Concurrent write benchmark using pytest-benchmark

Backward Compatibility

• Fully backward compatible: Default strategy is Pessimistic, which preserves the existing behavior.
• No changes to public API signatures; only an additive new method with_commit_strategy().

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 45.45455% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/write/commit.rs	0.00%	4 Missing ⚠️
rust/lance/src/io/commit.rs	66.66%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[wjones127]

This is safe but adds significant I/O overhead on every attempt, even when there is no conflict. Under high concurrency, this rebase cost becomes a bottleneck.

This is a helpful finding. I originally wrote the initial behavior here: #3117

Back then, I would have thought the pessimistic strategy would be better for high contention, at the cost of being slower when there is low contention. But looks like empirically I was wrong. Thanks for running this benchmark.

TBH, I'm not sure we need to add a setting for this. I think it would be totally acceptable just change the default to optimistic, especially given "Optimistic strategy consistently outperforms Pessimistic across all operation types and concurrency levels." If the results were mixed, I could see the case for having a setting. But if it's better in all cases, I think it's fine to change the behavior. Plus this isn't a behavior that users rely upon, it's most of an optimization. So it's not a breaking change in that sense.

What do you think of just making Optimistic the default?

[Jay-ju]

This is safe but adds significant I/O overhead on every attempt, even when there is no conflict. Under high concurrency, this rebase cost becomes a bottleneck.

This is a helpful finding. I originally wrote the initial behavior here: #3117

Back then, I would have thought the pessimistic strategy would be better for high contention, at the cost of being slower when there is low contention. But looks like empirically I was wrong. Thanks for running this benchmark.

TBH, I'm not sure we need to add a setting for this. I think it would be totally acceptable just change the default to optimistic, especially given "Optimistic strategy consistently outperforms Pessimistic across all operation types and concurrency levels." If the results were mixed, I could see the case for having a setting. But if it's better in all cases, I think it's fine to change the behavior. Plus this isn't a behavior that users rely upon, it's most of an optimization. So it's not a breaking change in that sense.

What do you think of just making Optimistic the default?

@wjones127 Thanks for the context! I agree — since Optimistic outperforms across all scenarios, making it the default is the right call. I have updated the PR already.

For context, the motivation came from LLM agent workflows where LanceDB serves as the memory module: each conversation turn triggers an individual write, creating high write concurrency that made the rebase overhead very visible.

One question while on this topic — under extreme write concurrency (e.g. many agents writing simultaneously), even the optimistic strategy still faces repeated conflict retries. I've been thinking about whether a WAL/buffer layer on top of Lance could help absorb writes and batch-flush them, similar to LSM-tree approaches. Do you have any thoughts on whether this is a direction worth exploring, or if there are other approaches you'd recommend for high-concurrency scenarios?

[wjones127]

One question while on this topic — under extreme write concurrency (e.g. many agents writing simultaneously), even the optimistic strategy still faces repeated conflict retries. I've been thinking about whether a WAL/buffer layer on top of Lance could help absorb writes and batch-flush them, similar to LSM-tree approaches. Do you have any thoughts on whether this is a direction worth exploring, or if there are other approaches you'd recommend for high-concurrency scenarios?

Yeah, that's essentially the work that's going on in #3985

[wjones127]

since Optimistic outperforms across all scenarios, making it the default is the right call. I have updated the PR already.

Actually, what do you think about going a step further, and just eliminating the option? Just change the behavior directly to use optimistic always.

[wjones127]

issue(blocking): this seems like a drive-by bug fix. Could we save this for a different PR and make sure we have a issue describing it?

[wjones127]

issue(blocking): this also seems like a drive-by change that could use it's own issue.