fix: prevent concurrent merge_insert duplicate rows without PK metadata

[ozzieba]

Summary

Fixes the concurrent merge_insert duplicate row bug (#4585) by always including a bloom filter for inserted rows in the transaction, regardless of whether the schema has unenforced-primary-key metadata, and making conflict detection symmetric for asymmetric bloom filter pairs.

Problem

When multiple concurrent workers call merge_insert("id") with overlapping key ranges, the second writer's commit succeeds silently even when it inserts keys that the first writer already inserted, producing duplicate rows. This happens because:

1. The bloom filter for inserted rows was only included in the transaction when the schema had lance-schema:unenforced-primary-key metadata on the merge column. Most users call merge_insert("id") without this metadata.

2. The conflict resolver only detected (Some, None) bloom filter asymmetry as a conflict. The (None, Some) case — when the committed transaction has a filter but the current doesn't — fell through silently.

Bug reproduction

A self-contained Python script reproduces the bug against stock PyPI lancedb:

```bash
uv run --with lancedb --no-project test_concurrent_merge_insert_bug.py
```

Against lancedb 0.29.2:

Test	Expected	Actual	Duplicates
5 workers, overlapping keys (0..19)	21 rows	101 rows	80
8 workers, identical data (100 rows)	101 rows	801 rows	700

Fix (2 changes)

1. `merge_insert/exec/write.rs`

Always include the bloom filter for inserted rows in the transaction, even without PK metadata.

2. `conflict_resolver.rs`

Treat `(None, Some)` bloom filter pair as a retryable conflict, symmetric with the existing `(Some, None)` handling.

Tests

• Rust regression test: 5 concurrent workers with overlapping keys, no PK metadata
• Rust unit tests: conflict detection for all 4 bloom filter pair combinations
• Rust unit tests: concurrent insert same/different keys with/without PK metadata
• Rust proptests: parameterized over dataset size, key values, PK metadata presence
• Python reproducer script: demonstrates bug against stock PyPI lancedb

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[ozzieba]

@wjones127 @jackye1995 @yanghua
Disclaimer: AI generated code

does this make sense?

[jackye1995]

Thanks for doing this! In order to enforce uniqueness of inserted rows, we need to make sure that if user runs 2 merge-inserts and they are using different keys, it should be treated as incompatible because the bloom filter is incompatible. That means we need to track what merge key we are using in the Transaction model: https://github.com/lance-format/lance/blob/main/protos/transaction.proto#L228

This is a change of the Lance specification, so I would suggest creating a dedicated PR for that change first, and it would require a quick community vote. See example vote here: #5485