[Feature]: A new design of resolving data conflicts without relying on AMS to generate TransactionId

[wangtaohz]

Description

To resolve the data conflict of Arctic KeyedTable(Mixed-format table) with concurrent batch write and stream write, we introduce a design of version for each record, in order to keep the data with a higher version when row conflicts arise.

Use case/motivation

In the case of a concurrent batch write and stream write, row conflicts are likely to arise, especially when batch write takes a long time, and the batch write will fail to commit.

We have introduced TransactionId and offset to determine the record version, but the generation of TransactionId relies on the AMS, which leads to reliability issues for data writing.

Describe the solution

The design of the record version:
Version = (TransactionId, offset)

• TransactionId: the sequence of each Transaction
• offset: the offset of record in a Transaction, to determine the record version in a transaction

Two ways to generate TransactionId:

• generate on transaction begin: commit an empty snapshot to ChangeTable, and get the snapshot sequence as TransactionId
• generate on transaction commit: get the change snapshot sequence when commit as TransactionId

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct