[ARCTIC-994] Introduce design of `TransactionId` generation to resolving data conflicts #1010

wangtaohz · 2023-01-12T14:06:24Z

Why are the changes needed?

fix #994

Brief change log

generate TransactionId from change snapshot sequence
spark operation begins a transaction with new TransactionId
flink operation does not begin a transaction, and uses snapshot sequence as TransactionId when committing. The TxId in fileName is set to be 0, and the correct TransactionId should get from iceberg metadata in this case
flink incremental pull using TableEntriesScan instead of the iceberg TableScan, to get the sequence number from iceberg metadata when TxId in fileName is 0
import a UUID into the file name to distinguish files with the same TxId = 0, in CommonOutputFileFactory and AdaptHiveOutputFileFactory
Introduce com.netease.arctic.io.FileNameHandle to resolve file name.
Introduce com.netease.arctic.data.file.ContentFileWithSequence to wrap ContentFile.
Make com.netease.arctic.scan.BaseChangeTableIncrementalScan can return SequenceNumber.
make the using change snapshot sequence as TransactionId be compatible with old TransactionId (make sure snapshot sequence >= old transactionId) when ams start, in UpdateTool

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduces a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

wangtaohz · 2023-01-13T05:42:11Z

When TransactionId is generated on commit, the TransactionId in the file name is 0, so we have to import a UUID into the file name to distinguish files.

Adapt new Transaction model

core/src/main/java/com/netease/arctic/data/DefaultKeyedFile.java

Adapt new Transaction model

YesOrNo828

@wangtaohz Thanks for your contribution. I left some comments.

@zstraw Do you have a time to take a look?

flink/v1.12/flink/src/main/java/com/netease/arctic/flink/write/ArcticFileWriter.java

...v1.12/flink/src/main/java/com/netease/arctic/flink/write/ArcticRowDataTaskWriterFactory.java

core/src/main/java/com/netease/arctic/io/FileNameHandle.java

...v1.12/flink/src/main/java/com/netease/arctic/flink/write/ArcticRowDataTaskWriterFactory.java

flink/v1.12/flink/src/main/java/com/netease/arctic/flink/write/ArcticFileWriter.java

2.fix unit test of tracer in core

wangtaohz · 2023-02-02T05:44:26Z

This PR also fix #1045

baiyangtx · 2023-02-02T07:43:42Z

spark/v2.3/spark/src/test/java/com/netease/arctic/spark/SparkTestContext.java

    try (TaskWriter<Record> writer = GenericTaskWriters.builderFor(table)
-        .withTransactionId(txId)
        .withChangeAction(action)
        .buildChangeWriter()) {


Why remove transactionId here?

Here, it writes some data into ChangeStore, there is no need to begin a transaction here, so I remove the transactionId.

2.support generate hive sub dir without txId

YesOrNo828

LGTM

…ing data conflicts (#1010) * generate transaction from change table sequence number * remove gap * add operationId into OutputFileFactory * remove allocateTransactionId from TablePropertyUtil for spark * remove useless import * flink not begin transaction * remove begin transaction when write into change store * add UpdateTool to be compatible with old transactionId * fix unit test * Adapt new Transaction model * Adapt new Transaction model * Adapt new Transaction model * Adapt new Transaction model * Adapt new Transaction model * Adapt new Transaction model * fix checkstyle * fix compile error * 1.fix flink incremental pull in 1.12 2.fix unit test in 1.12 * 1.fix flink incremental pull in 1.14 2.fix unit test in 1.14 * 1.fix flink incremental pull in 1.15 2.fix unit test in 1.15 * remove useless import * Adapt new Transaction model * Adapt new Transaction model * fix ams unit test * fix hive commit target files serialization * fix compile error * add summary to empty snapshot for Transaction begin and add some comment * Adapt new Transaction model * fix parse file for tracer and refactor some methods of FileNameHandle" * remove useless code in flink * 1. add more comment for FileNameHandle 2.fix unit test of tracer in core * add comment * remove useless import * refactor FileNameHandle to FileNameGenerator * add unit test for #1045 * 1.spark remove txId for operations on UnkeyedTable 2.support generate hive sub dir without txId * remove FileName transaction > 0 check --------- Co-authored-by: shidayang <530847445@qq.com>

…ing data conflicts (apache#1010) * generate transaction from change table sequence number * remove gap * add operationId into OutputFileFactory * remove allocateTransactionId from TablePropertyUtil for spark * remove useless import * flink not begin transaction * remove begin transaction when write into change store * add UpdateTool to be compatible with old transactionId * fix unit test * Adapt new Transaction model * Adapt new Transaction model * Adapt new Transaction model * Adapt new Transaction model * Adapt new Transaction model * Adapt new Transaction model * fix checkstyle * fix compile error * 1.fix flink incremental pull in 1.12 2.fix unit test in 1.12 * 1.fix flink incremental pull in 1.14 2.fix unit test in 1.14 * 1.fix flink incremental pull in 1.15 2.fix unit test in 1.15 * remove useless import * Adapt new Transaction model * Adapt new Transaction model * fix ams unit test * fix hive commit target files serialization * fix compile error * add summary to empty snapshot for Transaction begin and add some comment * Adapt new Transaction model * fix parse file for tracer and refactor some methods of FileNameHandle" * remove useless code in flink * 1. add more comment for FileNameHandle 2.fix unit test of tracer in core * add comment * remove useless import * refactor FileNameHandle to FileNameGenerator * add unit test for apache#1045 * 1.spark remove txId for operations on UnkeyedTable 2.support generate hive sub dir without txId * remove FileName transaction > 0 check --------- Co-authored-by: shidayang <530847445@qq.com>

generate transaction from change table sequence number

40800f9

github-actions bot added the module:core Core module label Jan 12, 2023

wangtaohz added 3 commits January 13, 2023 19:58

remove gap

030cd4e

Merge branch 'master' into fix-994-1

d4bbfdb

add operationId into OutputFileFactory

29d1515

github-actions bot added the module:mixed-hive Hive moduel for Mixed Format label Jan 16, 2023

remove allocateTransactionId from TablePropertyUtil for spark

f00063b

github-actions bot added the module:mixed-spark Spark module for Mixed Format label Jan 16, 2023

wangtaohz added 3 commits January 16, 2023 15:58

remove useless import

0741f8a

flink not begin transaction

64a736a

remove begin transaction when write into change store

4b77efc

github-actions bot added module:ams-server Ams server module module:ams-dashboard Ams dashboard module module:mixed-flink Flink moduel for Mixed Format module:mixed-trino trino module for Mixed Format labels Jan 16, 2023

wangtaohz and others added 5 commits January 16, 2023 20:59

add UpdateTool to be compatible with old transactionId

d88f313

fix unit test

b0a14d5

Adapt new Transaction model

72aa736

Merge pull request #3 from shidayang/fix-994-1

b7883c7

Adapt new Transaction model

Merge branch 'fix-994-1' of github.com:wangtaohz/arctic into fix-994-1

7c02fca

wangtaohz commented Jan 17, 2023

View reviewed changes

core/src/main/java/com/netease/arctic/data/DefaultKeyedFile.java Outdated Show resolved Hide resolved

wangtaohz and others added 8 commits January 18, 2023 10:02

Merge branch 'master' of github.com:NetEase/arctic into fix-994-1

8947395

Adapt new Transaction model

cb0bf56

Adapt new Transaction model

eafd180

Adapt new Transaction model

4a24f21

Adapt new Transaction model

2f95ac9

Merge pull request #4 from shidayang/fix-994-1

8405d54

Adapt new Transaction model

Adapt new Transaction model

aef7d7c

Merge pull request #5 from shidayang/fix-994-1

5bb5fb5

Adapt new Transaction model

YesOrNo828 reviewed Feb 1, 2023

View reviewed changes

zstraw reviewed Feb 1, 2023

View reviewed changes

...v1.12/flink/src/main/java/com/netease/arctic/flink/write/ArcticRowDataTaskWriterFactory.java Outdated Show resolved Hide resolved

zstraw reviewed Feb 1, 2023

View reviewed changes

flink/v1.12/flink/src/main/java/com/netease/arctic/flink/write/ArcticFileWriter.java Outdated Show resolved Hide resolved

wangtaohz and others added 7 commits February 1, 2023 14:04

add summary to empty snapshot for Transaction begin and add some comment

bf97319

Adapt new Transaction model

0e46c94

Merge remote-tracking branch 'xiaowangtao/fix-994-1' into fix-994-1

a8111f3

fix parse file for tracer and refactor some methods of FileNameHandle"

cadd6d6

remove useless code in flink

3dd8ea9

1. add more comment for FileNameHandle

ebdbe90

2.fix unit test of tracer in core

add comment

a1204c6

zstraw approved these changes Feb 2, 2023

View reviewed changes

wangtaohz added 2 commits February 2, 2023 11:44

Merge branch 'master' into fix-994-1

34f2344

remove useless import

ad8c723

zhoujinsong linked an issue Feb 2, 2023 that may be closed by this pull request

[Bug]: KeyedTableScanTask confuse files from BaseStore and ChangeStore #1045

Closed

1 task

wangtaohz added 2 commits February 2, 2023 13:55

refactor FileNameHandle to FileNameGenerator

3125755

add unit test for apache#1045

54897d6

baiyangtx reviewed Feb 2, 2023

View reviewed changes

wangtaohz added 2 commits February 2, 2023 17:11

1.spark remove txId for operations on UnkeyedTable

68324e6

2.support generate hive sub dir without txId

remove FileName transaction > 0 check

ac3a0dc

baiyangtx approved these changes Feb 2, 2023

View reviewed changes

zhoujinsong merged commit f7ce44e into apache:master Feb 2, 2023

YesOrNo828 approved these changes Feb 2, 2023

View reviewed changes

wangtaohz deleted the fix-994-1 branch February 17, 2023 03:23

YesOrNo828 mentioned this pull request Mar 24, 2023

[ARCTIC-1233] Arctic filestore write task can't been restored from checkpoint after modifying the sink parallelism #1275

Merged

3 tasks

This was referenced Apr 21, 2023

Issue for arctic Aireed/imgs#1

Open

[Subtask]: We need to perform regression testing on historical issues. #1411

Closed

wangtaohz mentioned this pull request May 8, 2023

[AMS-Refactor] merge allocating transaction id from AMS #1428

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ARCTIC-994] Introduce design of `TransactionId` generation to resolving data conflicts #1010

[ARCTIC-994] Introduce design of `TransactionId` generation to resolving data conflicts #1010

wangtaohz commented Jan 12, 2023 •

edited

Loading

wangtaohz commented Jan 13, 2023

YesOrNo828 left a comment

wangtaohz commented Feb 2, 2023

baiyangtx Feb 2, 2023

wangtaohz Feb 2, 2023

YesOrNo828 left a comment

[ARCTIC-994] Introduce design of TransactionId generation to resolving data conflicts #1010

[ARCTIC-994] Introduce design of TransactionId generation to resolving data conflicts #1010

Conversation

wangtaohz commented Jan 12, 2023 • edited Loading

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

wangtaohz commented Jan 13, 2023

YesOrNo828 left a comment

Choose a reason for hiding this comment

wangtaohz commented Feb 2, 2023

baiyangtx Feb 2, 2023

Choose a reason for hiding this comment

wangtaohz Feb 2, 2023

Choose a reason for hiding this comment

YesOrNo828 left a comment

Choose a reason for hiding this comment

[ARCTIC-994] Introduce design of `TransactionId` generation to resolving data conflicts #1010

[ARCTIC-994] Introduce design of `TransactionId` generation to resolving data conflicts #1010

wangtaohz commented Jan 12, 2023 •

edited

Loading