feature: add a Executor for INSERT ON DUPLICATE KEY UPDATE #3374

huan415 · 2020-12-14T12:10:41Z

Ⅰ. Describe what this PR did

新加一个Executor, 执行INSERT ON DUPLICATE KEY UPDATE

Ⅱ. Does this pull request fix one issue?

INSERT ON DUPLICATE KEY UPDATE当update时会抛异常

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

如果含有ON DUPLICATE KEY UPDATE则走新的执行器，利用唯一索引查前置和后置镜像，如果有前置镜像则包含update，没有前置镜像则为insert
如果采用foreach，一条sql可能有三种情况：
1.只有insert，SQLUndoLog的sqlType是insert（唯一索引都不存在）
2.只有update，SQLUndoLog的sqlType是update（唯一索引已全部存在）
3.既有insert，又有有update，两个SQLUndoLog，sqlType分别是insert、update（唯一索引检验部分存在）

Ⅴ. Special notes for reviews

rm-datasource/src/main/java/io/seata/rm/datasource/exec/ExecuteTemplate.java

caohdgege · 2020-12-14T16:54:26Z

为什么这个有executor，但是没有对应的undoExecutor的，这样的话，在回滚的场景下是怎么处理的？

rm-datasource/src/main/java/io/seata/rm/datasource/exec/ExecuteTemplate.java

huan415 · 2020-12-22T12:10:06Z

，

为什么这个有executor，但是没有对应的undoExecutor的，这样的话，在回滚的场景下是怎么处理的？

undoExecutor这个逻辑不用动。还是走原来的逻辑。
回滚场景是根据SQLUndoLog的SqlType来判断，这种sql生成的SQLUndoLog可能是insert也可能是update，也有可能两种都有，

sqlparser/seata-sqlparser-core/src/main/java/io/seata/sqlparser/SQLInsertRecognizer.java

rm-datasource/src/main/java/io/seata/rm/datasource/exec/mysql/MySQLInsertOrUpdateExecutor.java

caohdgege · 2020-12-22T14:54:34Z

    @GlobalTransactional(timeoutMills = 3000000)
    public void tcc(@RequestParam(value = "times", required = false, defaultValue = "1") int times,
                    @RequestParam(value = "batchNum", required = false, defaultValue = "100") int batchNum,
                    @RequestParam(value = "thr", required = false, defaultValue = "0") int thr) throws SQLException {
        Connection connection = dataSource.getConnection();
        System.out.println(connection instanceof ConnectionProxy);
        String sql = "insert into user(id, username, PASSWORD, email, phone, sex) values (1, 'fdafdasdfa', 'fdasfas', 'fdafasdd', 'fdasfasdfasdf', ?),  (3002, 'fdafdasdfa', 'fdasfas', 'fdafasdd', 'fdasfasdfasdf', ?) ON DUPLICATE KEY UPDATE username='fdafas'";
        PreparedStatement ps = connection.prepareStatement(sql);
        ps.setInt(1, 0);
        ps.setInt(2, 1);
        ps.executeUpdate();
        ps.executeUpdate();
        if (true) {
            throw new RuntimeException();
        }
    }

我数据库里面3002的数据是存在的，id=1的数据是不存在的，username是唯一键，这样测试的话，在回滚的时候会把我id=3002的数据误删除。
附上我user表的建表语句。

create table if not exists seata_tx_test.user
(
	id bigint unsigned auto_increment
		primary key,
	username varchar(255) not null comment '用户名',
	PASSWORD char(36) not null comment '密码',
	email char(100) not null comment '邮箱',
	phone char(20) not null comment '电话号码',
	logo char(255) default '/default-logo.png' not null comment '头像',
	address char(150) default '' not null comment '地址',
	sex tinyint default 0 not null comment '0->未设置 1->男 2->女',
	constraint email
		unique (email),
	constraint uniq_phone_sex
		unique (phone, sex),
	constraint username
		unique (username)
)
collate=utf8mb4_general_ci;

caohdgege · 2020-12-22T14:56:39Z

把开发分支合并到你的分支再推一下，看一下ci能不能过。

codecov-io · 2020-12-22T15:29:04Z

Codecov Report

Merging #3374 (7b4408c) into develop (aa17303) will decrease coverage by 0.21%.
The diff coverage is 29.94%.

@@              Coverage Diff              @@
##             develop    #3374      +/-   ##
=============================================
- Coverage      52.09%   51.88%   -0.22%     
- Complexity      3511     3524      +13     
=============================================
  Files            638      639       +1     
  Lines          21147    21333     +186     
  Branches        2629     2658      +29     
=============================================
+ Hits           11016    11068      +52     
- Misses          9044     9168     +124     
- Partials        1087     1097      +10

Impacted Files	Coverage Δ	Complexity Δ
...a/io/seata/rm/datasource/exec/ExecuteTemplate.java	`5.88% <0.00%> (-0.57%)`	`2.00 <0.00> (ø)`
...parser/antlr/mysql/AntlrMySQLInsertRecognizer.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...sqlparser/druid/oracle/OracleInsertRecognizer.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...r/druid/postgresql/PostgresqlInsertRecognizer.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...a/sqlparser/druid/mysql/MySQLInsertRecognizer.java	`75.47% <9.09%> (-17.56%)`	`16.00 <2.00> (+1.00)`	⬇️
...source/exec/mysql/MySQLInsertOrUpdateExecutor.java	`32.93% <32.93%> (ø)`	`13.00 <13.00> (?)`
...n/src/main/java/io/seata/common/util/IdWorker.java	`77.08% <0.00%> (-6.25%)`	`11.00% <0.00%> (-1.00%)`

caohdgege · 2020-12-23T02:13:04Z

我觉得这个应该算是特性 feature，之前不支持 ON DUPLICATE KEY UPDATE，这个pr就是让seata支持这种语法。
还有标题稍微改一下，pr的标题不要用中文。

huan415 · 2020-12-24T02:46:14Z

    @GlobalTransactional(timeoutMills = 3000000)
    public void tcc(@RequestParam(value = "times", required = false, defaultValue = "1") int times,
                    @RequestParam(value = "batchNum", required = false, defaultValue = "100") int batchNum,
                    @RequestParam(value = "thr", required = false, defaultValue = "0") int thr) throws SQLException {
        Connection connection = dataSource.getConnection();
        System.out.println(connection instanceof ConnectionProxy);
        String sql = "insert into user(id, username, PASSWORD, email, phone, sex) values (1, 'fdafdasdfa', 'fdasfas', 'fdafasdd', 'fdasfasdfasdf', ?),  (3002, 'fdafdasdfa', 'fdasfas', 'fdafasdd', 'fdasfasdfasdf', ?) ON DUPLICATE KEY UPDATE username='fdafas'";
        PreparedStatement ps = connection.prepareStatement(sql);
        ps.setInt(1, 0);
        ps.setInt(2, 1);
        ps.executeUpdate();
        ps.executeUpdate();
        if (true) {
            throw new RuntimeException();
        }
    }

我数据库里面3002的数据是存在的，id=1的数据是不存在的，username是唯一键，这样测试的话，在回滚的时候会把我id=3002的数据误删除。
附上我user表的建表语句。

create table if not exists seata_tx_test.user
(
	id bigint unsigned auto_increment
		primary key,
	username varchar(255) not null comment '用户名',
	PASSWORD char(36) not null comment '密码',
	email char(100) not null comment '邮箱',
	phone char(20) not null comment '电话号码',
	logo char(255) default '/default-logo.png' not null comment '头像',
	address char(150) default '' not null comment '地址',
	sex tinyint default 0 not null comment '0->未设置 1->男 2->女',
	constraint email
		unique (email),
	constraint uniq_phone_sex
		unique (phone, sex),
	constraint username
		unique (username)
)
collate=utf8mb4_general_ci;

刚刚试了一下，用mybatis不会。这个是因为单引号的问题（SELECT * FROM user WHERE (email = ' 'fdafasdd'' ) OR (phone = ' 'fdasfasdfasdf'' and sex = 0 ) OR (username = ' 'fdafdasdfa'' ) OR (email = ' 'fdafasdd'' ) OR (phone = ' 'fdasfasdfasdf'' and sex = 1 ) OR (username = ' 'fdafdasdfa'' ) FOR UPDATE），所以认为获取不到前置镜像，认为两条都是insert，回滚时都删掉。这是个bug，我想想

huan415 · 2020-12-24T02:46:46Z

feature

ok

merge origin seata project

huan415 · 2021-03-16T05:12:23Z

@huan415 CI failed.Please fix it.

已解决

xingfudeshi

LGTM.

...eata-sqlparser-druid/src/main/java/io/seata/sqlparser/druid/mysql/MySQLInsertRecognizer.java

rm-datasource/src/main/java/io/seata/rm/datasource/exec/mysql/MySQLInsertOrUpdateExecutor.java

codecov-commenter · 2021-04-24T05:43:24Z

Codecov Report

Merging #3374 (442b795) into develop (0e98040) will decrease coverage by 0.25%.
The diff coverage is 26.92%.

@@              Coverage Diff              @@
##             develop    #3374      +/-   ##
=============================================
- Coverage      51.94%   51.68%   -0.26%     
- Complexity      3506     3517      +11     
=============================================
  Files            639      640       +1     
  Lines          21184    21391     +207     
  Branches        2633     2670      +37     
=============================================
+ Hits           11004    11056      +52     
- Misses          9087     9231     +144     
- Partials        1093     1104      +11

Impacted Files	Coverage Δ	Complexity Δ
...a/io/seata/rm/datasource/exec/ExecuteTemplate.java	`5.88% <0.00%> (-0.57%)`	`2.00 <0.00> (ø)`
...parser/antlr/mysql/AntlrMySQLInsertRecognizer.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...a/sqlparser/druid/mysql/MySQLInsertRecognizer.java	`61.90% <0.00%> (-31.12%)`	`15.00 <1.00> (ø)`
...sqlparser/druid/oracle/OracleInsertRecognizer.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...r/druid/postgresql/PostgresqlInsertRecognizer.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...source/exec/mysql/MySQLInsertOrUpdateExecutor.java	`31.46% <31.46%> (ø)`	`13.00 <13.00> (?)`
...n/src/main/java/io/seata/common/util/IdWorker.java	`77.08% <0.00%> (-6.25%)`	`11.00% <0.00%> (-1.00%)`

caohdgege

LGTM

...eata-sqlparser-druid/src/main/java/io/seata/sqlparser/druid/mysql/MySQLInsertRecognizer.java

funky-eyes

LGTM

bugfix:INSERT ON DUPLICATE KEY UPDATE当update时会抛异常，新加一个Executor

88adb45

funky-eyes reviewed Dec 14, 2020

View reviewed changes

rm-datasource/src/main/java/io/seata/rm/datasource/exec/ExecuteTemplate.java Outdated Show resolved Hide resolved

caohdgege reviewed Dec 14, 2020

View reviewed changes

rm-datasource/src/main/java/io/seata/rm/datasource/exec/ExecuteTemplate.java Outdated Show resolved Hide resolved

huan415 added 7 commits December 15, 2020 08:30

bug:checkStyle

0dcb1f3

Not allow chinese character ! [RegexpSingleline]

7d2b96f

'->' is not preceded with whitespace

46d514a

'for' is not followed by whitespace

fd5b443

is not followed by whitespace

f8deb06

is not preceded with whitespace

6fe3166

is not preceded with whitespace. [WhitespaceAround]

635ab95

caohdgege reviewed Dec 22, 2020

View reviewed changes

It should be set to a constant

4155abd

funky-eyes added first-time contributor first-time contributor module/rm-datasource rm-datasource module labels Dec 23, 2020

huan415 and others added 7 commits December 24, 2020 21:29

bugfix:insertParams is characterstring constant

5c9f074

bugfix:insert params is characterstring constant

6202dea

bugfix:when buildImage support index is PRIMARY

0c89f8c

Merge pull request #1 from seata/develop

ad5d24e

merge origin seata project

'>' is not followed by whitespace

d4db6da

')' is preceded with whitespace

5323f25

'.*' form of import should be avoide

326a058

huan415 changed the title ~~bugfix:INSERT ON DUPLICATE KEY UPDATE当update时会抛异常，新加一个Executor~~ feature:INSERT ON DUPLICATE KEY UPDATE当update时会抛异常，新加一个Executor Dec 26, 2020

huan415 and others added 3 commits March 14, 2021 20:13

resove test case

4c6b844

Merge branch 'develop' into develop

ae35ac1

Merge branch 'develop' into develop

5891b9a

xingfudeshi approved these changes Mar 17, 2021

View reviewed changes

xingfudeshi and others added 4 commits March 19, 2021 10:16

Merge branch 'develop' into develop

5e88d85

afterImage=beforeImageId+uniqueIndex

5415715

Merge branch 'develop' of https://github.com/huan415/seata into develop

bff2baf

Merge branch 'develop' into develop

0548462

ymwangzq suggested changes Apr 6, 2021

View reviewed changes

...eata-sqlparser-druid/src/main/java/io/seata/sqlparser/druid/mysql/MySQLInsertRecognizer.java Outdated Show resolved Hide resolved

ymwangzq suggested changes Apr 7, 2021

View reviewed changes

rm-datasource/src/main/java/io/seata/rm/datasource/exec/mysql/MySQLInsertOrUpdateExecutor.java Show resolved Hide resolved

xingfudeshi and others added 5 commits April 16, 2021 17:13

Merge branch 'develop' into develop

7b4408c

fix bug when update primaryKey

918c537

Merge branch 'develop' into develop

a1af8db

import class

fdaf16b

fix bug of update pk

fe22815

huan415 added 2 commits April 27, 2021 13:18

Merge branch 'develop' into develop

54875f4

Merge branch 'develop' into develop

b9e0a55

caohdgege approved these changes May 3, 2021

View reviewed changes

funky-eyes requested changes May 3, 2021

View reviewed changes

...eata-sqlparser-druid/src/main/java/io/seata/sqlparser/druid/mysql/MySQLInsertRecognizer.java Outdated Show resolved Hide resolved

resove proplem of ci

090c84c

huan415 closed this May 3, 2021

huan415 reopened this May 3, 2021

funky-eyes changed the title ~~feature:add a Executor for INSERT ON DUPLICATE KEY UPDATE~~ feature: add a Executor for INSERT ON DUPLICATE KEY UPDATE May 3, 2021

Merge branch 'develop' into develop

442b795

funky-eyes approved these changes May 3, 2021

View reviewed changes

funky-eyes added this to the 1.5.0 milestone May 3, 2021

funky-eyes merged commit 38226ba into apache:develop May 3, 2021

funky-eyes mentioned this pull request Sep 5, 2022

seata 1.5.2 jpa session请求 cpu飚高疯狂gc #4909

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: add a Executor for INSERT ON DUPLICATE KEY UPDATE #3374

feature: add a Executor for INSERT ON DUPLICATE KEY UPDATE #3374

huan415 commented Dec 14, 2020 •

edited

caohdgege commented Dec 14, 2020

huan415 commented Dec 22, 2020

caohdgege commented Dec 22, 2020

caohdgege commented Dec 22, 2020

codecov-io commented Dec 22, 2020 •

edited

caohdgege commented Dec 23, 2020

huan415 commented Dec 24, 2020

huan415 commented Dec 24, 2020

huan415 commented Mar 16, 2021

xingfudeshi left a comment

codecov-commenter commented Apr 24, 2021 •

edited

caohdgege left a comment

funky-eyes left a comment

feature: add a Executor for INSERT ON DUPLICATE KEY UPDATE #3374

feature: add a Executor for INSERT ON DUPLICATE KEY UPDATE #3374

Conversation

huan415 commented Dec 14, 2020 • edited

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

caohdgege commented Dec 14, 2020

huan415 commented Dec 22, 2020

caohdgege commented Dec 22, 2020

caohdgege commented Dec 22, 2020

codecov-io commented Dec 22, 2020 • edited

Codecov Report

caohdgege commented Dec 23, 2020

huan415 commented Dec 24, 2020

huan415 commented Dec 24, 2020

huan415 commented Mar 16, 2021

xingfudeshi left a comment

Choose a reason for hiding this comment

codecov-commenter commented Apr 24, 2021 • edited

Codecov Report

caohdgege left a comment

Choose a reason for hiding this comment

funky-eyes left a comment

Choose a reason for hiding this comment

huan415 commented Dec 14, 2020 •

edited

codecov-io commented Dec 22, 2020 •

edited

codecov-commenter commented Apr 24, 2021 •

edited