The problem of using --rows and --chunk-filesize at the same time

[ronylei]

1. mydumer's version

# mydumper --version
mydumper 0.12.3-3, built against MySQL 5.7.37-40

2. Data information

mysql> select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
|  5000000 |
+----------+
1 row in set (6.97 sec)

3.start backup

# mydumper -u xxx -h xxx -p xxx --skip-tz-utc --outputdir=/data/sql/ -T test.sbtest1 --no-schemas -v 3 -t 2 --chunk-filesize=1 --complete-insert --no-locks -r 500000

** (mydumper:2030): WARNING **: 09:36:25.651: We are going to chunk by row and by filesize
** Message: 09:36:25.656: MyDumper backup version: 0.12.3-3
** Message: 09:36:25.700: Server version reported as: 8.0.29
** Message: 09:36:25.705: Connected to a MySQL server

** (mydumper:2030): WARNING **: 09:36:25.715: Executing in no-locks mode, snapshot will not be consistent
** Message: 09:36:25.726: Started dump at: 2022-06-21 09:36:25
** Message: 09:36:25.729: Written master status
** Message: 09:36:25.766: Thread 1 connected using MySQL connection ID 179
** Message: 09:36:25.777: Thread 2 connected using MySQL connection ID 180
** Message: 09:36:25.790: Thread 1 dumping schema create for `test`
** Message: 09:36:25.809: Thread 1 dumping data for `test`.`sbtest1` WHERE `id` IS NULL OR (`id` >= 1 AND `id` < 1000001) | Remaining jobs: 2
** Message: 09:36:25.809: Thread 2 dumping data for `test`.`sbtest1` WHERE (`id` >= 1000001 AND `id` < 2000001) | Remaining jobs: 3
** Message: 09:36:25.809: Shutdown jobs enqueued
** Message: 09:36:25.810: Waiting jobs to complete
** Message: 09:36:35.930: Thread 1 dumping data for `test`.`sbtest1` WHERE (`id` >= 2000001 AND `id` < 3000001) | Remaining jobs: 4
** Message: 09:36:35.987: Thread 2 dumping data for `test`.`sbtest1` WHERE (`id` >= 3000001 AND `id` < 4000001) | Remaining jobs: 3
** Message: 09:36:46.295: Thread 1 dumping data for `test`.`sbtest1` WHERE (`id` >= 4000001 AND `id` < 5000001) | Remaining jobs: 2
** Message: 09:36:46.296: Thread 2 shutting down
** Message: 09:36:56.041: Thread 1 shutting down
** Message: 09:36:56.042: Main connection closed
** Message: 09:36:56.043: Finished dump at: 2022-06-21 09:36:56

4. View backup files，2M files found

### --rows=500000 and  --chunk-filesize=1
# ls -lh
total 396M
-rw-r--r-- 1 root root   230 Jun 21 09:54 metadata
-rw-r--r-- 1 root root  2.0M Jun 21 09:54 test.sbtest1.00000.00001.sql
-rw-r--r-- 1 root root  2.0M Jun 21 09:54 test.sbtest1.00000.00002.sql
-rw-r--r-- 1 root root  2.0M Jun 21 09:54 test.sbtest1.00000.00003.sql
-rw-r--r-- 1 root root  2.0M Jun 21 09:54 test.sbtest1.00000.00004.sql
-rw-r--r-- 1 root root  2.0M Jun 21 09:54 test.sbtest1.00000.00005.sql
...

### --rows=500000 and no --chunk-filesize
# ls -lh
total 499M
-rw-r--r-- 1 root root  192 Jun 21 09:55 metadata.partial
-rw-r--r-- 1 root root 193M Jun 21 09:55 test.sbtest1.00000.sql
-rw-r--r-- 1 root root 194M Jun 21 09:55 test.sbtest1.00001.sql
...

5. There is an error in recovery, such as：Duplicate entry '4994851' for key 'sbtest1.PRIMARY'

[davidducos]

Hi @ronylei,
On the recovery command, are you using -o option?
Do you see in the files, 2 lines with 4994851 as primary key?

[ronylei]

Hi @davidducos ,

mydumper --help
...
  -r, --rows                      Try to split tables into chunks of this many rows. This option turns off --chunk-filesize
...

I understand that when --rows=500000 and --chunk-filesize=1 are used at the same time, --rows shall prevail. Why is the file size 2M?

[ronylei]

1. myloader command:

myloader -u xxx -h xxx -p xxx -d /data/sql/ -v 3 -t 16

2.error info is

[ronylei]

test.sbtest1.00000.00011.sql file information by error：

[ronylei]

mydumper/src/mydumper_working_thread.c

[davidducos]

Hi @ronylei,

Can you compile from the master and test again?

[ronylei]

It is no problem to use the following version

[root@percona /]# mydumper --version
mydumper 0.12.5-1, built against MySQL 5.7.38-41