Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[filerstore] avoid mysqld 8.0 deadlocks #5062

Open
kmlebedev opened this issue Dec 1, 2023 · 2 comments
Open

[filerstore] avoid mysqld 8.0 deadlocks #5062

kmlebedev opened this issue Dec 1, 2023 · 2 comments

Comments

@kmlebedev
Copy link
Contributor

Describe the bug

mysqld deadlocks cause complete sticking of filers.

System Setup
Install cluster seaweedfs and pxc

x3 weed 3.58
x3 pxc 8.0.34

run warp

warp put --duration 30m --concurrent=8 --host=192.168.1.1:8333,192.168.1.2:8333,192.168.1.2:8333 --noclear --obj.size 10KiB --md5 --host-select=roundrobin

Additional context

SHOW ENGINE INNODB STATUS;
first insert active file 1854.nLO2HWqgc(SaDhNk.rnd

---TRANSACTION 6195869, ACTIVE 216 sec
mysql tables in use 1, locked 1
2 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 1
MySQL thread id 155, OS thread handle 140489365198592, query id 29633 192.168.1.1 seaweedfs wsrep: replicating and certifying write set(-1)
INSERT INTO `filemeta` (`dirhash`,`name`,`directory`,`meta`) VALUES(4592385983745796278,'1854.nLO2HWqgc(SaDhNk.rnd','/buckets/benchmark-01/M52KkLgq',_binary'\Z7���� �������*�spaYBCR2TdYdqCWWdRSkCA==:\r����Ν����LUw\"+������ᦫ���� ��(��0�ᦫ�r�����$vM���%�u���')

second update locked file 1854.nLO2HWqgc(SaDhNk.rnd

---TRANSACTION 6195878, ACTIVE 46 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 1128, 1 row lock(s)
MySQL thread id 168, OS thread handle 140491468752640, query id 32705 192.168.1.1 seaweedfs updating
UPDATE `filemeta` SET `meta` = _binary'\Z7���� ����ʮ���*�spaYBCR2TdYdqCWWdRSkCA==:\r����ӝ����Š\Z\"+������⦫���� ��(��0�⦫�r�����$vM���%�u���' WHERE `dirhash` = 4592385983745796278 AND `name` = '1854.nLO2HWqgc(SaDhNk.rnd' AND `directory` = '/buckets/benchmark-01/M52KkLgq'
------- TRX HAS BEEN WAITING 46 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 5 page no 71181 n bits 88 index PRIMARY of table `seaweedfs`.`filemeta` trx id 6195878 lock_mode X locks rec but not gap waiting
Record lock, heap no 22 PHYSICAL RECORD: n_fields 6; compact format; info bits 0
 0: len 8; hex bfbb6eb880a6a8b6; asc   n     ;;
 1: len 25; hex 313835342e6e4c4f32485771676328536144684e6b2e726e64; asc 1854.nLO2HWqgc(SaDhNk.rnd;;
 2: len 6; hex 0000005e8a9d; asc    ^  ;;
 3: len 7; hex 82000001450110; asc     E  ;;
 4: len 30; hex 2f6275636b6574732f62656e63686d61726b2d30312f4d35324b6b4c6771; asc /buckets/benchmark-01/M52KkLgq;;
 5: len 30; hex 1a371880900320e884f5c28bababce172a18737061594243523254645964; asc  7              * spaYBCR2TdYd; (total 102 bytes);

@kmlebedev
Copy link
Contributor Author

Add dsn for set writeTimeout #5060

@kmlebedev kmlebedev changed the title [filerstore] avoid mysqld deadlocks [filerstore] avoid mysqld 8.0 deadlocks Dec 21, 2023
@kmlebedev
Copy link
Contributor Author

The problem is not reproducible on MySQL 5.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant