Skip to content

Fixed database deadlock when syncer crash - #294

Merged
w41ter merged 2 commits into
selectdb:devfrom
wubiaoi:fix_db_deadlock
Dec 16, 2024
Merged

Fixed database deadlock when syncer crash#294
w41ter merged 2 commits into
selectdb:devfrom
wubiaoi:fix_db_deadlock

Conversation

@wubiaoi

@wubiaoi wubiaoi commented Dec 5, 2024

Copy link
Copy Markdown
Contributor

Currently, in database transaction processing, if an exception occurs without abort, there is no lock release.
When a node dies and there is no sync task, there is a divisor 0 exception in RebalanceLoad.

ERROR checker failed, host info: 10.119.48.44:8411, err: [db] Error 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

panic: runtime error: integer divide by zero

goroutine 187 [running]:
github.com/selectdb/ccr_syncer/pkg/storage.RebalanceLoad(0x0, 0xb36dc8?, {0xc0006b82a0?, 0x1?, 0x30?})
        /root/src/ccr-syncer-data/pkg/storage/utils.go:59 +0xea
github.com/selectdb/ccr_syncer/pkg/storage.(*MysqlDB).RebalanceLoadFromDeadSyncers(0xc000534070, {0xc0000fb340, 0x1, 0x1})
        /root/src/ccr-syncer-data/pkg/storage/mysql.go:378 +0x1ab
github.com/selectdb/ccr_syncer/pkg/ccr.(*Checker).handleRebalance(0xc0001d4000)
        /root/src/ccr-syncer-data/pkg/ccr/checker.go:128 +0x92

@w41ter w41ter left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, since a defer function is executed, please remove the other rollback invoking, eg the below lines

			if err := txn.Rollback(); err != nil {
				return xerror.Wrap(err, xerror.DB, "mysql: rebalance rollback failed.")
			}

Comment thread pkg/storage/mysql.go
}

defer func() {
if err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The err is a shadow variable here.

A better way is to make err as a named return value, eg:

func (s *MysqlDB) GetStampAndJobs(hostInfo string) (int64, []string, err error) {

and change the usage of err from , err := to , err = .

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@w41ter
w41ter merged commit fd95559 into selectdb:dev Dec 16, 2024
w41ter pushed a commit that referenced this pull request Dec 16, 2024
Co-authored-by: wubiao02 <wubiao02@meituan.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants