sql: crdb_internal.leases could cause a deadlock with the lease manager #119305
Conversation
|
This change is |
fqazi
added
backport-22.2.x
fqazi
force-pushed
the
fixLeasingDeadlock
branch
from
93da138
to
21913d2
Compare
There was a problem hiding this comment.
lgtm! thanks for the quick fix, and great investigation for it
Reviewable status:
complete! 0 of 0 LGTMs obtained (waiting on @fqazi)
pkg/sql/crdb_internal.go line 919 at r1 (raw file):
generator: nil, unimplemented: false, resultColumns: nil,
super nit: do we need these extra nil fields defined?
Previously, a deadlock could occur between crdb_internal.leases and the lease manager when attempting to renew or release leases on the role_member inside crdb_internal.leases table. This deadlock occurs because lease manager lock is held while visting leases. The issue was easily reproducible when the allow_role_memberships_to_change_during_transaction setting was enabled. To resolve this, the patch introduces in-memory caching of all leases within crdb_internal.leases before privilege checks are performed, and any locks are subsequently released. Fixes: cockroachdb#119253 Release note (bug fix): crdb_internal.leases could cause a node to become unavailable due to a deadlock in the leasing subsystem.
fqazi
force-pushed
the
fixLeasingDeadlock
branch
from
21913d2
to
10618f3
Compare
There was a problem hiding this comment.
@rafiss TFTR!
Reviewable status:
pkg/sql/crdb_internal.go line 919 at r1 (raw file):
Previously, rafiss (Rafi Shamim) wrote…
super nit: do we need these extra nil fields defined?
Done.
|
bors r+ |
|
Build succeeded: |
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
Encountered an error creating backports. Some common things that can go wrong:
You might need to create your backport manually using the backport tool. error creating merge commit from 10618f3 to blathers/backport-release-22.2-119305: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict [] you may need to manually resolve merge conflicts with the backport tool. Backport to branch 22.2.x failed. See errors above. error creating merge commit from 10618f3 to blathers/backport-release-23.1-119305: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict [] you may need to manually resolve merge conflicts with the backport tool. Backport to branch 23.1.x failed. See errors above. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
Previously, a deadlock could occur between crdb_internal.leases and the lease manager when attempting to renew or release leases on the role_member inside crdb_internal.leases table. This deadlock occurs because lease manager lock is held while visting leases. The issue was easily reproducible when the
allow_role_memberships_to_change_during_transaction setting was enabled. To resolve this, the patch introduces in-memory caching of all leases within crdb_internal.leases before privilege checks are performed, and any locks are subsequently released.
Fixes: #119253
Fixes: #97280
Release note (bug fix): crdb_internal.leases could cause a node to become unavailable due to a deadlock in the leasing subsystem.