[unmerging] Hash getting stuck in locked state

[lobsterkatie]

A customer reported they were unable to unmerge a specific fingerprint, because they tried once, didn't see the result (perhaps this was before the "please wait up to 24 hours for changes to take effect" warning was there), and then every time they tried again, they got an error that unmerging was already in progress.

I checked the hash in the GroupHash table (sentry_grouphash), and indeed, its state was set to 1, which is the "locked in migration" state. 24 hours later, nothing had changed, so it seems to have gotten stuck that way somehow. The unmerging has not, in fact, happened (the hash is still associated with the old group in sentry_grouphash), so it seems that things fell off somewhere between the hash getting locked here

sentry/src/sentry/tasks/unmerge.py

Line 502 in 33283e4

locked_primary_hashes = lock_hashes(

and it getting unlocked here

sentry/src/sentry/tasks/unmerge.py

Line 529 in 33283e4

unlock_hashes(args.project_id, locked_primary_hashes)

.

This is the first report I've heard of this, and we don't seem to have any prior GH issues on the matter, either. I also can't reproduce it, so for now I'm going to chalk it up to gremlins, and we can work with the customer to get this particular hash unstuck. I'm writing up this issue just in case it happens again, though, so we have some place to collect the data, at which point we can think about investigating it further.

(For posterity, in case we do look further into this, the hash in question is 6618c48ad845c4be8fc8635ea060fd7f, from project 6239706, and it's currently stuck in group 3994122231.)