Lock DB Upgrades

[snadrus]

Allow max of 1 db upgrader at a time.

[snadrus]

Well no form of db locking is going to work:
github.com/filecoin-project/curio/harmony/harmonydb.(*DB).upgrade.func1.1
/home/circleci/curio/harmony/harmonydb/harmonydb.go:256
- ERROR: ACCESS EXCLUSIVE lock mode not supported yet (SQLSTATE 0A000)

[snadrus]

Is this a known problem @magik6k or a nice-to-have?
The only window of risk is on upgrade when a new schema arrives if 2 machines get there about the same time (single digit milliseconds). I think today it'll crash the "second" process (which should have a restart), then everything should be fine. I could "sleep on error and retry" to avoid that crash. WDYT?

[magik6k]

Is this a known problem @magik6k or a nice-to-have?

A problem that we didn't have any case that manifested in actual trouble, but I do foresee it being an issue at some point given that schema changes in YB do tend to take a while and are likely to be a large source of edge-cases given what YB does with postgres.

The only window of risk is on upgrade when a new schema arrives if 2 machines get there about the same time (single digit milliseconds).

It's usually much longer, in my deployments most migrations take 2-10s, depending on what they do

[LexLuthr]

https://yugabytedb.tips/suppress-the-error/ advisory locks are not supported. I found out yesterday the hard way after using them for libp2p table.

[snadrus]

I'd say we shelve this. If it becomes a problem, maybe the locks will arrive then and this code will be a drop-in fix.

[snadrus]

One alternative is to have an upgrade 'task' that gets scheduled, deduped, claimed by those only with that version, and ran that way, but we can't start the task engine with any other task. It sounds more difficult.

[magik6k]

I believe I see a good way to do this:

• Add a separate base_upgrades table
• When we don't see an entry in the base table for a given upgrade, write the entry to the base_upgrades table
  • If that write conflicts, wait a bit (~10s) and go to step one. Either someone else has applied the update, or it failed so manual intervention is needed anyways
  • If it didn't conflict, we're now responsible for applying the schema, so do that.

[snadrus]

Could we have a new bool column that is something like "in progress" and if anything shows to be "in progress" then we just wait 10s and try again? The writer would add the row with the field true, then do the change, finally remove the 'true'

…

On Tue, Oct 29, 2024 at 11:58 PM Łukasz Magiera ***@***.***> wrote: I believe I see a good way to do this: - Add a separate base_upgrades table - When we don't see an entry in the base table for a given upgrade, write the entry to the base_upgrades table - If that write conflicts, wait a bit (~10s) and go to step one. Either someone else has applied the update, or it failed so manual intervention is needed anyways - If it didn't conflict, we're now responsible for applying the schema, so do that. — Reply to this email directly, view it on GitHub <#160 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOU4LSYYXXXODJT3O3G753Z6BRN3AVCNFSM6AAAAABM2IO7JSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBVHA2DQNZRG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Andrew Jackson President of Curio Storage, Inc. <http://curiostorage.org>

[LexLuthr]

Closing for now as this has not been a problem in last few months.