Optionally prevent unintended database schema updates

[sjmudd]

Some time ago we had 2 configuration settings to optionally avoid database upgrades:

• SkipOrchestratorDatabaseUpdate
• SmartOrchestratorDatabaseUpdate

These were removed as the code to handle updates was improved and considered more reliable.
I think that a similar option would be good to reinstate, especially on large environments.

Reasoning is:

• the current assumption is that all versions of the code in use are identical. If you deploy large amounts of servers and also run orchestrator on application nodes too then achieving this instantly is likely to become harder and harder.
• if you have a cluster of orchestrator nodes (providing an http service) then you probably want to update the database only when no nodes are trying to write to the database so possibly only with a single node being active. You really don't want any other servers suddenly trying to upgrade the database without prior notice and preparation.

So probably in a small environment it's ok to let this happen automatically and I think it's fine for it to be the default behaviour. However, an option to prevent this would at least ensure that an unintended run by some orchestrator binary won't generate ALTER TABLE commands which potentially might lock up the main orchestrator node while it tries to write to the same table.

I suggest we add a configuration setting AvoidDatabaseUpdates bool which by default has a value of false and intend to write a patch to support such a change.

I have been bitten by this sort of thing happening a number of times now and would prefer to stop everything and only on a single node allow the update, after which point the configuration setting would be disabled again.

[sjmudd]

Might it be simplest to revert this: 54a0b54 ?

[shlomi-noach]

Yes, makes perfect sense!

[shlomi-noach]

addressed by #85

[shlomi-noach]

closed by #85