Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Add a `data_version` to the `db_write` hook an implement optimistic locking #3358
In addition this PR implements an optimistic locking strategy. The
It is not meant as a graceful shutdown, but rather a last resort protection. I
As a minor detail the order in which the increment and the reporting is
The lock provides linearizability of DB modifications: if a database is
When considering as well the possibillity of chaining
We may not have hook chaining yet, but it seems good to prepare the interface at this point as well, so that plugin authors for remote backup in particular can start adapting to the new interface, then later we can support hook chaining and thereby allow multiple backup plugins to hook the
Totally with you on the chainability of hooks, standardizing on
I'm not convinced that a
FWIW we could send a notification at
But the plugin doesn't know as a matter of fact, since we might terminate inbetween calling the hook and issuing the notification, so the plugin must handle the case where it doesn't get a notification but the DB transaction is still committed. Once you already have to handle that, why add yet another possible entrypoint for the happy case, and not just always assume the bad case? No matter how many notifications we add the plugin will have to have the ability to handle the case that a notification never comes :-)
I guess I'm worried about the overhead of notifications (it's small but not free) and plugin devs relying to heavily on the helpful notification and not bothering with the bad case, which would then result in loads of subtly broken plugins.
We were passing them in separately, while we could just retrieve them from the db instance instead.
We are about to do some more operations before committing, so moving this up allows us to reuse the same transaction.
This increments the `data_version` upon committing dirty transactions, reads the last data_version upon startup, and tracks the number in memory in parallel to the DB (see next commit for rationale). Changelog-Changed: JSON-RPC: Added a `data_version` field to the `db_write` hook which returns a numeric transaction counter.
The optimistic lock prevents multiple instances of c-lightning making concurrent modifications to the database. That would be unsafe as it messes up the state in the DB. The optimistic lock is implemented by checking whether a gated update on the previous value of the `data_version` actually results in an update. If that's not the case the DB has been changed under our feet. The lock provides linearizability of DB modifications: if a database is changed under the feet of a running process that process will `abort()`, which from a global point of view is as if it had crashed right after the last successful commit. Any process that also changed the DB must've started between the last successful commit and the unsuccessful one since otherwise its counters would not have matched (which would also have aborted that transaction). So this reduces all the possible timelines to an equivalent where the first process died, and the second process recovered from the DB. This is not that interesting for `sqlite3` where we are also protected via the PID file, but when running on multiple hosts against the same DB, e.g., with `postgres`, this protection becomes important. Changelog-Added: DB: Optimistic logging prevents instances from running concurrently against the same database, providing linear consistency to changes.