regionliveness: add support for sqlinstances and recovery

[fqazi]

This PR completes the regionliveness survivability goal work by implementing the following:

1. Logic to bound the sqlliveness reviewals based on the unavailable_at time set on a region
2. Updating the sqlinstance allocation logic to take into account regionliveness by working on live regions only.
3. Updating the change feed initial scan on sqlinstances to work on a per-region basis and consider region liveness.
4. Updating the sqlinstance allocation logic to recover from region failures by cleaning up system.leases, system.sqlinstances, and system.sqllivness after a failure.
5. A new roachtest focused on setting up a physical cluster and simulating failure scenarios for nightly builds.
6. A synthetic test for simulating region failures and recovery from them

This PR is stacked on top of: #115568, so the first 4 commits should be ignored.

informs: #103727
EPIC: CC-24173

[blathers-crl]

Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[fqazi]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DarrylWong, @herkolategan, and @JeffSwenson)

---

pkg/sql/sqlinstance/instancestorage/instancestorage.go line 292 at r7 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

Instead of saving the sql server's own session can we run the clean up operation before allocating a session ID?

Done.

---

pkg/sql/sqlliveness/slstorage/slstorage.go line 543 at r5 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

nit: this block can be simplified by removing the delta calculation. If the region is unavailable before the calculated expiration timestamp, then we can use that timestamp as the new expiration.

regionUnavailableAt := unavailableRegions[string(region)].GoTime()
if regionUnavailableAt.Less(exp) {
    exp = regionUnavailableAt
}

The problem that we have is that the timestamp is stored as DTimestamp, and the expiration is a hlc.Timestamp. The delta gives an approximate conversion.

---

pkg/sql/sqlliveness/slstorage/slstorage.go line 546 at r5 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

Is nil the right value to return if the expiration timestamp is in the past? I think Update is only called when heart beating a server's own session. So this case indicates the server's own session has expired.

Done.

---

pkg/sql/sqlliveness/slstorage/slstorage.go line 556 at r5 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

I don't think we need to set the transaction deadline here. It is valid to write a liveness row in an unavailable region as long as the expires_at timestamp is in the past, which would be true if the transaction gets pushed.

Done.

[fqazi]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DarrylWong, @herkolategan, and @JeffSwenson)

---

pkg/sql/sqlliveness/slstorage/slstorage.go line 517 at r17 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

The implementation of the recovery protocol looks correct to me. But there is one potential availability issue in the implementation I noticed while reviewing the RFC.

The RFC contains this line:

If unvailable_at is not null, but set to a timestamp after the transaction timestamp, a SQL server can attempt to set it to null in a transaction modifying only the region_liveness table.

I think the reason I wrote that is I was assuming sql liveness would attempt to clear an unavailable_at set for its region before the region is treated as unavailable. Currently, if any server sets unavailable_at the sql servers will all restart even if their region becomes available before the sessions would expire.

I believe it is safe to clear an unavailable_at without deleting any of the survive zone rows as long as the transaction deadline comes before the unavailable_at timestamp.

This logic probably belongs in Storage.Update and should run whenever an unavailable at timestamp is observed and the sql server was able to read from the sqlliveness table.

Done.

[fqazi]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @DarrylWong, @herkolategan, and @JeffSwenson)

---

pkg/sql/sqlliveness/slstorage/slstorage.go line 545 at r36 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

I think the exp := expiration.GoTime(); exp.After(ts) check is a little too aggressive. The error case should only occur if the unavailable at timestamp comes before the transaction timestamp. If the requested expiration time comes after the unavailable time, then we can clamp the expiration time to the unavailable time.

Done.

---

pkg/sql/sqlliveness/slstorage/slstorage.go line 596 at r38 (raw file):

Previously, JeffSwenson (Jeff Swenson) wrote…

I think this operation doesn't actually care about the expiration timestamp. It only cares about the unavailable at instance. As long as the unavailable at instance is still in the future it can clear the row.

That also makes it easier to extract the logic into a helper in the region liveness package because using the unavailable_at instant as the deadline allows the reset logic to only cares about the state of the region liveness table.

Done.

[JeffSwenson]

LGTM 💯

[fqazi]

@JeffSwenson TFTR!

bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)