Implement keyAboveWatermark for composite chunker with restrictions

[morgo]

It is documented that key{Above|Below}Watermark only applies to the optimistic chunker. We should add it to the composite chunker with restrictions:

• Only applies to key[0].
• Only applies when key[0] is numeric (initially, we could relax this in future).

The relevant code is:

spirit/pkg/table/chunker_composite.go

Lines 480 to 490 in 30d3e41

	// KeyAboveHighWatermark is not yet supported for composite chunker.
	// See: https://github.com/block/spirit/issues/479
	func (t *chunkerComposite) KeyAboveHighWatermark(key0 any) bool {
	return false
	}
	// KeyBelowLowWatermark is not yet supported for composite chunker.
	// See: https://github.com/block/spirit/issues/479
	func (t *chunkerComposite) KeyBelowLowWatermark(key0 any) bool {
	return true
	}

The advantage of implementing this feature is as follows:

1. For KeyAbove it eliminates some changes needing to be applied. This may be less important compared to auto_inc tables, because users are less likely to be inserting/modifying data at the end of the key range.
2. For KeyBelow it delays applying changes until after the copier is guaranteed to be past that point. This can be really useful in preventing contention cases (deadlocks, lock waits), which have been observed a few times between applier and copier.

[morgo]

We had a use case internally where we'd like this, but this might be a restriction:

Only applies when key[0] is numeric (initially, we could relax this in future).

It should be possible to implement this for non memory comparable keys too. It will potentially cause checksum failures, since order with collations is not easy to replicate in go code.

Before the checksum is started, the key above/below watermarks are disabled:

spirit/pkg/migration/runner.go

Lines 247 to 251 in 8ae9416

	// Disable both watermark optimizations so that all changes can be flushed.
	// The watermark optimizations can prevent some keys from being flushed,
	// which would cause flushedPos to not advance, leading to a mismatch
	// with bufferedPos and causing AllChangesFlushed() to return false.
	r.replClient.SetWatermarkOptimization(false)

This means that there is no risk of the changes post-checksum introducing any corruption.

[morgo]

When this is closed, remember to update the chunker docs which call this out since #570

[morgo]

For the cases that we've seen (and which we expect this to fix) the problem always manifests as deadlocks. This is the main issue we expect to fix by having keyAbove/keyBelow working.

The deadlocks possibly also become more likely because most of the case use a varchar PK, and thus the deltaQueue for changes and not a map. This is being tracked separately in #475 -- I think it can be explained in that deltaMap has some hot rows protection by de-duplicating, and deltaQueue does not have any.

Put another way: deltaQueue + missing keyAbove/keyBelow is particularly prone to hot row issues. In this issue we are proposing to fix 1 side of the deadlock, which should be enough. If #475 is non viable, there is also #480 that we can look at next if needed.