changefeedccl: cloudstorage sink file naming may not always be lexicographically ordered

[andyyang890]

Per #155015 (comment), the cloudstorage sink assumes that every time it's flushed, the changefeed's frontier timestamp has advanced. This isn't always true (e.g. when the feed is lagging and the lag checkpoint code path is hit:

cockroach/pkg/ccl/changefeedccl/changefeed_processors.go

Lines 959 to 967 in 94b54b7

	checkpointSpans := (ca.frontier.InBackfill(resolved) || ca.frontier.HasLaggingSpans(sv)) &&
	canCheckpointSpans(sv, ca.lastSpanFlush)
	if checkpointSpans {
	defer func() {
	ca.lastSpanFlush = timeutil.Now()
	}()
	return ca.flushFrontier(ctx)
	}

) and may have ramifications for making sure that new cloudstorage sink files have names that sort lexicographically after all previously written files.

Jira issue: CRDB-55306

[blathers-crl]

cc @cockroachdb/cdc

[blathers-crl]

Hi @andyyang890, please add a branch-* label to identify the earliest affected branch for this C-bug

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[asg0451]

looks like it could potentially also happen under backpressure:

cockroach/pkg/ccl/changefeedccl/changefeed_processors.go

Line 886 in 2336d7b

return ca.flushBufferedEvents(ca.Ctx())

from

cockroach/pkg/ccl/changefeedccl/kvevent/blocking_buffer.go

Line 158 in 2336d7b

e = Event{et: TypeFlush}