changefeedccl: re-enable periodic aggregator frontier flushes

[andyyang890]

Periodic aggregator frontier flushes (which was introduced in #153758) was reverted in #154982/#155082 due to a test failure (#154325). We should re-enable it once we figure out what the problem was (potentially a cloudstorage sink/mock cloud sink bug). Periodic flushing is necessary for us to be able to update the persisted frontier when the changefeed is lagging (which also makes it a prerequisite for deprecating the lag checkpoint).

Jira issue: CRDB-55236

Epic CRDB-48882

[blathers-crl]

cc @cockroachdb/cdc

[asg0451]

My RCA for the linked issue:

the cloud storage sink sets the timestamp for the next flushed batch of files after cutting the previous ones (

cockroach/pkg/ccl/changefeedccl/sink_cloudstorage.go

Line 723 in 9d0c9c9

)
so the file timestamp is only advanced when we Flush()...
-> there is an assumption that when we Flush, the timestamp has changed. this is not true under the change in #153758

in order to move forward with this we either need to maintain this invariant or change how the cloudstorage sink does its file naming

[andyyang890]

this is not true under the change in #153758

Note that this assumption could be false even before the periodic flushing change through the lag checkpoint code path

[asg0451]

really? that's cool. can you open an issue to investigate that as well @andyyang890

[andyyang890]

really? that's cool. can you open an issue to investigate that as well @andyyang890

Filed #155174