changefeedccl: document high-water-mark semantics in REGIONAL BY ROW tables during network partition

[stevekuznetsov]

A client uses change-feeds to keep an up-to-date client-side cache of a table. This client requires observing events in a global order, so they wait for high-water-marks to re-order previously-received events client-side before consuming them. When this client applies this approach to a REGIONAL BY ROW table, what is the expected semantic of high-water-marks delivered in the change-feed during a network partition between regions that the table has rows for?

If the client wants to keep at the very least a strongly consistent view of local data in the table, would it suffice to issue a geo-filtered change-feed, and would the high-water marks for those filtered change-feeds be delivered when the change-feed in question has seen all relevant events up to the mark, or are the high-water-marks still table-global?

Jira issue: CRDB-22219

Epic CRDB-21737

[blathers-crl]

Hello, I am Blathers. I am here to help you get the issue triaged.

It looks like you have not filled out the issue in the format of any of our templates. To best assist you, we advise you to use one of these templates.

I was unable to automatically find someone to ping.

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[blathers-crl]

cc @cockroachdb/cdc

[amruss]

@kathancox I think this is a docs issue, we should possibly add a section on network paritions and how changefeeds respon. TLDR all the nodes will keep emitting and changefeed guarantees still hold

[stevekuznetsov]

@amruss so in a REGIONAL BY ROW table, if one region gets partitioned away, the entire table is unavailable?

[kathancox]

@amruss I have created a docs issue to update the docs around this: https://cockroachlabs.atlassian.net/browse/DOC-6492

[amruss]

@stevekuznetsov there is some complexity here depending on where the coordinator node is. The coordinator will not be able to see some of the aggregator nodes & vice versa. I don't think we've tested in this exact scenario so I'm not 100% sure whether we will fail the changefeed or not. I think this issue should be to test this scenario and document it.

[stevekuznetsov]

@amruss sounds good - it would be awesome as well if we could differentiate in the doc between design constraints for the system and emergent behavior from e.g. the query planner or your comment about the coordinator. The broader use-case is, in a REGIONAL BY ROW table, can I ask for a changefeed for "only" data in my own region, and continue to make forward progress (high-water-marks) if a partition occurs? I would assume that the goal-state here is yes, since this looks to be a selling point for REGIONAL BY ROW topology in partition scenarios?

[miretskiy]

Highwater mark is tracked per range -- so, I assume under partitioning scenario (and assuming changefeed
remains running, which may not be the case actually since changefeed might observe
transient errors because of various rpc errors that it will see), but anyway, if the region falls out, then we stop
updating high water mark -- and once it's back we'll catchup.

It does sound to me that in such a case, running geo filtered changefeed should work.
But currently it probably would not work very well since we do not have
a mechanism to pin changefeed coordinators/aggregators to a particular region

[p0lyn0mial]

@miretskiy I'm not sure if I understood your reply correctly, please confirm.

I think you are saying that in a multi-region table (possibly with many ranges) there will only be a single coordinator and CRDB doesn't allow for specifying in which region the coordinator runs.

[p0lyn0mial]

In addition to that, does it mean that running a changefeed over a multi-region table and losing a region stops propagating the high-water mark altogether?

I think this is true, at least based on some info I have found in the docs: Resolved timestamp events do not emit until all ranges in the changefeed have progressed to a specific point in time.

I'd appreciate if somebody could confirm. Thanks!

[stevekuznetsov]

@p0lyn0mial the caveat I understood is that your change-feed can be filtered, and if the filter pins a specific geo, you get resolved timestamps in that filtered view.

[sttts]

you get resolved timestamps in that filtered view.

@miretskiy's comment sounds like that this is not implemented today in a sensible way (but could) because there is no changefeed aggregator geo pinning.

[stevekuznetsov]

Yep, that was the gist of my request for:

... differentiate in the doc between design constraints for the system and emergent behavior from e.g. the query planner ...