Do not fail RisingWave checkpoint if upstream source is unavailable

[hzxa21]

Is your feature request related to a problem? Please describe.

Suppose user created 3 tables:

• Streaming table T1 from kafka source
• Streaming table T2 from direct CDC of PG database1
• Streaming table T3 from direct CDC of PG database2
• Batch table T4

When the upstream PG database1 is not available, RisingWave cluster will enter a recovery loop until the upstream DB is back online causing 1) data ingestion to be stopped in T1, T3 and T4; 2) batch queries on T1, T3 and T4 will constantly get interrupted by recovery.

This issue is found during debugging a cluster failure reported by one of our user. It takes us some time to realize that RisingWave is working fine but the upstream source is down.

Describe the solution you'd like

We can pause the source from polling data from an unavailable source but still allow barrier injection from source. In this case, RisingWave checkpoint is unaffected when there is an issue in an external system.

Data will still be lagging for the unavailable source but this is unavoidable.

Describe alternatives you've considered

No response

Additional context

No response

[hzxa21]

cc @fuyufjh @tabVersion @BugenZhao Any opinion?

[BugenZhao]

Seems we had a similar discussion before -> #7192 (comment)

[tabVersion]

had a discussion with @fuyufjh yesterday, there will be a doc about the expected behavior for the source part when the upstream is unavailable. stay tuned!

[hzxa21]

had a discussion with @fuyufjh yesterday, there will be a doc about the expected behavior for the source part when the upstream is unavailable. stay tuned!

Thanks for the heads-up. Look forward to it! ❤️

[fuyufjh]

Completely agree. Just discussed exactly the same issue with @tabVersion yesterday. We should start working on this soon.

[fuyufjh]

We can pause the source from polling data from an unavailable source but still allow barrier injection from source. In this case, RisingWave checkpoint is unaffected when there is an issue in an external system.

Yeah. I proposed to let it retry forever when upstream source is unavailable. Otherwise, as you said, let the barrier injection run normally.

Besides, the Sink side need a similar mechanism but the implementation would be more difficult, which depends on the Sink-MV decoupling aka. Log Store, I think.

[fuyufjh]

While for testing, @cyliu0 may take a look as well