You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The upstream producer writes a batch of data, then waits. RisingWave ingests it, updates its materialized views, and sinks the results to S3.
We want to know when RisingWave has finished processing the batch and all resulting files are committed and visible in the S3 sink. This would let us:
Apply backpressure: Don’t start producing the next batch while RisingWave is still processing the current one.
Give downstream consumers a consistent view: Once the sink has finished and the producer is idle, the downstream consumer can read the complete batch without more files arriving during its read.
Can source and sink progress be tracked through epochs? Is there a way to know the epoch through which the S3 source has been consumed and the epoch through which the decoupled S3 sink has committed?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Hi everyone,
Our pipeline is:
The upstream producer writes a batch of data, then waits. RisingWave ingests it, updates its materialized views, and sinks the results to S3.
We want to know when RisingWave has finished processing the batch and all resulting files are committed and visible in the S3 sink. This would let us:
Can source and sink progress be tracked through epochs? Is there a way to know the epoch through which the S3 source has been consumed and the epoch through which the decoupled S3 sink has committed?
Thanks in advance for any guidance!
All reactions