docs: add design doc for consistent error handling and surfacing connector failures

[aljoscha]

rendered

Currently, Materialize reacts to errors happening at different stages of a connectors lifecycle in different ways. For example, an error happening during purification/creation of a connector will be reported back to the user immediately while errors that happen during runtime are only logged. We should agree on what the behavior should be and fix it where needed.

Additionally, I propose to formally add the concept of a lifecycle for connectors, which will allow the system and users to determine the status of a connector. Right now, this would only be possible by looking through log files for error messages and manually associating them with connectors.

This addresses #7115

[aljoscha]

cc @petrosagg: This should be mostly orthogonal to your work on the Source Trait. But there will have to be a mechanism to get the timeout configs to the relevant pieces.

[benesch]

This has always irked me about librdkafka. I think we'd be well served by finding some way to surface a transient "degraded" state while librdkafka is spewing errors.

[aljoscha]

👍 Once we have the machinery in place to track connector state this should be doable. If we can coax it out of librdkafka...

[benesch]

Are these user-settable globals? Or just a default bundle of Retry options that get plumbed around but are hardcoded into the binary? I think I like the second thing better! Retry settings to me feel like something that the user typically doesn't have much useful insight into, at least not at the global level. If you do have an opinion on retry configuration it's usually in the context of one specific retry loop that is misconfigured.

[aljoscha]

I initially intended it to be user-settable globals, but then thought along the same lines as you and added some text under Alternatives that takes that back. I'll remove this from here.

By now, I'm not even sure we need global Retry options except maybe for restarting whole connectors/views.

[aljoscha]

I tweaked the title to make the intent clearer, moved the section about user-settable global retry settings to Alternatives, and added a task about changing mz_views to add a status column, along with a new mz_view_errors.

[philip-stoev]

Listing the keys in an S3 bucket is also a one-time task that can fail.

[aljoscha]

It is, but this is only started after the source was successfully created, I believe. I've looked at this one:

materialize/src/dataflow/src/source/s3.rs

Line 219 in 480aa30

async fn scan_bucket_task(

I'm adding these examples to the doc as well, so thanks!

[philip-stoev]

also consuming the postgres replication stream

[aljoscha]

Adding that as an example 👌

[philip-stoev]

reading the initial snapshot from postgres

[aljoscha]

I think this also starts only after the source was created and added to the catalog, in the dataflow layer.

Or did you observe that a missing Postgres will prevent Materialize from starting?

[philip-stoev]

failures in the source should also cause selecting from the source to start erroring out, rather than continuing to return stale or no data. This should be in addition to sufracing that same error in a mz_ table.

[aljoscha]

Do you mean fatal errors, like a malformed message that we can't parse. Or transient errors, like a Kafka broker not being reachable. For the former, we should already prevent querying, but you're right that we don't do anything about the latter right now.

[philip-stoev]

Thank you for taking up on this. I believe a consistent implementation will save our customers and our own customer-facing peope a LOT of grief.

[benesch]

I gave this a very thorough review and, uh... I straight up don't have any comments. Not one! This is fantastic work, @aljoscha, and something we'd been meaning to get to for so long. Really excited that you're going to make it happen.

[aljoscha]

Thanks! 🎉

Also, you all did have helpful comments earlier, so thanks for those as well!