docs: add design doc for consistent error handling and surfacing connector failures #7584
Conversation
|
cc @petrosagg: This should be mostly orthogonal to your work on the Source Trait. But there will have to be a mechanism to get the timeout configs to the relevant pieces. |
aljoscha
changed the title
| again they will start consuming again. At least that's the case in our setup, | ||
| with split consumer keys. And the consumer will log errors still. | ||
|
|
||
| We can be fine with that, or also try and cover this with a timeout and try to |
There was a problem hiding this comment.
This has always irked me about librdkafka. I think we'd be well served by finding some way to surface a transient "degraded" state while librdkafka is spewing errors.
There was a problem hiding this comment.
👍 Once we have the machinery in place to track connector state this should be doable. If we can coax it out of librdkafka...
|
|
||
| The concrete steps to achieve this, in the order we should address them: | ||
|
|
||
| - Introduce global settings for `timeout`, `num retries`, and potentially |
There was a problem hiding this comment.
Are these user-settable globals? Or just a default bundle of Retry options that get plumbed around but are hardcoded into the binary? I think I like the second thing better! Retry settings to me feel like something that the user typically doesn't have much useful insight into, at least not at the global level. If you do have an opinion on retry configuration it's usually in the context of one specific retry loop that is misconfigured.
There was a problem hiding this comment.
I initially intended it to be user-settable globals, but then thought along the same lines as you and added some text under Alternatives that takes that back. I'll remove this from here.
By now, I'm not even sure we need global Retry options except maybe for restarting whole connectors/views.
|
I tweaked the title to make the intent clearer, moved the section about user-settable global retry settings to Alternatives, and added a task about changing |
| - (coord) reading Kafka consistency topic | ||
| - (coord) publishing schemas | ||
| - (coord) creating topics | ||
| - (coord) creating output file, checking input file |
There was a problem hiding this comment.
Listing the keys in an S3 bucket is also a one-time task that can fail.
There was a problem hiding this comment.
It is, but this is only started after the source was successfully created, I believe. I've looked at this one:
materialize/src/dataflow/src/source/s3.rs
Line 219 in 480aa30
I'm adding these examples to the doc as well, so thanks!
| - Continuously: | ||
| - (coord) metadata loops, for example fetching new Kafka partitions, listening | ||
| on BYO topics, listening on SQS for S3 updates | ||
| - (dataflow) actual data ingest and writing |
There was a problem hiding this comment.
also consuming the postgres replication stream
There was a problem hiding this comment.
Adding that as an example 👌
| - (coord) purification | ||
| - When creating a source/sink or when materialized is restarted: | ||
| - (coord) initial connector setup, for example: | ||
| - (coord) reading Kafka consistency topic |
There was a problem hiding this comment.
reading the initial snapshot from postgres
There was a problem hiding this comment.
I think this also starts only after the source was created and added to the catalog, in the dataflow layer.
Or did you observe that a missing Postgres will prevent Materialize from starting?
| - Failures that occur during a restart with an already filled catalog must not | ||
| bring down Materialize. Instead, errors that occur must be reported to the | ||
| user for each individual connector. | ||
|
|
There was a problem hiding this comment.
failures in the source should also cause selecting from the source to start erroring out, rather than continuing to return stale or no data. This should be in addition to sufracing that same error in a mz_ table.
There was a problem hiding this comment.
Do you mean fatal errors, like a malformed message that we can't parse. Or transient errors, like a Kafka broker not being reachable. For the former, we should already prevent querying, but you're right that we don't do anything about the latter right now.
There was a problem hiding this comment.
I gave this a very thorough review and, uh... I straight up don't have any comments. Not one! This is fantastic work, @aljoscha, and something we'd been meaning to get to for so long. Really excited that you're going to make it happen.
|
Thanks! 🎉 Also, you all did have helpful comments earlier, so thanks for those as well! |
aljoscha
force-pushed
the
doc-error-handling
branch
from
e096cab
to
ea16262
Compare
rendered
Currently, Materialize reacts to errors happening at different stages of a connectors lifecycle in different ways. For example, an error happening during purification/creation of a connector will be reported back to the user immediately while errors that happen during runtime are only logged. We should agree on what the behavior should be and fix it where needed.
Additionally, I propose to formally add the concept of a lifecycle for connectors, which will allow the system and users to determine the status of a connector. Right now, this would only be possible by looking through log files for error messages and manually associating them with connectors.
This addresses #7115