Tracking: Report user errors

[jon-chuang]

Implements: #7824

• Simple report ExprError as compute_error in stream jobs (feat(stream): Report compute_error_count to prometheus #7832)
• Report source errors (parser) (feat(stream): source_error_count reporting to prometheus  #7877)
• Supress errors if unique errors are exceeded. Reset error table every 1 hour. Cluster config for max_unique. Set to U64::MAX for test.
  • Compute (feat(stream): ErrorSuppressor for user compute errors #8132)
  • Stream Source (feat(connector): Refactor metrics, source_info into source_ctx and add ErrorSuppressor for user source errors #8156)
• Warn user for batch source errors (feat(error-reporting): WARN user about batch source errors #8135)
• (Optional?) define e2e test that queries prometheus/the prometheus endpoint directly after triggering some user errors (maybe takes too long). Can be part of longevity testing? @lmatz (test(stream): Test reporting of stream errors in e2e setting  #8037)
  - [ ] Truncate errors if blacklisted (Stream Error Truncation Mechanism #7871)

Orthogonal:

• Improve error messages of user errors so that they are more informative (e.g. show the expression when encountering ExprError - however, this might be quite hard as the Debug fmt may not clear for the nested BoxedExpression used in our executors)
• Incidentally, we could also report errors which kill actor to prometheus, and display in grafana under the same Stream Errors panel (as suggested here: streaming: report actor error #37)

[BugenZhao]

• Clean ExprError to UserError

I hope I understand it right, the ExprError itself should be preserved as part of the interfaces of the expr crate. But for the usages in the stream crate, we can erase the specific variant and represent it as a string.

• Simple report ExprError in stream jobs

Do we report the concrete error message or only the error count? If it's the former, is there a mechanism for persisting (or buffering) the error messages or throttling a bulk of errors (for example, wrong column descriptors lead to failure for every line in the parser)?

[jon-chuang]

Do we report the concrete error message or only the error count?

We will record the concrete error message if it has not been truncated.

We will truncate only if the message is blacklisted (i.e. we do not know if it can have unbounded cardinality) and if the number of unique blacklisted messages recorded exceed some threshold (e.g. 50).

Our hope is to recover the full, informative error message in the happy path.

EDIT: truncation is no longer planned