metric sink: skip rows with a null label value (SQL-645) - #38375
Conversation
4503157 to
4346c1b
Compare
Problem:
A `map[text=>text]` has no per-value nullability, so a metric sink's
`labels` map can hold nulls. Such a row hit `extract_row`, which unwrapped the
value as a string and panicked the worker. That takes down clusterd, and the
sink re-renders over the same persisted row on restart, crash-looping the whole
cluster.
Separately, `drop_optimizer_notices` handled only `Index` and
`MaterializedView`, so a dropped sink's notices were never retracted.
Solution:
Skip a row whose label set is not representable, counting it in
`mz_compute_metric_sink_skipped`. A null value has nothing to encode, and an
empty string is not a stand-in either, since Prometheus reads it as absent and
would fold `{a => ''}` into `{}`.
Retract a dropped sink's notices through a new `dataflow_metainfo_mut`, the
mutable twin of the existing `dataflow_metainfo` getter, shared by both drop
sites.
Testing:
- unit tests
- a sqllogictest over a filtered, indexed view that exercises both the
panic and the notice retraction on `DROP METRIC SINK`.
Closes: SQL-645
4346c1b to
6edb149
Compare
| .map(|(k, v)| (k, v.unwrap_str())) | ||
| .map(|(k, v)| (k, (!v.is_null()).then(|| v.unwrap_str()))) |
There was a problem hiding this comment.
It appears to me the alternative to dropping rows with null label values would be to just discard this label? This seems to be more consistent to me than dropping the whole row, but I also think it's a policy question, and both ways are acceptable.
There was a problem hiding this comment.
You are right it is a policy decision. The reason why I opted for this policy is that if we collapse the label then we could have two rows that are the same. And then we will have a collision.
E.g., {foo='a', bar=NULL} and {foo='a'} would then collide.
There was a problem hiding this comment.
Makes sense, and we can always revisit if needed. Values are string, so another option would be canonicalize as "NULL", which might be equally surprising :)
Problem:
A
map[text=>text]has no per-value nullability, so a metric sink'slabelsmap can hold nulls. Such a row hitextract_row, which unwrapped thevalue as a string and panicked the worker. That takes down clusterd, and the
sink re-renders over the same persisted row on restart, crash-looping the whole
cluster.
Separately,
drop_optimizer_noticeshandled onlyIndexandMaterializedView, so a dropped sink's notices were never retracted.Solution:
Skip a row whose label set is not representable, counting it in
mz_compute_metric_sink_skipped. A null value has nothing to encode, and anempty string is not a stand-in either, since Prometheus reads it as absent and
would fold
{a => ''}into{}.Retract a dropped sink's notices through a new
dataflow_metainfo_mut, themutable twin of the existing
dataflow_metainfogetter, shared by both dropsites.
Testing:
panic and the notice retraction on
DROP METRIC SINK.Closes: SQL-645