Support nested OTEL attributes for Clickhouse exporter#313
Conversation
|
I pulled this into the Lambda Forwarder and verified it's support on some nested CloudTrail logs. I'm considering making a depth of 5 default for the forwarder. |
|
Confirmed in previous PR that this works for the nested GenAI attributes as well. |
rjenkins
left a comment
rjenkins
left a comment
There was a problem hiding this comment.
Left a question about one of the depth checks, but otherwise looks good.
|
|
||
| Some(Value::ArrayValue(a)) => { | ||
| let within_depth = match max_depth { | ||
| None => depth == 0, |
There was a problem hiding this comment.
Why is this check for max depth different than the one above?
There was a problem hiding this comment.
There's a comment earlier on the previous method, but it's a combination of the existing key/value handling that is done with cheaper key flattening (in transformer) and the need to maintain the flat simple value arrays. In this "flat mode" (depth=0) we maintain arrays of simple values at the top, but don't allow complex types. Key/values would be flattened earlier in the transformer, so the check is less important.
3 participants
This is a version based on the PR #293 from @brightsparc that adds support for nested OTEL attributes when using the Clickhouse exporter with JSON attribute type. The depth of nesting is controlled by the
--clickhouse-exporter-nested-kv-max-depthparameter, which defaults to 3. We have a limit to prevent possible DoS from infinitely recursing too deep. In most cases, 3 should be plenty, but some GenAI attributes may require more depth.A chunk of this is a test added that can be run locally against CH, but is not included by default. This helped Claude implement and verify the encoding change.