High cardinality on server request observations due to `graphql.operation`

[denniseffing]

Hi,

the documentation states that the graphql.operation tag for server request observations is derived from the operation’s custom name (if provided) or the default operation name, and that it is intended to be low cardinality.

In practice, because clients control the custom operation name, we’re seeing high cardinality in our metrics. For example, in our environment graphql.operation currently has over 1,000 distinct values (for just one service/subgraph).

Could you consider changing the default behavior of graphql.operation to ensure lower cardinality by default?

[bclozel]

Thanks for reporting this @denniseffing .

I think the most sensible fix for this is to change the instrumentation. graphql.operation (low cardinality) becomes graphql.operation.name (high cardinality) and we add a graphql.operation.type (low cardinality).

I think we get around any breakages in observability systems by configuring a custom ObservationFilter that removes/adds key values from observations. I can provide sample applications, depending on the expected behavior and in which applications this filter should be applied. I'm thinking that a custom ObservationFilter that renames graphql.observation (low) to graphql.operation.name (high) could be a good choice for apps running with spring-graphql < 1.4.3. If you're looking to keep the existing behavior but limit the number of tag values, MeterFilter provides factory methods for filtering such cases.

I'm pushing a change that should be available shortly in the 1.4.3-SNAPSHOT. Let me know if you can test the fix. I'll work on writing something in the release notes for this, any feedback from your experience testing the SNAPSHOT version should help.

[denniseffing]

Hi @bclozel,

thank you so much for the incredibly fast fix! I tested v1.4.3-SNAPSHOT and results are looking promising on our side.