[collectd 6] Add an option to configure resource attributes.

[octo]

ChangeLog: daemon: The new "Resource" option allows users to configure resource attributes manually if desired.

[eero-t]

Code changes look fine. But I think documentation needs some improvements:

• Name is too generic, so some short explanation how this "Resource" differs from all the other "Resource" items already mentioned in the document, would be good
• Text is on the abstract side and IMHO would need some example of how it changes collectd output, so users unfamiliar with OpenTelemetry know what it's about

[eero-t]

There should be also None option for disabling resource attribute labels completely, because resource label names use characters that are not valid for Prometheus metrics, see #4204 and: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels

[eero-t]

There should be also None option for disabling resource attribute labels completely, because resource label names use characters that are not valid for Prometheus metrics, see #4204 and: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels

That issue is now fixed, but need for disabling the resource labels has gone nowhere, because they just duplicate information that Prometheus already adds to the metrics.

EDIT: there may also be other write plugins / things reading them, that resource labels could have broken. None option would be a good workaround / test for such things too.

[octo]

My thinking was that the following config should give you the same effect as a None resource:

<Resource Generic>
  Attribute "service.name" ""
</Resource>

Can you tell me more how the resource attributes and the Prometheus export clash? It's important to me to get this right.

[eero-t]

My thinking was that the following config should give you the same effect as a None resource:

True, and it's a useful feature, but I'm assuming that you're going to be adding more attributes in the future?

None option would be a more user-friendly way of making sure that all extra labels are disabled.

Can you tell me more how the resource attributes and the Prometheus export clash? It's important to me to get this right.

Problem is that extra labels (on every metric) can noticeably increase Prometheus resource usage in larger clusters, so it's important that one can easily disable anything that is not needed, especially when Prometheus already provides corresponding information in (more relevant kubernetes namespace) labels.

EDIT: other telemetry systems ingesting labels are likely to be impacted too, not just Prometheus.

[octo]

My thinking was that the following config should give you the same effect as a None resource:

True, and it's a useful feature, but I'm assuming that you're going to be adding more attributes in the future?

My design idea is this:

• Metrics that already have resource attributes, e.g. because they were received from somewhere else via gRPC or something similar, are simply passed through. No resource attributes are added, removed, or modified.
• Metrics that are passed to plugin_dispatch_metric_family without resource attributes are considered to be "local", i.e. as originating from this collectd instance. The resource attributes associated with this instance of collectd are attached to the metric family.
• Strike a good balance between auto-discovery and user control. Users likely don't want to manually configure the host name. If you're managing a large fleet, having a different config file for each host is quite annoying.
  • Provide presets that users can chose by passing an appropriate argument to the Preset block. At the moment, Generic and Host are the only available presets. Other presets, e.g. for GCE instances, K8s pods, etc. might follow.
  • Allow users to amend or overwrite the defaults, using the Attribute option inside the Resource block.

None option would be a more user-friendly way of making sure that all extra labels are disabled.

I think you're coming from a position where these labels cause problems with the write_prometheus plugin. Once these are properly integrated (I'm working on this in #4213) I think that the existing Generic preset provides more utility for you than a None preset, because they allow you to set the job and instance labels appropriately.

Problem is that extra labels (on every metric) can noticeably increase Prometheus resource usage in larger clusters, so it's important that one can easily disable anything that is not needed, especially when Prometheus already provides corresponding information in (more relevant kubernetes namespace) labels.

Gotcha, thanks for the info! IMHO you have two options already:

1. Use the Generic preset and unset the one resource attribute it installs if required, as outlined above.
2. Use metric relabeling in Prometheus to discard unwanted labels.

I'm sure collectd will gain additional features in the future, but we have to start somewhere, and this seems like quite a good position to be in for a first step.

[eero-t]

Ok, seems reasonable, especially considering #4213.