Additional kubelet_volume_stats_*_bytes metrics not forwarded

[acabrele]

Prior to v4 (prometheus based) we used to modify the regex in kube-prometheus-stack.kubelet.serviceMonitor.metricRelabelings and kube-prometheus-stack.prometheus.prometheusSpec.remoteWrite so that sumologic would receive kubelet_volume_stats_*_bytes metrics.

In v4 we can do the same by reverting to v3 config as documented here.

However it would be nice to get the same volume metrics working without having to do this. In the guidance here I believe this falls under category 1 which suggests that "You can safely delete the added remote write definition. No further action is required.". But we do not see the added metrics.

Category 2 suggests a filter processor may be needed, but I have had no luck in getting one to work.

I also note that the docs/scraped_metrics.m suggests a whole host of kubelet stats will be forwarded but there are actually only 4 available now.

What is the new trick to make this work?

[swiatekm-sumo]

The source of this problem is that kubelet and cAdvisor metrics are no longer configured via a ServiceMonitor with otel. This ServiceMonitor only works with prometheus-operator present, and in its absence we use a custom scrape config, which until now was only customizable by overriding the otel configuration directly. #3528 will make that much more convenient.

You're right that this should be made explicit in the migration guide.