Drop container=pod label when scraping container network metrics

[pmalek-sumo]

Description

Fill in your description here.

Testing performed

• ci/build.sh
• Redeploy fluentd and fluentd-events pods
• Confirm events, logs, and metrics are coming in

[frankreno]

One change as discussed

[frankreno]

As discussed, we want to use label drop to drop the label container=POD from the container_network_receive_bytes_total and container_network_transmit_bytes_total metrics. Not drop the metrics themselves.

[pmalek-sumo]

Updated, should be good now

[frankreno]

Did you test this locally and confirm that we still get the metrics in Sumo but they no longer have the container=POD ? The reason I ask is I looked at the Prometheus relabel docs - https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

This indicates that when using labelDrop action, it matches against a regex property. Does not explicitly state it works for sourceLabels so just want to make sure.

[pmalek-sumo]

So I based that on this https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#relabelconfig specifically on:

sourceLabels - The source labels select values from existing labels. Their content is concatenated using the configured separator and matched against the configured regular expression for the replace, keep, and drop actions.

at the same time:

regex - Regular expression against which the extracted value is matched. Default is '(.*)'

And as I look at the metrics coming from my KOPS cluster it indeed remove the cluster label so I cannot see it anymore (unless I make the timerange wide enough to cover my testing yesterday where I haven't managed to configure the relabel_config yet)

[pmalek-sumo]

I obviously meant container label, not cluster label.

[frankreno]

LGTM