Skip to content

Commit

Permalink
[chore] Adopt for the kubelet cpu utilization metrics deprecation
Browse files Browse the repository at this point in the history
The following metrics emitted by kubeletstats receiver were deprecated:
- k8s.node.cpu.utilization
- k8s.pod.cpu.utilization
- container.cpu.utilization

See open-telemetry/opentelemetry-collector-contrib#25901 for more details.

Those metrics are already excluded by default in the signalfx exporter. So if we don't do anything, user will see the following confusing warnings:
```
2024-02-06T19:21:44.773Z	warn	metadata/generated_metrics.go:2894	[WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.773Z	warn	metadata/generated_metrics.go:2897	[WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.773Z	warn	metadata/generated_metrics.go:2900	[WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.774Z	warn	metadata/generated_metrics.go:2894	[WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.774Z	warn	metadata/generated_metrics.go:2897	[WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.774Z	warn	metadata/generated_metrics.go:2900	[WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.774Z	warn	metadata/generated_metrics.go:2894	[WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.774Z	warn	metadata/generated_metrics.go:2897	[WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.774Z	warn	metadata/generated_metrics.go:2900	[WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.774Z	warn	metadata/generated_metrics.go:2894	[WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.774Z	warn	metadata/generated_metrics.go:2897	[WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
2024-02-06T19:21:44.774Z	warn	metadata/generated_metrics.go:2900	[WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead.	{"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"}
```

Potentially we couldn't just explicitly disable them in the default k8s configuration on the receiver side. The problem is that the exclusion on the signalfx exporter side potentially can be disabled by users. For those users, we actually want to show the warnings.

So we need to add another converter until the metric is disabled on the kubelet receiver side by default. The converter looks at the signalfx exporter in user's config, and if metrics are not explicitly enabled on the exporter side, it disables them on the kubelet receiver side to supress the warnings.

The converted required to copy `dpfilters` package from signal exporter. This will be removed along with the converter in a few releases.
  • Loading branch information
dmitryax committed Feb 7, 2024
1 parent c1ecc0a commit ec3972a
Show file tree
Hide file tree
Showing 25 changed files with 1,704 additions and 2 deletions.
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ require (
github.com/go-sql-driver/mysql v1.7.1 // indirect
github.com/go-stack/stack v1.8.1 // indirect
github.com/go-test/deep v1.1.0 // indirect
github.com/gobwas/glob v0.2.4-0.20181002190808-e7a84e9525fe // indirect
github.com/gobwas/glob v0.2.4-0.20181002190808-e7a84e9525fe
github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 // indirect
github.com/gogo/googleapis v1.4.1 // indirect
github.com/golang-jwt/jwt v3.2.2+incompatible // indirect
Expand Down Expand Up @@ -466,7 +466,7 @@ require (
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.21 // indirect
github.com/shirou/gopsutil/v3 v3.24.1 // indirect
github.com/signalfx/com_signalfx_metrics_protobuf v0.0.3 // indirect
github.com/signalfx/com_signalfx_metrics_protobuf v0.0.3
github.com/signalfx/defaults v1.2.2-0.20180531161417-70562fe60657 // indirect
github.com/signalfx/gohistogram v0.0.0-20160107210732-1ccfd2ff5083 // indirect
github.com/signalfx/ingest-protocols v0.2.0 // indirect
Expand Down
145 changes: 145 additions & 0 deletions internal/configconverter/disable_kubelet_utilization_metrics.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
// Copyright Splunk, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package configconverter

import (
"context"
"fmt"
"regexp"

sfxpb "github.com/signalfx/com_signalfx_metrics_protobuf/model"
"go.opentelemetry.io/collector/confmap"

"github.com/signalfx/splunk-otel-collector/internal/configconverter/dpfilters"
)

// Metrics deprecated by kubeletstats receiver that need to be disabled if not explicitly included in signalfx exporter.
const (
k8sNodeCPUUtilization = "k8s.node.cpu.utilization"
k8sPodCPUUtilization = "k8s.pod.cpu.utilization"
containerCPUUtilization = "container.cpu.utilization"
)

// signalfxExporterConfig is the configuration for the signalfx exporter that contains the metrics filters.
type signalfxExporterConfig struct {
ExcludeMetrics []dpfilters.MetricFilter `mapstructure:"exclude_metrics"`
IncludeMetrics []dpfilters.MetricFilter `mapstructure:"include_metrics"`
}

// DisableKubeletUtilizationMetrics is a MapConverter that disables the following deprecated metrics:
// - `k8s.node.cpu.utilization`
// - `k8s.pod.cpu.utilization`
// - `container.cpu.utilization`
// The converter disables the metrics at the receiver level to avoid showing users a warning message because
// they are excluded in signalfx exporter by default.
// We don't disable them in case if users explicitly include them in signalfx exporter.
type DisableKubeletUtilizationMetrics struct{}

func (DisableKubeletUtilizationMetrics) Convert(_ context.Context, cfgMap *confmap.Conf) error {
if cfgMap == nil {
return fmt.Errorf("cannot DisableKubeletUtilizationMetrics on nil *confmap.Conf")
}

receivers, err := cfgMap.Sub("receivers")
if err != nil {
return nil // Ignore invalid config. Rely on the config validation to catch this.
}
kubeletReceiverConfigs := map[string]map[string]any{}
for receiverName, receiverCfg := range receivers.ToStringMap() {
if regexp.MustCompile("kubeletstats(/\\w+)?").MatchString(receiverName) {
if v, ok := receiverCfg.(map[string]any); ok {
kubeletReceiverConfigs[receiverName] = v
}
}
}

exporters, err := cfgMap.Sub("exporters")
if err != nil {
return nil // Ignore invalid config. Rely on the config validation to catch this.
}
sfxExporterConfigs := map[string]map[string]any{}
for exporterName, exporterCfg := range exporters.ToStringMap() {
if regexp.MustCompile("signalfx(/\\w+)?").MatchString(exporterName) {
if v, ok := exporterCfg.(map[string]any); ok {
sfxExporterConfigs[exporterName] = v
}
}
}

// If there is no signalfx exporter or kubeletstats receiver, there is nothing to do.
if len(kubeletReceiverConfigs) == 0 || len(sfxExporterConfigs) == 0 {
return nil
}

disableMetrics := map[string]bool{
k8sNodeCPUUtilization: true,
k8sPodCPUUtilization: true,
containerCPUUtilization: true,
}

// Check if the metrics are explicitly included in signalfx exporter.
// If they are not included, we will disable them in kubeletstats receiver.
for _, cm := range sfxExporterConfigs {
cfg := signalfxExporterConfig{}
err = confmap.NewFromStringMap(cm).Unmarshal(&cfg, confmap.WithIgnoreUnused())
if err != nil {
return nil // Ignore invalid config. Rely on the config validation to catch this.
}
if len(cfg.ExcludeMetrics) == 0 {
// Apply default excluded metrics if not explicitly set.
// https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/f2d8efe507083b0f38b6567f8dba3f37053bfa86/exporter/signalfxexporter/internal/translation/default_metrics.go#L133
cfg.ExcludeMetrics = []dpfilters.MetricFilter{
{MetricNames: []string{"/^(?i:(container)|(k8s\\.node)|(k8s\\.pod))\\.cpu\\.utilization$/"}},
}
}

filter, err := dpfilters.NewFilterSet(cfg.ExcludeMetrics, cfg.IncludeMetrics)
if err != nil {
return nil // Ignore invalid config. Rely on the config validation to catch this.
}
for metricName := range disableMetrics {
if !filter.Matches(&sfxpb.DataPoint{Metric: metricName}) {
disableMetrics[metricName] = false
}
}
}

// Disable the metrics in kubeletstats receiver.
for receiverName, cfg := range kubeletReceiverConfigs {
metricsCfg := map[string]any{}
if cfg["metrics"] != nil {
if v, ok := cfg["metrics"].(map[string]any); ok {
metricsCfg = v
}
}
if _, ok := metricsCfg[k8sNodeCPUUtilization]; !ok && disableMetrics[k8sNodeCPUUtilization] {
metricsCfg[k8sNodeCPUUtilization] = map[string]any{"enabled": false}
}
if _, ok := metricsCfg[k8sPodCPUUtilization]; !ok && disableMetrics[k8sPodCPUUtilization] {
metricsCfg[k8sPodCPUUtilization] = map[string]any{"enabled": false}
}
if _, ok := metricsCfg[containerCPUUtilization]; !ok && disableMetrics[containerCPUUtilization] {
metricsCfg[containerCPUUtilization] = map[string]any{"enabled": false}
}
metricsCfgKey := fmt.Sprintf("receivers::%s::metrics", receiverName)
if len(metricsCfg) > 0 {
if err = cfgMap.Merge(confmap.NewFromStringMap(map[string]any{metricsCfgKey: metricsCfg})); err != nil {
return err
}
}
}

return nil
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
// Copyright Splunk, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package configconverter

import (
"context"
"testing"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.opentelemetry.io/collector/confmap/confmaptest"
)

func TestDisableKubeletUtilizationMetrics(t *testing.T) {
tests := []struct {
name string
input string
wantOutput string
}{
{
name: "no_kubeletstats_receiver",
input: "testdata/disable_kubelet_utilization_metrics/no_kubeletstats_receiver.yaml",
wantOutput: "testdata/disable_kubelet_utilization_metrics/no_kubeletstats_receiver.yaml",
},
{
name: "no_signalfx_exporter",
input: "testdata/disable_kubelet_utilization_metrics/no_signalfx_exporter.yaml",
wantOutput: "testdata/disable_kubelet_utilization_metrics/no_signalfx_exporter.yaml",
},
{
name: "disable_all_metrics",
input: "testdata/disable_kubelet_utilization_metrics/disable_all_metrics_input.yaml",
wantOutput: "testdata/disable_kubelet_utilization_metrics/disable_all_metrics_output.yaml",
},
{
name: "do_not_change_enabled_metrics",
input: "testdata/disable_kubelet_utilization_metrics/do_not_change_enabled_metrics_input.yaml",
wantOutput: "testdata/disable_kubelet_utilization_metrics/do_not_change_enabled_metrics_output.yaml",
},
{
name: "all_metrics_included_in_signalfx_exporter",
input: "testdata/disable_kubelet_utilization_metrics/all_metrics_included_in_signalfx_exporter.yaml",
wantOutput: "testdata/disable_kubelet_utilization_metrics/all_metrics_included_in_signalfx_exporter.yaml",
},
{
name: "partially_excluded_in_signalfx_exporter",
input: "testdata/disable_kubelet_utilization_metrics/partially_excluded_in_signalfx_exporter_input.yaml",
wantOutput: "testdata/disable_kubelet_utilization_metrics/partially_excluded_in_signalfx_exporter_output.yaml",
},
{
name: "partially_included_in_signalfx_exporter",
input: "testdata/disable_kubelet_utilization_metrics/partially_included_in_signalfx_exporter_input.yaml",
wantOutput: "testdata/disable_kubelet_utilization_metrics/partially_included_in_signalfx_exporter_output.yaml",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
expectedCfgMap, err := confmaptest.LoadConf(tt.wantOutput)
require.NoError(t, err)
require.NotNil(t, expectedCfgMap)

cfgMap, err := confmaptest.LoadConf(tt.input)
require.NoError(t, err)
require.NotNil(t, cfgMap)

err = DisableKubeletUtilizationMetrics{}.Convert(context.Background(), cfgMap)
require.NoError(t, err)

assert.Equal(t, expectedCfgMap, cfgMap)
})
}

}
56 changes: 56 additions & 0 deletions internal/configconverter/dpfilters/datapoint.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0
// Temporary copied from the SignalFx exporter to be used in DisableKubeletUtilizationMetrics converter.

package dpfilters // import "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/signalfxexporter/internal/translation/dpfilters"

import (
"errors"

sfxpb "github.com/signalfx/com_signalfx_metrics_protobuf/model"
)

type dataPointFilter struct {
metricFilter *StringFilter
dimensionsFilter *dimensionsFilter
}

// newDataPointFilter returns a new dataPointFilter filter with the given configuration.
func newDataPointFilter(metricNames []string, dimSet map[string][]string) (*dataPointFilter, error) {
var metricFilter *StringFilter
if len(metricNames) > 0 {
var err error
metricFilter, err = NewStringFilter(metricNames)
if err != nil {
return nil, err
}
}

var dimensionsFilter *dimensionsFilter
if len(dimSet) > 0 {
var err error
dimensionsFilter, err = newDimensionsFilter(dimSet)
if err != nil {
return nil, err
}
}

if metricFilter == nil && dimensionsFilter == nil {
return nil, errors.New("metric filter must have at least one metric or dimension defined on it")
}

return &dataPointFilter{
metricFilter: metricFilter,
dimensionsFilter: dimensionsFilter,
}, nil
}

// Matches tests a datapoint to see whether it is excluded by this
func (f *dataPointFilter) Matches(dp *sfxpb.DataPoint) bool {
metricNameMatched := f.metricFilter == nil || f.metricFilter.Matches(dp.Metric)
if metricNameMatched {
return f.dimensionsFilter == nil || f.dimensionsFilter.Matches(dp.Dimensions)
}
return false

}
64 changes: 64 additions & 0 deletions internal/configconverter/dpfilters/dimensions.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0
// Temporary copied from the SignalFx exporter to be used in DisableKubeletUtilizationMetrics converter.

package dpfilters // import "github.com/open-telemetry/opentelemetry-collector-contrib/exporter/signalfxexporter/internal/translation/dpfilters"

import (
"errors"

sfxpb "github.com/signalfx/com_signalfx_metrics_protobuf/model"
)

type dimensionsFilter struct {
filterMap map[string]*StringFilter
}

// newDimensionsFilter returns a filter that matches against a
// sfxpb.Dimension slice. The filter will return false if there's
// at least one dimension in the slice that fails to match. In case`
// there are no filters for any of the dimension keys in the slice,
// the filter will return false.
func newDimensionsFilter(m map[string][]string) (*dimensionsFilter, error) {
filterMap := map[string]*StringFilter{}
for k := range m {
if len(m[k]) == 0 {
return nil, errors.New("string map value in filter cannot be empty")
}

var err error
filterMap[k], err = NewStringFilter(m[k])
if err != nil {
return nil, err
}
}

return &dimensionsFilter{
filterMap: filterMap,
}, nil
}

func (f *dimensionsFilter) Matches(dimensions []*sfxpb.Dimension) bool {
if len(dimensions) == 0 {
return false
}

var atLeastOneMatchedDimension bool
for _, dim := range dimensions {
dimF := f.filterMap[dim.Key]
// Skip if there are no filters associated with current dimension key.
if dimF == nil {
continue
}

if !dimF.Matches(dim.Value) {
return false
}

if !atLeastOneMatchedDimension {
atLeastOneMatchedDimension = true
}
}

return atLeastOneMatchedDimension
}
Loading

0 comments on commit ec3972a

Please sign in to comment.