Skip to content

Commit

Permalink
Metrics (3/4)
Browse files Browse the repository at this point in the history
related to #124

In this pull request I implemented the metrics system in k8gb. I added one metric - counter
incrementing the number of successful reconciliation loops. Adding specific metrics is part of the next PR.
Like the logger, the metrics are accessed through a single singleton object and like the logger it is initialized.

```go
var log = logging.Logger()

var m = metrics.Metrics()
```
then anywhere in the code we can use this:
```
// increase the number of successful reconciliations
m.ReconciliationIncrement()
```
The singleton is initialized in the `main()` function using the `metrics.Init(*config)` function.
In case the singleton is not initialized, it will be set to the default value and all metrics will
start with the prefix `k8gb_default` otherwise they will start with the prefix from the env variable `K8GB_NAMESPACE`.

In addition to the singleton, I have changed the original metrics and added tests at the package level.

In addition to the singleton, I significantly refactored the original metrics and added tests at the package level.

The prometheus metrics implementation uses reflection. In short, I read the collectors structure during the runtime:
```go
type collectors struct {
	HealthyRecords *prometheus.GaugeVec
	IngressHostsPerStatus *prometheus.GaugeVec
	ZoneUpdateTotal prometheus.Counter
	ReconciliationTotal prometheus.Counter
}
```
Now I know all the types and names. From this I generate the map <name>: <instance>.
I will then use the map for Register(), Unregister() and Get().

<name> is the name of the metric and is generated from the name of the structure in `collectors`.
ReconciliationTotal => for example: `k8gb_gslb_healthy_records` (k8gb is `K8GB_NAMESPACE`)

`Get()` is only for testing purposes - and is public only because controller_tests use metrics.
I'll consider deleting tests from controller_tests and leaving only terratests and unit_tests at the package level (96% coverage now).
But that will come in the next PR.

Signed-off-by: kuritka <kuritka@gmail.com>
  • Loading branch information
kuritka committed Jul 22, 2021
1 parent f84bfe0 commit 3d19cfa
Show file tree
Hide file tree
Showing 10 changed files with 518 additions and 100 deletions.
10 changes: 6 additions & 4 deletions controllers/gslb_controller.go
Expand Up @@ -21,14 +21,14 @@ import (
"fmt"
"strconv"

"github.com/AbsaOSS/k8gb/controllers/providers/metrics"

str "github.com/AbsaOSS/gopkg/strings"
k8gbv1beta1 "github.com/AbsaOSS/k8gb/api/v1beta1"
"github.com/AbsaOSS/k8gb/controllers/depresolver"
"github.com/AbsaOSS/k8gb/controllers/internal/utils"
"github.com/AbsaOSS/k8gb/controllers/logging"
"github.com/AbsaOSS/k8gb/controllers/providers/dns"
"github.com/AbsaOSS/k8gb/controllers/providers/metrics"

str "github.com/AbsaOSS/gopkg/strings"
corev1 "k8s.io/api/core/v1"
v1beta1 "k8s.io/api/networking/v1beta1"
"k8s.io/apimachinery/pkg/api/errors"
Expand All @@ -50,7 +50,6 @@ type GslbReconciler struct {
Scheme *runtime.Scheme
Config *depresolver.Config
DepResolver *depresolver.DependencyResolver
Metrics *metrics.PrometheusMetrics
DNSProvider dns.Provider
}

Expand All @@ -67,6 +66,8 @@ const (

var log = logging.Logger()

var m = metrics.Metrics()

// +kubebuilder:rbac:groups=k8gb.absa.oss,resources=gslbs,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=k8gb.absa.oss,resources=gslbs/status,verbs=get;update;patch

Expand Down Expand Up @@ -169,6 +170,7 @@ func (r *GslbReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.
// Everything went fine, requeue after some time to catch up
// with external Gslb status
// TODO: potentially enhance with smarter reaction to external Event
m.ReconciliationIncrement()
return result.Requeue()
}

Expand Down
73 changes: 39 additions & 34 deletions controllers/gslb_controller_test.go
Expand Up @@ -27,6 +27,7 @@ import (
"testing"
"time"

str "github.com/AbsaOSS/gopkg/strings"
k8gbv1beta1 "github.com/AbsaOSS/k8gb/api/v1beta1"
"github.com/AbsaOSS/k8gb/controllers/depresolver"
"github.com/AbsaOSS/k8gb/controllers/internal/utils"
Expand All @@ -35,7 +36,6 @@ import (
"github.com/AbsaOSS/k8gb/controllers/providers/dns"
"github.com/AbsaOSS/k8gb/controllers/providers/metrics"

str "github.com/AbsaOSS/gopkg/strings"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/testutil"
"github.com/stretchr/testify/assert"
Expand Down Expand Up @@ -148,13 +148,10 @@ func TestHealthyServiceStatus(t *testing.T) {
func TestIngressHostsPerStatusMetric(t *testing.T) {
// arrange
settings := provideSettings(t, predefinedConfig)
err := settings.reconciler.Metrics.Register()
require.NoError(t, err)
defer settings.reconciler.Metrics.Unregister()
expectedHostsMetricCount := 3
// act
ingressHostsPerStatusMetric := settings.reconciler.Metrics.GetIngressHostsPerStatusMetric()
err = settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
ingressHostsPerStatusMetric := metrics.Metrics().Get("k8gb_gslb_ingress_hosts_per_status").AsGaugeVec()
err := settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
actualHostsMetricCount := testutil.CollectAndCount(ingressHostsPerStatusMetric)
// assert
assert.NoError(t, err, "Failed to get expected gslb")
Expand All @@ -168,17 +165,14 @@ func TestIngressHostsPerStatusMetricReflectionForHealthyStatus(t *testing.T) {
func() {
// arrange
settings := provideSettings(t, predefinedConfig)
defer settings.reconciler.Metrics.Unregister()
err := settings.reconciler.Metrics.Register()
require.NoError(t, err)
serviceName := "frontend-podinfo"
defer deleteHealthyService(t, &settings, serviceName)
expectedHostsMetric := 1.
createHealthyService(t, &settings, serviceName)
reconcileAndUpdateGslb(t, settings)
// act
err = settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
ingressHostsPerStatusMetric := settings.reconciler.Metrics.GetIngressHostsPerStatusMetric()
err := settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
ingressHostsPerStatusMetric := metrics.Metrics().Get("k8gb_gslb_ingress_hosts_per_status").AsGaugeVec()
healthyHosts := ingressHostsPerStatusMetric.With(prometheus.Labels{"namespace": settings.gslb.Namespace,
"name": settings.gslb.Name, "status": metrics.HealthyStatus})
actualHostsMetric := testutil.ToFloat64(healthyHosts)
Expand All @@ -193,13 +187,10 @@ func TestIngressHostsPerStatusMetricReflectionForHealthyStatus(t *testing.T) {
func TestIngressHostsPerStatusMetricReflectionForUnhealthyStatus(t *testing.T) {
// arrange
settings := provideSettings(t, predefinedConfig)
defer settings.reconciler.Metrics.Unregister()
err := settings.reconciler.Metrics.Register()
require.NoError(t, err)
err = settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
err := settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
expectedHostsMetricCount := 0.
// act
ingressHostsPerStatusMetric := settings.reconciler.Metrics.GetIngressHostsPerStatusMetric()
ingressHostsPerStatusMetric := metrics.Metrics().Get("k8gb_gslb_ingress_hosts_per_status").AsGaugeVec()
unhealthyHosts := ingressHostsPerStatusMetric.With(prometheus.Labels{"namespace": settings.gslb.Namespace,
"name": settings.gslb.Name, "status": metrics.UnhealthyStatus})
actualHostsMetricCount := testutil.ToFloat64(unhealthyHosts)
Expand Down Expand Up @@ -227,9 +218,6 @@ func TestIngressHostsPerStatusMetricReflectionForUnhealthyStatus(t *testing.T) {
func TestIngressHostsPerStatusMetricReflectionForNotFoundStatus(t *testing.T) {
// arrange
settings := provideSettings(t, predefinedConfig)
defer settings.reconciler.Metrics.Unregister()
err := settings.reconciler.Metrics.Register()
require.NoError(t, err)
expectedHostsMetricCount := 2.0

serviceName := "unhealthy-app"
Expand All @@ -238,9 +226,9 @@ func TestIngressHostsPerStatusMetricReflectionForNotFoundStatus(t *testing.T) {
deleteUnhealthyService(t, &settings, serviceName)

// act
err = settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
err := settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
require.NoError(t, err, "Failed to get expected gslb")
ingressHostsPerStatusMetric := settings.reconciler.Metrics.GetIngressHostsPerStatusMetric()
ingressHostsPerStatusMetric := metrics.Metrics().Get("k8gb_gslb_ingress_hosts_per_status").AsGaugeVec()
unknownHosts, err := ingressHostsPerStatusMetric.GetMetricWith(
prometheus.Labels{"namespace": settings.gslb.Namespace, "name": settings.gslb.Name, "status": metrics.NotFoundStatus})
require.NoError(t, err, "Failed to get ingress metrics")
Expand All @@ -260,10 +248,7 @@ func TestHealthyRecordMetric(t *testing.T) {
}
serviceName := "frontend-podinfo"
settings := provideSettings(t, predefinedConfig)
defer settings.reconciler.Metrics.Unregister()
err := settings.reconciler.Metrics.Register()
require.NoError(t, err)
err = settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
err := settings.client.Get(context.TODO(), settings.request.NamespacedName, settings.gslb)
require.NoError(t, err, "Failed to get expected gslb")
defer deleteHealthyService(t, &settings, serviceName)
createHealthyService(t, &settings, serviceName)
Expand All @@ -274,7 +259,7 @@ func TestHealthyRecordMetric(t *testing.T) {
require.NoError(t, err, "Failed to update gslb Ingress Address")
reconcileAndUpdateGslb(t, settings)
// act
healthyRecordsMetric := settings.reconciler.Metrics.GetHealthyRecordsMetric()
healthyRecordsMetric := metrics.Metrics().Get("k8gb_gslb_healthy_records").AsGaugeVec()
actualHealthyRecordsMetricCount := testutil.ToFloat64(healthyRecordsMetric)
reconcileAndUpdateGslb(t, settings)
// assert
Expand All @@ -284,15 +269,13 @@ func TestHealthyRecordMetric(t *testing.T) {

func TestMetricLinterCheck(t *testing.T) {
// arrange
settings := provideSettings(t, predefinedConfig)
defer settings.reconciler.Metrics.Unregister()
err := settings.reconciler.Metrics.Register()
require.NoError(t, err)
healthyRecordsMetric := settings.reconciler.Metrics.GetHealthyRecordsMetric()
ingressHostsPerStatusMetric := settings.reconciler.Metrics.GetIngressHostsPerStatusMetric()
healthyRecordsMetric := metrics.Metrics().Get("k8gb_gslb_healthy_records").AsGaugeVec()
ingressHostsPerStatusMetric := metrics.Metrics().Get("k8gb_gslb_ingress_hosts_per_status").AsGaugeVec()
reconciliationTotal := metrics.Metrics().Get("k8gb_gslb_reconciliation_total").AsCounter()
for name, scenario := range map[string]prometheus.Collector{
"healthy_records": healthyRecordsMetric,
"ingress_hosts_per_status": ingressHostsPerStatusMetric,
"reconciliation_total": reconciliationTotal,
} {
// act
// assert
Expand All @@ -302,6 +285,19 @@ func TestMetricLinterCheck(t *testing.T) {
}
}

func TestGslbReconciliationTotalIncrement(t *testing.T) {
// arrange
const key = "k8gb_gslb_reconciliation_total"
settings := provideSettings(t, predefinedConfig)
cnt := testutil.ToFloat64(metrics.Metrics().Get(key).AsCounter())
// act
_, err := settings.reconciler.Reconcile(context.TODO(), settings.request)
cnt2 := testutil.ToFloat64(metrics.Metrics().Get(key).AsCounter())
// assert
assert.NoError(t, err)
assert.Equal(t, cnt+1, cnt2)
}

func TestGslbCreatesDNSEndpointCRForHealthyIngressHosts(t *testing.T) {
// arrange
serviceName := "frontend-podinfo"
Expand Down Expand Up @@ -1168,7 +1164,6 @@ func provideSettings(t *testing.T, expected depresolver.Config) (settings testSe
r.Config = &expected
// Mock request to simulate Reconcile() being called on an event for a
// watched resource .
r.Metrics = metrics.NewPrometheusMetrics(expected)
req := reconcile.Request{
NamespacedName: types.NamespacedName{
Name: gslb.Name,
Expand Down Expand Up @@ -1210,7 +1205,6 @@ func provideSettings(t *testing.T, expected depresolver.Config) (settings testSe
assistant: a,
}
reconcileAndUpdateGslb(t, settings)
logging.Init(&expected)
return settings
}

Expand All @@ -1221,3 +1215,14 @@ func oldEdgeTimestamp(threshold string) string {
edgeTimestamp := fmt.Sprint(before.UTC().Format("2006-01-02T15:04:05"))
return edgeTimestamp
}

func TestMain(m *testing.M) {
logging.Init(&predefinedConfig)
metrics.Init(&predefinedConfig)
defer metrics.Metrics().Unregister()
err := metrics.Metrics().Register()
if err != nil {
logging.Logger().Fatal().Err(err).Msg("metrics register")
}
m.Run()
}
40 changes: 40 additions & 0 deletions controllers/internal/utils/regex.go
@@ -0,0 +1,40 @@
/*
Copyright 2021 The k8gb Contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Generated by GoLic, for more details see: https://github.com/AbsaOSS/golic
*/
package utils

import (
"regexp"
"strings"
)

// SplitAfter works as the same way as strings.SplitAfter() but the separator is regexp
func SplitAfter(s string, re *regexp.Regexp) (r []string) {
if re == nil {
return
}
re.ReplaceAllStringFunc(s, func(x string) string {
s = strings.ReplaceAll(s, x, "::"+x)
return s
})
for _, x := range strings.Split(s, "::") {
if x != "" {
r = append(r, x)
}
}
return
}
52 changes: 52 additions & 0 deletions controllers/internal/utils/regex_test.go
@@ -0,0 +1,52 @@
/*
Copyright 2021 The k8gb Contributors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Generated by GoLic, for more details see: https://github.com/AbsaOSS/golic
*/
package utils

import (
"regexp"
"strings"
"testing"

"github.com/stretchr/testify/assert"
)

func TestSplitAfterIsValid(t *testing.T) {
// arrange
// act
result1 := SplitAfter("HElloFromTest", regexp.MustCompile("[A-Z]"))
result2 := SplitAfter("hellofromtest", regexp.MustCompile("[A-Z]"))
result3 := SplitAfter("1.2.3.4.5", regexp.MustCompile("[0-9]"))
result4 := SplitAfter("HHHHH", regexp.MustCompile("[A-Z]"))
result5 := SplitAfter("", regexp.MustCompile("[A-Z]"))
result6 := SplitAfter("HEllo", regexp.MustCompile(""))
// assert
assert.Equal(t, "H Ello From Test", strings.Join(result1, " "))
assert.Equal(t, "hellofromtest", strings.Join(result2, " "))
assert.Equal(t, "1. 2. 3. 4. 5", strings.Join(result3, " "))
assert.Equal(t, "H H H H H", strings.Join(result4, " "))
assert.Equal(t, "", strings.Join(result5, " "))
assert.Equal(t, "H E l l o", strings.Join(result6, " "))
}

func TestSplitAfterWithNilRegexp(t *testing.T) {
// arrange
// act
result1 := SplitAfter("HElloFromTest", nil)
// assert
assert.Equal(t, "", strings.Join(result1, "_"))
}

0 comments on commit 3d19cfa

Please sign in to comment.