Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose advanced metrics #124

Closed
somaritane opened this issue May 15, 2020 · 1 comment
Closed

Expose advanced metrics #124

somaritane opened this issue May 15, 2020 · 1 comment
Labels
enhancement New feature or request
Projects
Milestone

Comments

@somaritane
Copy link
Contributor

This issue is a spin off from #47

Implement relevant metrics that provide insight into:

  • Edge DNS integrations
  • Split brain scenarios / heartbeat
  • Failover scenarios
  • Reconciliation
  • Load balancing strategies used

#97 can be used as reference implementation

@somaritane somaritane added the enhancement New feature or request label May 15, 2020
@somaritane somaritane added this to the 0.7 milestone May 15, 2020
@ytsarev ytsarev modified the milestones: 0.7, 0.8 Oct 19, 2020
kuritka added a commit that referenced this issue Jan 14, 2021
This is first of PR

 -  move Dig to utils
 -  move prettyPrint to utils and rename to `ToJSON()`
 -  cover `ToJSON()` by tests
 -  move environment variable `POD_NAMESPACE` to `depresolver`
 -  extend depresolver tests
 -  extend depresolver validators for namespace validator
 -  Extract Prometheus metrics into standalone module
 -  Extend prometheus metrics
 -  Fix controller tests
 -  remove dead registry/k3s.yaml file (pushed by mistake)

TODO:
  - dnsupdate refactor to provider pattern in next PR
  - fianally, some of utils will go into [gopkg](https://github.com/AbsaOSS/gopkg)
  - IMetrics introduction (#124)
kuritka added a commit that referenced this issue Jan 14, 2021
This is first of PR

 -  move Dig to utils
 -  move prettyPrint to utils and rename to `ToJSON()`
 -  cover `ToJSON()` by tests
 -  move environment variable `POD_NAMESPACE` to `depresolver`
 -  extend depresolver tests
 -  extend depresolver validators for namespace validator
 -  Extract Prometheus metrics into standalone module
 -  Extend prometheus metrics
 -  Fix controller tests
 -  remove dead registry/k3s.yaml file (pushed by mistake)

TODO:
  - dnsupdate refactor to provider pattern in next PR
  - fianally, some of utils will go into [gopkg](https://github.com/AbsaOSS/gopkg)
  - IMetrics introduction (#124)
kuritka added a commit that referenced this issue Jan 14, 2021
This is first of PR

 -  move Dig to utils
 -  move prettyPrint to utils and rename to `ToJSON()`
 -  cover `ToJSON()` by tests
 -  move environment variable `POD_NAMESPACE` to `depresolver`
 -  extend depresolver tests
 -  extend depresolver validators for namespace validator
 -  Extract Prometheus metrics into standalone module
 -  Extend prometheus metrics
 -  Fix controller tests
 -  remove dead registry/k3s.yaml file (pushed by mistake)

TODO:
  - dnsupdate refactor to provider pattern in next PR
  - fianally, some of utils will go into [gopkg](https://github.com/AbsaOSS/gopkg)
  - IMetrics introduction (#124)
kuritka added a commit that referenced this issue Jan 15, 2021
This is first of PR

 -  move Dig to utils
 -  move prettyPrint to utils and rename to `ToJSON()`
 -  cover `ToJSON()` by tests
 -  move environment variable `POD_NAMESPACE` to `depresolver`
 -  extend depresolver tests
 -  extend depresolver validators for namespace validator
 -  Extract Prometheus metrics into standalone module
 -  Extend prometheus metrics
 -  Fix controller tests
 -  remove dead registry/k3s.yaml file (pushed by mistake)

TODO:
  - dnsupdate refactor to provider pattern in next PR
  - fianally, some of utils will go into [gopkg](https://github.com/AbsaOSS/gopkg)
  - IMetrics introduction (#124)
kuritka added a commit that referenced this issue Jan 15, 2021
This is first of PR

 -  move environment variable `POD_NAMESPACE` to `depresolver`
 -  extend depresolver tests
 -  extend depresolver validators for namespace validator
 -  Extract Prometheus metrics into standalone module
 -  Extend prometheus metrics
 -  Fix controller tests
 -  remove dead registry/k3s.yaml file (pushed by mistake)

TODO:
  - dnsupdate refactor to provider pattern in next PR
  - fianally, some of utils will go into [gopkg](https://github.com/AbsaOSS/gopkg)
  - IMetrics introduction (#124)
kuritka added a commit that referenced this issue Jan 15, 2021
This is first of PR

 -  move environment variable `POD_NAMESPACE` to `depresolver`
 -  extend depresolver tests
 -  extend depresolver validators for namespace validator
 -  Extract Prometheus metrics into standalone module
 -  Extend prometheus metrics
 -  Fix controller tests
 -  remove dead registry/k3s.yaml file (pushed by mistake)

TODO:
  - dnsupdate refactor to provider pattern in next PR
  - fianally, some of utils will go into [gopkg](https://github.com/AbsaOSS/gopkg)
  - IMetrics introduction (#124)
@kuritka
Copy link
Collaborator

kuritka commented Jan 15, 2021

ℹ️ note : Consider implementing against the IMetrics interface.
e.g.

type IMetrics interface {
   Register() error
   Unregister() error
   UpdateIngressHostsPerStatusMetric(gslb *k8gbv1beta1.Gslb, serviceHealth map[string]string)
   UpdateHealthyRecordsMetric(gslb *k8gbv1beta1.Gslb, healthyRecords map[string][]string)

   ConcreteMetricA() (data, err)
   ConcreteMetricB() (data, err)
   ConcreteMetricC() (data, err)
...
}

@ytsarev ytsarev modified the milestones: 0.8, 0.9 Mar 16, 2021
kuritka added a commit that referenced this issue Jun 15, 2021
ted to #124

I created a PrometheusMetrics structure with one field - Port.  The default port value is 9090 and defines where Prometheus metrics endpoint is listening. PrometheusMetrics  structure can return metrics local address via `GetAddress()` function. At the moment I don’t expect we will expose metrics on other than “0.0.0.0” host, that’s why I hardcoded host value for now. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver (following PR will remove flags completely).
kuritka added a commit that referenced this issue Jun 15, 2021
ted to #124

I created a PrometheusMetrics structure with one field - Port.  The default port value is 9090 and defines where Prometheus metrics endpoint is listening. PrometheusMetrics  structure can return metrics local address via `GetAddress()` function. At the moment I don’t expect we will expose metrics on other than “0.0.0.0” host, that’s why I hardcoded host value for now. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver (following PR will remove flags completely).

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 15, 2021
Related to #124

I created a PrometheusMetrics structure with one field - Port.  The default port value is 9090 and defines where Prometheus metrics endpoint is listening. PrometheusMetrics  structure can return metrics local address via `GetAddress()` function. At the moment I don’t expect we will expose metrics on other than “0.0.0.0” host, that’s why I hardcoded host value for now. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver (following PR will remove flags completely).

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 15, 2021
Related to #124

I created a PrometheusMetrics structure with one field - Port.  The default port value is 9090 and defines where Prometheus metrics endpoint is listening. PrometheusMetrics  structure can return metrics local address via `GetAddress()` function. At the moment I don’t expect we will expose metrics on other than “0.0.0.0” host, that’s why I hardcoded host value for now. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver (following PR will remove flags completely).

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 15, 2021
Related to #124

I created a PrometheusMetrics structure with one field - Port.  The default port value is 9090 and defines where Prometheus metrics endpoint is listening. PrometheusMetrics  structure can return metrics local address via `GetAddress()` function. At the moment I don’t expect we will expose metrics on other than “0.0.0.0” host, that’s why I hardcoded host value for now. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver (following PR will remove flags completely).

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 15, 2021
Related to #124

I created a PrometheusMetrics structure with one field - Port.  The default port value is 9090 and defines where Prometheus metrics endpoint is listening. PrometheusMetrics  structure can return metrics local address via `GetAddress()` function. At the moment I don’t expect we will expose metrics on other than “0.0.0.0” host, that’s why I hardcoded host value for now. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver (following PR will remove flags completely).

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 15, 2021
Related to #124
I created a MetricsAddress field.  The default port value is “0.0.0.0:8080" and defines where Prometheus metrics endpoint is listening. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver (following PR will remove flags completely).

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 15, 2021
Related to #124
I created a MetricsAddress field.  The default port value is “0.0.0.0:8080" and defines where Prometheus metrics endpoint is listening. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver and also, removed enableLeaderElection and overall flags completely.

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 15, 2021
Related to #124
I created a MetricsAddress field.  The default port value is “0.0.0.0:8080" and defines where Prometheus metrics endpoint is listening. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver and also, removed enableLeaderElection and overall flags completely.

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 15, 2021
Related to #124
I created a MetricsAddress field.  The default port value is “0.0.0.0:8080" and defines where Prometheus metrics endpoint is listening. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver and also, removed enableLeaderElection and overall flags completely.

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 16, 2021
Related to #124
I created a MetricsAddress field.  The default port value is “0.0.0.0:8080" and defines where Prometheus metrics endpoint is listening. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver and also, removed enableLeaderElection and overall flags completely.

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 16, 2021
Related to #124
I created a MetricsAddress field.  The default port value is “0.0.0.0:8080" and defines where Prometheus metrics endpoint is listening. (see: https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-monitoring-prometheus.html#osdk-monitoring-prometheus-metrics-helper_osdk-monitoring-prometheus)

I also removed flag.StringVar `"metrics-addr", ":8080”,` from `main.go` as moving inputs into depresolver and also, removed enableLeaderElection and overall flags completely.

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jun 18, 2021
related to #124

I want to see the metrics that will be implemented. That's why I'm adding the option to install prometheus on both clusters with `make deploy-prometheus`.

Such deployed prometheus can scrapes metrics of k8gb operator and is accessible directly from the browser:

 - http://127.0.0.1:8080 - test-gslb1
 - http://127.0.0.1:8081 - test-gslb2
kuritka added a commit that referenced this issue Jun 18, 2021
related to #124

I want to check metrics that will be implemented. That's why I'm adding the option to install prometheus on both clusters with `make deploy-prometheus`.

Such deployed prometheus scrapes metrics of k8gb operator and is accessible directly from the browser:

 - http://127.0.0.1:8080 - test-gslb1
 - http://127.0.0.1:8081 - test-gslb2
kuritka added a commit that referenced this issue Jun 18, 2021
related to #124

I want to check metrics that will be implemented. That's why I'm adding the option to install prometheus on both clusters with `make deploy-prometheus`.

Such deployed prometheus scrapes metrics of k8gb operator and is accessible directly from the browser:

 - http://127.0.0.1:8080 - test-gslb1
 - http://127.0.0.1:8081 - test-gslb2
kuritka added a commit that referenced this issue Jul 22, 2021
related to #124

In this pull request I implemented the metrics system in k8gb. I added one metric - counter
incrementing the number of successful reconciliation loops. Adding specific metrics is part of the next PR.
Like the logger, the metrics are accessed through a single singleton object and like the logger it is initialized.

```go
var log = logging.Logger()

var m = metrics.Metrics()
```
then anywhere in the code we can use this:
```
// increase the number of successful reconciliations
m.ReconciliationIncrement()
```
The singleton is initialized in the `main()` function using the `metrics.Init(*config)` function.
In case the singleton is not initialized, it will be set to the default value and all metrics will
start with the prefix `k8gb_default` otherwise they will start with the prefix from the env variable `K8GB_NAMESPACE`.

In addition to the singleton, I have changed the original metrics and added tests at the package level.

In addition to the singleton, I significantly refactored the original metrics and added tests at the package level.

The prometheus metrics implementation uses reflection. In short, I read the collectors structure during the runtime:
```go
type collectors struct {
	HealthyRecords *prometheus.GaugeVec
	IngressHostsPerStatus *prometheus.GaugeVec
	ZoneUpdateTotal prometheus.Counter
	ReconciliationTotal prometheus.Counter
}
```
Now I know all the types and names. From this I generate the map <name>: <instance>.
I will then use the map for Register(), Unregister() and Get().

<name> is the name of the metric and is generated from the name of the structure in `collectors`.
ReconciliationTotal => for example: `k8gb_gslb_healthy_records` (k8gb is `K8GB_NAMESPACE`)

`Get()` is only for testing purposes - and is public only because controller_tests use metrics.
I'll consider deleting tests from controller_tests and leaving only terratests and unit_tests at the package level (96% coverage now).
But that will come in the next PR.

Signed-off-by: kuritka <kuritka@gmail.com>
@kuritka kuritka mentioned this issue Jul 22, 2021
kuritka added a commit that referenced this issue Jul 22, 2021
related to #124

In this pull request I implemented the metrics system in k8gb. I added one metric - counter
incrementing the number of successful reconciliation loops. Adding specific metrics is part of the next PR.
Like the logger, the metrics are accessed through a single singleton object and like the logger it is initialized.

```go
var log = logging.Logger()

var m = metrics.Metrics()
```
then anywhere in the code we can use this:
```
// increase the number of successful reconciliations
m.ReconciliationIncrement()
```
The singleton is initialized in the `main()` function using the `metrics.Init(*config)` function.
In case the singleton is not initialized, it will be set to the default value and all metrics will
start with the prefix `k8gb_default` otherwise they will start with the prefix from the env variable `K8GB_NAMESPACE`.

In addition to the singleton, I have changed the original metrics and added tests at the package level.

In addition to the singleton, I significantly refactored the original metrics and added tests at the package level.

The prometheus metrics implementation uses reflection. In short, I read the collectors structure during the runtime:
```go
type collectors struct {
	HealthyRecords *prometheus.GaugeVec
	IngressHostsPerStatus *prometheus.GaugeVec
	ZoneUpdateTotal prometheus.Counter
	ReconciliationTotal prometheus.Counter
}
```
Now I know all the types and names. From this I generate the map <name>: <instance>.
I will then use the map for Register(), Unregister() and Get().

<name> is the name of the metric and is generated from the name of the structure in `collectors`.
ReconciliationTotal => for example: `k8gb_gslb_healthy_records` (k8gb is `K8GB_NAMESPACE`)

`Get()` is only for testing purposes - and is public only because controller_tests use metrics.
I'll consider deleting tests from controller_tests and leaving only terratests and unit_tests at the package level (96% coverage now).
But that will come in the next PR.

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jul 22, 2021
related to #124

In this pull request I implemented the metrics system in k8gb. I added one metric - counter
incrementing the number of successful reconciliation loops. Adding specific metrics is part of the next PR.
Like the logger, the metrics are accessed through a single singleton object and like the logger it is initialized.

```go
var log = logging.Logger()

var m = metrics.Metrics()
```
then anywhere in the code we can use this:
```
// increase the number of successful reconciliations
m.ReconciliationIncrement()
```
The singleton is initialized in the `main()` function using the `metrics.Init(*config)` function.
In case the singleton is not initialized, it will be set to the default value and all metrics will
start with the prefix `k8gb_default` otherwise they will start with the prefix from the env variable `K8GB_NAMESPACE`.

In addition to the singleton, I have changed the original metrics and added tests at the package level.

In addition to the singleton, I significantly refactored the original metrics and added tests at the package level.

The prometheus metrics implementation uses reflection. In short, I read the collectors structure during the runtime:
```go
type collectors struct {
	HealthyRecords *prometheus.GaugeVec
	IngressHostsPerStatus *prometheus.GaugeVec
	ZoneUpdateTotal prometheus.Counter
	ReconciliationTotal prometheus.Counter
}
```
Now I know all the types and names. From this I generate the map <name>: <instance>.
I will then use the map for Register(), Unregister() and Get().

<name> is the name of the metric and is generated from the name of the structure in `collectors`.
ReconciliationTotal => for example: `k8gb_gslb_healthy_records` (k8gb is `K8GB_NAMESPACE`)

`Get()` is only for testing purposes - and is public only because controller_tests use metrics.
I'll consider deleting tests from controller_tests and leaving only terratests and unit_tests at the package level (96% coverage now).
But that will come in the next PR.

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jul 22, 2021
related to #124

In this pull request I implemented the metrics system in k8gb. I added one metric - counter
incrementing the number of successful reconciliation loops. Adding specific metrics is part of the next PR.
Like the logger, the metrics are accessed through a single singleton object and like the logger it is initialized.

```go
var log = logging.Logger()

var m = metrics.Metrics()
```
then anywhere in the code we can use this:
```
// increase the number of successful reconciliations
m.ReconciliationIncrement()
```
The singleton is initialized in the `main()` function using the `metrics.Init(*config)` function.
In case the singleton is not initialized, it will be set to the default value and all metrics will
start with the prefix `k8gb_default` otherwise they will start with the prefix from the env variable `K8GB_NAMESPACE`.

In addition to the singleton, I have changed the original metrics and added tests at the package level.

In addition to the singleton, I significantly refactored the original metrics and added tests at the package level.

The prometheus metrics implementation uses reflection. In short, I read the collectors structure during the runtime:
```go
type collectors struct {
	HealthyRecords *prometheus.GaugeVec
	IngressHostsPerStatus *prometheus.GaugeVec
	ZoneUpdateTotal prometheus.Counter
	ReconciliationTotal prometheus.Counter
}
```
Now I know all the types and names. From this I generate the map <name>: <instance>.
I will then use the map for Register(), Unregister() and Get().

<name> is the name of the metric and is generated from the name of the structure in `collectors`.
ReconciliationTotal => for example: `k8gb_gslb_healthy_records` (k8gb is `K8GB_NAMESPACE`)

`Get()` is only for testing purposes - and is public only because controller_tests use metrics.
I'll consider deleting tests from controller_tests and leaving only terratests and unit_tests at the package level (96% coverage now).
But that will come in the next PR.

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Jul 22, 2021
related to #124

In this pull request I implemented the metrics system in k8gb. I added one metric - counter
incrementing the number of successful reconciliation loops. Adding specific metrics is part of the next PR.
Like the logger, the metrics are accessed through a single singleton object and like the logger it is initialized.

```go
var log = logging.Logger()

var m = metrics.Metrics()
```
then anywhere in the code we can use this:
```
// increase the number of successful reconciliations
m.ReconciliationIncrement()
```
The singleton is initialized in the `main()` function using the `metrics.Init(*config)` function.
In case the singleton is not initialized, it will be set to the default value and all metrics will
start with the prefix `k8gb_default` otherwise they will start with the prefix from the env variable `K8GB_NAMESPACE`.

In addition to the singleton, I have changed the original metrics and added tests at the package level.

In addition to the singleton, I significantly refactored the original metrics and added tests at the package level.

The prometheus metrics implementation uses reflection. In short, I read the collectors structure during the runtime:
```go
type collectors struct {
	HealthyRecords *prometheus.GaugeVec
	IngressHostsPerStatus *prometheus.GaugeVec
	ZoneUpdateTotal prometheus.Counter
	ReconciliationTotal prometheus.Counter
}
```
Now I know all the types and names. From this I generate the map <name>: <instance>.
I will then use the map for Register(), Unregister() and Get().

<name> is the name of the metric and is generated from the name of the structure in `collectors`.
ReconciliationTotal => for example: `k8gb_gslb_healthy_records` (k8gb is `K8GB_NAMESPACE`)

`Get()` is only for testing purposes - and is public only because controller_tests use metrics.
I'll consider deleting tests from controller_tests and leaving only terratests and unit_tests at the package level (96% coverage now).
But that will come in the next PR.

Signed-off-by: kuritka <kuritka@gmail.com>
@kuritka kuritka mentioned this issue Jul 22, 2021
kuritka added a commit that referenced this issue Aug 10, 2021
related to #124

In this pull request I implemented the metrics system in k8gb. I added one metric - counter
incrementing the number of successful reconciliation loops. Adding specific metrics is part of the next PR.
Like the logger, the metrics are accessed through a single singleton object and like the logger it is initialized.

```go
var log = logging.Logger()

var m = metrics.Metrics()
```
then anywhere in the code we can use this:
```
// increase the number of successful reconciliations
m.ReconciliationIncrement()
```
The singleton is initialized in the `main()` function using the `metrics.Init(*config)` function.
In case the singleton is not initialized, it will be set to the default value and all metrics will
start with the prefix `k8gb_default` otherwise they will start with the prefix from the env variable `K8GB_NAMESPACE`.

In addition to the singleton, I have changed the original metrics and added tests at the package level.

In addition to the singleton, I significantly refactored the original metrics and added tests at the package level.

The prometheus metrics implementation uses reflection. In short, I read the collectors structure during the runtime:
```go
type collectors struct {
	HealthyRecords *prometheus.GaugeVec
	IngressHostsPerStatus *prometheus.GaugeVec
	ZoneUpdateTotal prometheus.Counter
	ReconciliationTotal prometheus.Counter
}
```
Now I know all the types and names. From this I generate the map <name>: <instance>.
I will then use the map for Register(), Unregister() and Get().

<name> is the name of the metric and is generated from the name of the structure in `collectors`.
ReconciliationTotal => for example: `k8gb_gslb_healthy_records` (k8gb is `K8GB_NAMESPACE`)

`Get()` is only for testing purposes - and is public only because controller_tests use metrics.
I'll consider deleting tests from controller_tests and leaving only terratests and unit_tests at the package level (96% coverage now).
But that will come in the next PR.

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 19, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 19, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 19, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 19, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 19, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Signed-off-by: kuritka <kuritka@gmail.com>
@kuritka kuritka mentioned this issue Aug 19, 2021
kuritka added a commit that referenced this issue Aug 23, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 23, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Signed-off-by: kuritka <kuritka@gmail.com>
@kuritka kuritka mentioned this issue Aug 23, 2021
kuritka added a commit that referenced this issue Aug 23, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 24, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 24, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Another metrics are:

 - k8gb_infoblox_heartbeats_total
 - k8gb_infoblox_heartbeat_errors_total
 - k8gb_infoblox_zone_updates_total
 - k8gb_infoblox_zone_update_errors_total

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 24, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Infoblox metrics are:
 - k8gb_infoblox_heartbeats_total
 - k8gb_infoblox_heartbeat_errors_total
 - k8gb_infoblox_zone_updates_total
 - k8gb_infoblox_zone_update_errors_total

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 24, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Infoblox metrics are:
 - k8gb_infoblox_heartbeats_total
 - k8gb_infoblox_heartbeat_errors_total
 - k8gb_infoblox_zone_updates_total
 - k8gb_infoblox_zone_update_errors_total

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 24, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Infoblox metrics are:
 - k8gb_infoblox_heartbeats_total
 - k8gb_infoblox_heartbeat_errors_total
 - k8gb_infoblox_zone_updates_total
 - k8gb_infoblox_zone_update_errors_total

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 24, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Infoblox metrics are:
 - k8gb_infoblox_heartbeats_total
 - k8gb_infoblox_heartbeat_errors_total
 - k8gb_infoblox_zone_updates_total
 - k8gb_infoblox_zone_update_errors_total

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 25, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Infoblox metrics are:
 - k8gb_infoblox_heartbeats_total
 - k8gb_infoblox_heartbeat_errors_total
 - k8gb_infoblox_zone_updates_total
 - k8gb_infoblox_zone_update_errors_total

Signed-off-by: kuritka <kuritka@gmail.com>
kuritka added a commit that referenced this issue Aug 25, 2021
related to #124

This PR contains several specific metrics
```gaolang
	Health records *prometheus.GaugeVec
	StatusPerIngressHosts *prometheus.GaugeVec
	StatusFailover *prometheus.GaugeVec
	StatusRoundRobin *prometheus.GaugeVec
	StatusGeoIP *prometheus.GaugeVec
	ZoneUpdateTotal *prometheus.CounterVec
	ErrorTotal *prometheus.CounterVec
	ReconciliationTotal prometheus.Counter
```

```shell
make destroy-full-local-setup deploy-full-local-setup upgrade-candidate deploy-prometheus
make stop-test-app
make start-test-app
```

![Screenshot 2021-08-19 at 14 38 53](https://user-images.githubusercontent.com/7195836/130069631-6ad73421-9a76-4327-a856-7873615fba86.png)

![Screenshot 2021-08-19 at 14 38 10](https://user-images.githubusercontent.com/7195836/130069543-f446fa68-5364-4ff3-a1de-9ffc7c9a8bc2.png)

![Screenshot 2021-08-19 at 14 39 16](https://user-images.githubusercontent.com/7195836/130069680-026e5ec4-f66a-4ea0-9ebd-6a6c5eaa2d60.png)

![Screenshot 2021-08-19 at 14 44 53](https://user-images.githubusercontent.com/7195836/130070434-712c9583-e176-46bd-9994-6059de5d64a1.png)

![Screenshot 2021-08-19 at 14 45 40](https://user-images.githubusercontent.com/7195836/130070537-d46e410b-6a8e-4d27-86ce-474b803a988c.png)

![Screenshot 2021-08-19 at 14 49 21](https://user-images.githubusercontent.com/7195836/130071022-b82942f8-e469-48b8-946a-583f49c54497.png)

![Screenshot 2021-08-19 at 14 49 45](https://user-images.githubusercontent.com/7195836/130071080-3b4a4d63-a86f-47c1-bc64-211beb819749.png)

![Screenshot 2021-08-19 at 14 50 26](https://user-images.githubusercontent.com/7195836/130071169-ddc065de-dd21-47a3-a35b-3f0a1be9c416.png)

Infoblox metrics are:
 - k8gb_infoblox_heartbeats_total
 - k8gb_infoblox_heartbeat_errors_total
 - k8gb_infoblox_zone_updates_total
 - k8gb_infoblox_zone_update_errors_total

Signed-off-by: kuritka <kuritka@gmail.com>
@kuritka kuritka closed this as completed Aug 26, 2021
k8gb automation moved this from In progress to Done Aug 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
k8gb
Done
Development

No branches or pull requests

3 participants