Using MonitorService to setup Prometheus

[camilamacedo86]

Description of the change:
Scaffold out optional ServiceMonitor objects to setup monitor metrics for Prometheus

Motivation for the change:
SDK Integration:

• OSDK->KB: Prometheus ServiceMonitor instead of annotations #887
  Prometheus upstream no longer recommends using annotations.

Following the steps to test it

1. Install https://github.com/coreos/kube-prometheus
2. With this PR create a project to test it following the quick start

$ mkdir monitor
$ cd monitor/
$ kubebuilder init --domain monitor.domain
$ kubebuilder create api --group example --version v1 --kind MonitorTest

3. Uncomment the ../prometheus in the config/default/kustomization.yaml to apply the ServiceMonitor
4. Then, run the commands to apply all and build the project and deploy it.

$ make install
$ kubectl apply -f config/samples/
$ make docker-build docker-push IMG=cmacedo/monitor:0.0.1 # use  your registry
$ make deploy IMG=cmacedo/monitor:0.0.1

5. Check the resources it should have a metric service and the ServiceMonitor.

• Check all resources

$ kubectl get  all -n monitor-system
NAME                                              READY   STATUS    RESTARTS   AGE
pod/monitor-controller-manager-7488d8fd54-s6gt6   2/2     Running   0          33s

NAME                                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/monitor-controller-manager-metrics-service   ClusterIP   10.99.246.247   <none>        8443/TCP   33s

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/monitor-controller-manager   1/1     1            1           33s

NAME                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/monitor-controller-manager-7488d8fd54 

• Check the ServiceMonitor

$ kubectl get ServiceMonitor -n monitor-system 
NAME                                         AGE
monitor-controller-manager-metrics-monitor   2m8s

• Check the ServiceMonitor values

$ kubectl describe ServiceMonitor/monitor-controller-manager-metrics-monitor -n monitor-system
Name:         monitor-controller-manager-metrics-monitor
Namespace:    monitor-system
Labels:       control-plane=controller-manager
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"monitoring.coreos.com/v1","kind":"ServiceMonitor","metadata":{"annotations":{},"labels":{"control-plane":"controller-manage...
API Version:  monitoring.coreos.com/v1
Kind:         ServiceMonitor
Metadata:
  Creation Timestamp:  2019-10-02T09:50:53Z
  Generation:          1
  Resource Version:    26878
  Self Link:           /apis/monitoring.coreos.com/v1/namespaces/monitor-system/servicemonitors/monitor-controller-manager-metrics-monitor
  UID:                 b30307ca-bdb7-47b1-adab-29e5ecb7b690
Spec:
  Endpoints:
    Path:  /metrics
    Port:  https
  Selector:
    Control - Plane:  controller-manager
Events:               <none>

• Check the Service values

$ kubectl describe service/monitor-controller-manager-metrics-service -n monitor-system
Name:              monitor-controller-manager-metrics-service
Namespace:         monitor-system
Labels:            control-plane=controller-manager
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"control-plane":"controller-manager"},"name":"monitor-controlle...
Selector:          control-plane=controller-manager
Type:              ClusterIP
IP:                10.99.246.247
Port:              https  8443/TCP
TargetPort:        https/TCP
Endpoints:         172.17.0.9:8443
Session Affinity:  None
Events:            <none>

• Search for the metrics in the namespace where the POC was deployed.

Search: {namespace="monitor-system"}

[camilamacedo86]

HI @DirectXMan12,

It is done now. Please, feel free to review it.
Really thank you for your time and help with.

Cheers,

[DirectXMan12]

/assign @mengqiy

PTAL

[mengqiy]

When updating the scaffolding, please ensure to run generated_golden.sh to update the testdata directory.

[camilamacedo86]

HI @mengqiy,

Thank you I am doing it now. If is it required, should not be better call it in the default make command and add a ci test to ensure that it is updated? Wdyt? Let's track an issue for that?

[camilamacedo86]

/retest

[k8s-ci-robot]

@camilamacedo86: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mengqiy]

The code LG, but we need to make travis happen to merge the PR.

It is complaining about

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received

We probably should add travis_wait in https://github.com/kubernetes-sigs/kubebuilder/blob/master/.travis.yml#L41 to give it 20 minutes to complete.

[DirectXMan12]

I'd rather figure out what's going on than just increasing the timelimit. It looks like it's failing on download dependencies...

[DirectXMan12]

for instance: https://travis-ci.org/kubernetes-sigs/kubebuilder/jobs/596368671#L571

Why is that just hanging? Perhaps we can run with go test -v and see if it's hitting the test bit, or is still stuck downloading.

[DirectXMan12]

oh, hey, @camilamacedo86 already has a fix up @ #1090 :-)

[camilamacedo86]

Hi @mengqiy,

Shows that all is OK now :-) Really tks for your guidance and help here.
Could we merge this one or have anything else that we should do here to achieve it?

[mengqiy]

Thanks for your hard work to get this through!
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86, mengqiy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mengqiy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[caneGuy]

the searched result in prometheus UI do not show anything about the monitor?
could you help explain the relation between ui and the code above? thanks @camilamacedo86

[camilamacedo86]

Hi @caneGuy the documentation explains how to configure it with kubebuilder is in: https://book.kubebuilder.io/reference/metrics.html?highlight=metr#metrics

However, if you face any problem or have any question could you please address it in an issue in the repo which would be a better place? Feel free to ping me there if you wish.

[caneGuy]

Thanks @camilamacedo86
i have follow the doc step by step.but i can not see the registered metrics on the UI, and i can't find the problem. all the metrics i saw was just what prometheus collected natively.like "cpu_cfs"

[caneGuy]

#1522 a new issue here @camilamacedo86 thanks