pkg/{sdk,generator} Expose Prometheus metrics port #323

etiennecoutaud · 2018-06-22T13:10:58Z

related to #222 discuss

This PR is similar to #241

etiennecoutaud · 2018-06-25T14:12:01Z

CI fail because during memcached-operator generation, latest sdk code from github is vendored.
CI will always fail if we submit a change in pkg/sdk

@theishshah In travis.yml after cd memcached-operator we could manually vendor current code with : ln -sf ${TRAVIS_BUILD_DIR}/pkg vendor/github.com/operator-framework/operator-sdk/pkg
WDYT ?

theishshah · 2018-06-25T18:30:03Z

The travis.yml has been updated on master, so you can pull down that version and re-trigger Travis and it should work as intended.

etiennecoutaud · 2018-06-26T09:40:33Z

My bad, it's still failing

After investigation :

ln -sf ${TRAVIS_BUILD_DIR}/pkg vendor/github.com/operator-framework/operator-sdk/pkg is created a simlink under pkg so we have vendor/github.com/operator-framework/operator-sdk/pkg/pkg
We cannot overwrite directory using ln
We can :
1. rm -r vendor/github.com/operator-framework/operator-sdk/pkg
2. cp -r ${TRAVIS_BUILD_DIR}/pkg vendor/github.com/operator-framework/operator-sdk/pkg

It should finally work 😄

theishshah · 2018-06-26T16:12:42Z

Hey, we noticed that last night as well, PR #329 should address this issue

etiennecoutaud · 2018-06-27T08:50:42Z

Thanks @theishshah

hasbro17 · 2018-07-04T00:15:57Z

@etiennecoutaud Sorry for the delay. I've actually been trying to test your PR for a day now but I'm running into an issue.
The problem is that the prometheus dependencies are not being vendored for the generated project and so I can't build it.

When I do opeartor-sdk new app-operator --api-version=app.example.com/v1alpha1 --kind=App it will vendor the upstream operator-sdk master branch.
Then to test the changes in your PR I replace the vendored sdk pkg with the one from your PR, like we do it in the travis script

rm -r vendor/github.com/operator-framework/operator-sdk/pkg
cp -r ${PATH_TO_LOCAL_SDK}/pkg vendor/github.com/operator-framework/operator-sdk/pkg

However now the issue is that the updated vendored SDK depends on imports like github.com/prometheus/client_golang/prometheus/promhttp which were not vendored in app-opeartor/vendor on the first dep ensure run. And so I can't build the operator due to the missing imports.

$ operator-sdk build quay.io/coreos/operator-sdk-dev:app-operator
Error: failed to build: (building app-operator...
vendor/github.com/operator-framework/operator-sdk/pkg/sdk/metrics.go:8:2: cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of:
	/Users/haseeb/work/go-space/src/github.com/example-inc/app-operator/vendor/github.com/prometheus/client_golang/prometheus/promhttp (vendor tree)
	/usr/local/go/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOROOT)
	/Users/haseeb/work/go-space/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOPATH)
)

Since app-operator does not directly use that import you can't force the prometheus dependencies to be vendored by adding it as a constraint in app-operator/Gopkg.toml either.
This is one of those issues that get's fixed if your change gets merged into master since then everything is vendored correctly from the start but before that it's really hard to test.

So I was wondering if you had managed to do a local build and test of this with the README example?
And given my issue it's surprising is how the CI is able to build this successfully.

etiennecoutaud · 2018-07-04T08:30:37Z

@hasbro17 no problem for the delay.

I reproduced your issue, It works and I'm able to build on my side because I have github.com/prometheus/client_golang in my GOPATH from older works.

ls ~/go/src/github.com/prometheus/
client_golang client_model  common        procfs

operator-sdk build quay.io/coreos/operator-sdk-dev:app-operator
building app-operator...

building container quay.io/coreos/operator-sdk-dev:app-operator...
Sending build context to Docker daemon  186.1MB
Step 1/4 : FROM alpine:3.6
 ---> 77144d8c6bdc
Step 2/4 : RUN adduser -D app-operator
 ---> Using cache
 ---> 2c9b1acc3402
Step 3/4 : USER app-operator
 ---> Using cache
 ---> b0a51d072c95
Step 4/4 : ADD tmp/_output/bin/app-operator /usr/local/bin/app-operator
 ---> d8e320875381
Successfully built d8e320875381
Successfully tagged quay.io/coreos/operator-sdk-dev:app-operator

Create deploy/operator.yaml

➜ (⎈ |minikube:default) ▶ kubectl create -f deploy/operator.yaml
deployment.apps "app-operator" created
➜ (⎈ |minikube:default)  ▶ kubectl get po
NAME                            READY     STATUS    RESTARTS   AGE
app-operator-5dd49b487d-2x55r   1/1       Running   0          5s
➜ (⎈ |minikube:default) ▶ kubectl logs app-operator-5dd49b487d-2x55r
time="2018-07-04T08:26:38Z" level=info msg="Go Version: go1.10"
time="2018-07-04T08:26:38Z" level=info msg="Go OS/Arch: linux/amd64"
time="2018-07-04T08:26:38Z" level=info msg="operator-sdk Version: 0.0.5+git"
time="2018-07-04T08:26:38Z" level=info msg="Metrics service app-operator created"
time="2018-07-04T08:26:38Z" level=info msg="Watching app.example.com/v1alpha1, App, default, 5"
➜ (⎈ |minikube:default) ▶ kubectl get svc
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
app-operator   ClusterIP   10.106.156.196   <none>        60000/TCP   17s
kubernetes     ClusterIP   10.96.0.1        <none>        443/TCP     2m

Regarding this, why CI is green 🤔 ?

hasbro17 · 2018-07-05T23:49:23Z

pkg/generator/generator_test.go

@@ -276,7 +282,7 @@ func TestGenDeploy(t *testing.T) {
 	}

 	buf = &bytes.Buffer{}
-	if err := renderFile(buf, operatorTmplName, operatorYamlTmpl, tmplData{ProjectName: appProjectName, Image: appImage}); err != nil {
+	if err := renderFile(buf, operatorTmplName, operatorYamlTmpl, tmplData{ProjectName: appProjectName, Image: appImage, MetricsPort: k8sutil.PrometheusMetricsPort, MetricsPortName: k8sutil.PrometheusMetricsPortName, OperatorNameEnv: k8sutil.OperatorNameEnvVar}); err != nil {


Define the tmplData before this since it's too long:

td := tmplData{ ProjectName: appProjectName, Image: appImage, MetricsPort: k8sutil.PrometheusMetricsPort, MetricsPortName: k8sutil.PrometheusMetricsPortName, OperatorNameEnv: k8sutil.OperatorNameEnvVar, } if err := renderFile(buf, operatorTmplName, operatorYamlTmpl, td); err != nil { t.Error(err) }

hasbro17 · 2018-07-05T23:57:30Z

@etiennecoutaud I've just tested this out by pulling the dependencies locally and it works. We can figure out what's wrong with the CI later.

I'm going to push this change up as a separate branch on the repo and so I can test whether or not operator-sdk new will correctly vendor the prometheus dependencies for the generated operator when it's merged into the master branch. Once that works this should be good to go.

etiennecoutaud · 2018-07-06T07:45:22Z

@hasbro17 okay !
Do not hesitate to ask if you need help.

secat · 2018-07-06T12:08:31Z

pkg/sdk/metrics.go

+	"strconv"
+
+	k8sutil "github.com/operator-framework/operator-sdk/pkg/util/k8sutil"
+	"github.com/prometheus/client_golang/prometheus/promhttp"


As a suggestion, the prometheus OpenCensus exporter library could be used.

Here is a code snippet to illustrate the usage:

import ( "github.com/sirupsen/logrus" "github.com/prometheus/client_golang/prometheus" ocprometheus "go.opencensus.io/exporter/prometheus" ocview "go.opencensus.io/stats/view" ) metricsRegistry := prometheus.NewRegistry() metricsRegistry.MustRegister(prometheus.NewGoCollector()) prometheusExporter, err := ocprometheus.NewExporter(ocprometheus.Options{ Namespace: "operator_sdk", Registry: metricsRegistry, }) if err != nil { logrus.Fatalf("Failed to create prometheus exporter: %v", err) return } ocview.RegisterExporter(prometheusExporter) http.Handle("/"+k8sutil.PrometheusMetricsPortName, prometheusExporter)

IMHO, this project should integrate and use the OpenCensus library.

@secat I'm not too familiar with the OpenCensus library so I can't comment on the need for it yet. However this PR is only meant to be the starting point for exposing the Prometheus metrics so I would prefer to keep it simple right now and iterate on more changes after this.
It'll be helpful if you can create an issue so we can discuss the use cases for the OpenCensus library in the SDK.

@hasbro17 I have created the issue #343

hasbro17 · 2018-07-06T22:05:37Z

LGTM.
Thanks for working through all the iterations.

…ebase-master Merge upstream tag v1.29.0

etiennecoutaud force-pushed the expose-metrics branch from 55bc84f to 404791e Compare June 25, 2018 07:57

etiennecoutaud force-pushed the expose-metrics branch from 404791e to 6b89548 Compare June 25, 2018 20:31

etiennecoutaud force-pushed the expose-metrics branch from 6b89548 to 825da34 Compare June 27, 2018 07:46

hasbro17 reviewed Jul 5, 2018

View reviewed changes

pkg/{sdk,generator} Expose Prometheus metrics port

d1c5dbd

etiennecoutaud force-pushed the expose-metrics branch from 825da34 to d1c5dbd Compare July 6, 2018 07:42

secat reviewed Jul 6, 2018

View reviewed changes

hasbro17 merged commit 3c5667f into operator-framework:master Jul 6, 2018

hasbro17 mentioned this pull request Jul 9, 2018

k8sclient.client: refactored to create singleton pattern, remove init #340

Merged

secat mentioned this pull request Jul 11, 2018

Use the OpenCensus golang library in the SDK #343

Closed

hasbro17 mentioned this pull request Sep 17, 2018

pkg/sdk: Add /debug and /metrics http endpoints #248

Closed

m1kola pushed a commit to m1kola/operator-sdk that referenced this pull request Jun 7, 2024

Merge pull request operator-framework#323 from everettraven/v1.29.0-r…

13bc575

…ebase-master Merge upstream tag v1.29.0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/{sdk,generator} Expose Prometheus metrics port #323

pkg/{sdk,generator} Expose Prometheus metrics port #323

etiennecoutaud commented Jun 22, 2018

etiennecoutaud commented Jun 25, 2018 •

edited

Loading

theishshah commented Jun 25, 2018

etiennecoutaud commented Jun 26, 2018 •

edited

Loading

theishshah commented Jun 26, 2018

etiennecoutaud commented Jun 27, 2018

hasbro17 commented Jul 4, 2018

etiennecoutaud commented Jul 4, 2018 •

edited

Loading

hasbro17 Jul 5, 2018

hasbro17 commented Jul 5, 2018

etiennecoutaud commented Jul 6, 2018

secat Jul 6, 2018 •

edited

Loading

hasbro17 Jul 6, 2018

secat Jul 11, 2018

hasbro17 commented Jul 6, 2018

pkg/{sdk,generator} Expose Prometheus metrics port #323

pkg/{sdk,generator} Expose Prometheus metrics port #323

Conversation

etiennecoutaud commented Jun 22, 2018

etiennecoutaud commented Jun 25, 2018 • edited Loading

theishshah commented Jun 25, 2018

etiennecoutaud commented Jun 26, 2018 • edited Loading

theishshah commented Jun 26, 2018

etiennecoutaud commented Jun 27, 2018

hasbro17 commented Jul 4, 2018

etiennecoutaud commented Jul 4, 2018 • edited Loading

hasbro17 Jul 5, 2018

Choose a reason for hiding this comment

hasbro17 commented Jul 5, 2018

etiennecoutaud commented Jul 6, 2018

secat Jul 6, 2018 • edited Loading

Choose a reason for hiding this comment

hasbro17 Jul 6, 2018

Choose a reason for hiding this comment

secat Jul 11, 2018

Choose a reason for hiding this comment

hasbro17 commented Jul 6, 2018

etiennecoutaud commented Jun 25, 2018 •

edited

Loading

etiennecoutaud commented Jun 26, 2018 •

edited

Loading

etiennecoutaud commented Jul 4, 2018 •

edited

Loading

secat Jul 6, 2018 •

edited

Loading