Adds pipeline metrics 🔭 #1387

hrishin · 2019-10-07T13:11:13Z

Changes

Often, as a developer or administrator(ops) I want some insights
about pipeline behavior in terms of time taken to execute pipleinerun/taskrun,
its success or failure ratio, pod latencies etc.
At present tekton pipelines has very limited ways to surface such information
or it's hard to get those details looking at resources yamls.

This patch exposes above mentioned pipelines metrics on '/metrics'
endpoint using knative pkg/metrics package. User can collect such
metrics using prometheus, stackdriver or other supported metrics system.

To some extent its solves

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes tests (if functionality changed/added)
Includes docs (if user facing)
Commit messages follow commit message best practices

See the contribution guide for more details.

Release Notes

Following Pipelines metrics are available at `tekton-pipelines-controller:9090/metrics` endpoint
1. Taskrun/Pipeline execution duration time
2. Pipelinerun/Taskrun `success` and `failure` duration rates
3. Number of Taskruns and Pipelineruns are executing currently
4. Pod scheduling latency for Taskruns

tekton-robot · 2019-10-07T13:13:15Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	72.7%	53.3%	-19.4
pkg/reconciler/pipelinerun/metrics.go	Do not exist	87.5%
pkg/reconciler/pipelinerun/pipelinerun.go	83.1%	83.5%	0.4
pkg/reconciler/taskrun/metrics.go	Do not exist	87.5%
pkg/reconciler/taskrun/taskrun.go	71.6%	71.6%	-0.0
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	85.6%	84.7%	-0.9

tekton-robot · 2019-10-07T13:38:38Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	72.7%	53.3%	-19.4
pkg/reconciler/pipelinerun/metrics.go	Do not exist	87.5%
pkg/reconciler/pipelinerun/pipelinerun.go	83.1%	83.5%	0.4
pkg/reconciler/taskrun/metrics.go	Do not exist	87.5%
pkg/reconciler/taskrun/taskrun.go	71.6%	71.6%	-0.0
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	85.6%	84.7%	-0.9

tekton-robot · 2019-10-07T15:56:10Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	72.7%	53.3%	-19.4
pkg/reconciler/pipelinerun/metrics.go	Do not exist	87.5%
pkg/reconciler/pipelinerun/pipelinerun.go	83.1%	83.5%	0.4
pkg/reconciler/taskrun/metrics.go	Do not exist	87.5%
pkg/reconciler/taskrun/taskrun.go	71.6%	71.6%	-0.0
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	85.6%	84.7%	-0.9

imjasonh

Really excited to see this coming together, thanks Hrishikesh!

After this lands let me know if you'd like me to do some of the other metrics (taskruns, etc.)

pkg/reconciler/pipelinerun/metrics_test.go

pkg/reconciler/pipelinerun/pipelinerun.go

pkg/reconciler/pipelinerun/metrics_test.go

imjasonh · 2019-10-07T16:06:37Z

pkg/reconciler/pipelinerun/pipelinerun.go

@@ -156,6 +156,7 @@ func (c *Reconciler) Reconcile(ctx context.Context, key string) error {
 			c.Logger.Errorf("Failed to update TaskRun status for PipelineRun %s: %v", pr.Name, err)
 			return err
 		}
+		_ = c.metrics.DurationAndCount(pr)


Since DurationAndCount might take a long time (see comment below), have you considered running this in a goroutine so that Reconcile is not blocked?

vdemeester

exciting 🕺

tekton-robot · 2019-10-12T06:35:52Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	79.4%	2.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.7%	83.4%	0.7
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.7%	-0.9
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

tekton-robot · 2019-10-13T05:16:01Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	79.4%	2.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.7%	83.4%	0.7
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.7%	-0.9
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

tekton-robot · 2019-10-13T05:31:50Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	79.4%	2.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.7%	83.4%	0.7
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.7%	-0.9
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

tekton-robot · 2019-10-13T09:52:24Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	79.4%	2.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.7%	83.4%	0.7
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.7%	-0.9
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

afrittoli · 2019-10-14T16:25:00Z

https://thenewstack.io/opentracing-opencensus-merge-into-a-single-new-project-opentelemetry/

I wonder we should start right away with opentelemetry, or if the new API is too young (it's still alpha)?

imjasonh · 2019-10-12T12:44:48Z

cmd/controller/main.go

@@ -28,7 +28,7 @@ import (

 const (
 	// ControllerLogKey is the name of the logger for the controller cmd
-	ControllerLogKey = "controller"
+	ControllerLogKey = "tekton"


Is this necessary in this change? It seems useful to key messages from this binary as coming from the controller, as opposed to "Tekton" which could mean lots of things.

A valid argument,

issue: this name used in a metric name as a prefix which I didn't like controller_***.
Ideally, we should fix the limitation in knative pkg or I'm open for any alternative suggestions

Oh, so this "log key" is used both for logging, and for metrics, is that the case?

Maybe just file an issue with knative/pkg to make these separate, and link that here in a TODO.

I was not aware it's is used as log key as well

imjasonh · 2019-10-12T12:46:01Z

pkg/apis/pipeline/v1alpha1/taskrun_types.go

+}
+
+// IsOfAPipeline return true if taskrun is a part of pipeline
+func (tr *TaskRun) IsOfAPipeline() (bool, map[string]string) {


Add a description of the returned map to the doc comment. And since it returns something more than just a simple bool it might make sense to rephrase the method name?

Maybe it should return the Pipeline name and PipelineRun name, instead of a map containing both of those strings?

Is it better now?

pkg/reconciler/pipelinerun/metrics.go

imjasonh · 2019-10-12T12:49:31Z

pkg/reconciler/taskrun/metrics.go

+
+	if err != nil {
+		r.initialized = false
+		return r, err


Should this return a nil Recorder, instead of one that will error when asked to record anything?

We wanted to return the nil, however everytime in before calling the metrics methods in controller flow we need to check for nil, right? instead of doing that way we thought to add that check while calling metrics methods. Now trade here is the method has to check if the instance is actually initialized which is either not that great but it avoid many if check scatter accross caller places.

release.yaml

pkg/reconciler/pipelinerun/metrics_test.go

pkg/reconciler/stats.go

pkg/reconciler/pipelinerun/pipelinerun.go

pkg/reconciler/taskrun/metrics.go

hrishin · 2019-10-15T13:09:48Z

After this lands let me know if you'd like me to do some of the other metrics (taskruns, etc.)

Yes @imjasonh why not. I will address all review comments today and get this PR in shape (sorry I was for some time)

hrishin · 2019-10-15T13:12:41Z

https://thenewstack.io/opentracing-opencensus-merge-into-a-single-new-project-opentelemetry/

I wonder we should start right away with opentelemetry, or if the new API is too young (it's still alpha)?

@afrittoli thats the valid suggestion. We discussed it in the upstream call. It turned to be very young project but progressively we will make the transition over open telemetry as it becomes consumable

tekton-robot · 2019-10-16T06:38:48Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	77.4%	0.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.6%	83.3%	0.7
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.7%	-0.9
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

tekton-robot · 2019-10-16T08:25:08Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	77.4%	0.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.6%	83.3%	0.7
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.7%	-0.9
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

vdemeester

SGTM for me but commits will need to be squashed 😝

imjasonh

Looks really good, just a couple more fixes/questions.

imjasonh · 2019-10-16T16:10:05Z

pkg/reconciler/pipelinerun/metrics.go

+		return errors.New("ignoring the metrics recording, failed to initialize the metrics recorder")
+	}
+
+	prs, err := lister.List(labels.Everything())


Let's run this in another goroutine, so that this doesn't contribute to reconciliation slowness. I'm glad it's coming from an in-memory cache, but this might still take long enough to matter, especially if some things aren't guaranteed to be in the cache.

pkg/reconciler/pipelinerun/metrics_test.go

pkg/reconciler/taskrun/metrics.go

imjasonh · 2019-10-16T16:13:05Z

pkg/reconciler/taskrun/metrics.go

+		duration = tr.Status.CompletionTime.Sub(tr.Status.StartTime.Time)
+	}
+
+	taskName := "annonymous"


I think the default value should be "". It's possible to name a Task anonymous which would make this misleading.

Ideally yes, exporter drops labels with empty values when I tried initially(will confirm it again).
We need it for when TaskRun embeds Task specification.

kind: TaskRun metadata: generateName: home-volume- spec: taskSpec: steps: - name: write image: ubuntu command: ['bash'] ...

imjasonh · 2019-10-16T16:13:37Z

pkg/reconciler/taskrun/metrics.go

+
+	status := "success"
+	if tr.Status.Conditions[0].Status == corev1.ConditionFalse {
+		status = "failed"


Do you think we might want to break this out to signal timeout, cancelled, and possibly future states?

Feels like yes, we may have to 🤔

pkg/reconciler/taskrun/metrics_test.go

tekton-robot · 2019-10-17T07:06:20Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	77.4%	0.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.6%	83.3%	0.7
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.7%	-0.9
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

tekton-robot · 2019-10-17T07:24:40Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	77.4%	0.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.6%	83.5%	0.8
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.5%	-1.1
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

tekton-robot · 2019-10-17T07:44:07Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	77.4%	0.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.6%	83.5%	0.8
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.5%	-1.1
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

Often, as a developer or administrator(ops) I want some insights about pipeline behavior in terms of time taken to execute pipleinerun/taskrun, its success or failure ratio, pod latencies etc. At present tekton pipelines has very limited ways to surface such information or it's hard to get those details looking at resources yamls. This patch exposes above mentioned pipelines metrics on '/metrics' endpoint using knative `pkg/metrics` package. User can collect such metrics using prometheus, stackdriver or other supported metrics system. To some extent its solves - tektoncd#540 - tektoncd#164

tekton-robot · 2019-10-17T08:08:05Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/pipeline/v1alpha1/taskrun_types.go	76.9%	77.4%	0.5
pkg/reconciler/pipelinerun/metrics.go	Do not exist	86.0%
pkg/reconciler/pipelinerun/pipelinerun.go	82.6%	83.5%	0.8
pkg/reconciler/taskrun/metrics.go	Do not exist	86.7%
pkg/reconciler/taskrun/taskrun.go	71.6%	70.5%	-1.1
test/builder/pod.go	97.4%	79.2%	-18.3
test/builder/task.go	84.3%	83.4%	-0.8

vdemeester

/lgtm

tekton-robot · 2019-10-17T08:44:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

vdemeester · 2019-10-17T09:08:28Z

/test pull-tekton-pipeline-integration-tests

imjasonh

Looks like this got lgtm'ed and merged before addressing some of the comments. We can follow up on those later, but it would be great to have issues to track/assign that work.

imjasonh · 2019-10-18T15:02:47Z

pkg/reconciler/pipelinerun/metrics.go

+
+	status := "success"
+	if pr.Status.Conditions[0].Status == corev1.ConditionFalse {
+		status = "failed"


Can you file an issue to track adding metrics for timeout/cancelled/etc runs?

imjasonh · 2019-10-18T15:04:21Z

pkg/reconciler/taskrun/metrics.go

+		duration = tr.Status.CompletionTime.Sub(tr.Status.StartTime.Time)
+	}
+
+	taskName := "anonymous"


Can you file an issue to track a better solution for this?

In your previous comment you said the exporter drops labels with empty-string values, does that mean no data will be reported at all? Or just that it will be reported without a taskName? Because the latter is probably what we want, assuming we can query for data points that don't report a taskName.

hrishin · 2019-10-19T04:17:27Z

Yes @imjasonh I'm creating the follow-up issue for it. 👍

googlebot added the cla: yes Trying to make the CLA bot happy with ppl from different companies work on one commit label Oct 7, 2019

tekton-robot requested review from abayer and afrittoli October 7, 2019 13:11

tekton-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 7, 2019

hrishin changed the title ~~Adds pipeline metrics~~ Adds pipeline metrics 🔭 Oct 7, 2019

imjasonh requested changes Oct 7, 2019

View reviewed changes

vdemeester reviewed Oct 8, 2019

View reviewed changes

hrishin force-pushed the pipeline-metric branch from 12a9afe to ec8a3cf Compare October 12, 2019 06:33

imjasonh reviewed Oct 14, 2019

View reviewed changes

hrishin force-pushed the pipeline-metric branch from 5764542 to 24afa9b Compare October 16, 2019 06:36

vdemeester reviewed Oct 16, 2019

View reviewed changes

imjasonh reviewed Oct 16, 2019

View reviewed changes

vdemeester added this to the Pipelines 0.8 🐱 milestone Oct 16, 2019

hrishin force-pushed the pipeline-metric branch from 7884ced to 23c9067 Compare October 17, 2019 07:03

hrishin force-pushed the pipeline-metric branch from 23c9067 to 856601c Compare October 17, 2019 07:22

hrishin force-pushed the pipeline-metric branch from bc3fbaf to 34813dc Compare October 17, 2019 08:06

tekton-robot assigned vdemeester Oct 17, 2019

vdemeester approved these changes Oct 17, 2019

View reviewed changes

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 17, 2019

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 17, 2019

tekton-robot merged commit 4bbd797 into tektoncd:master Oct 17, 2019

imjasonh reviewed Oct 18, 2019

View reviewed changes

This was referenced Oct 19, 2019

Metrics name prefix appears as component name of sharedmain.Main() knative/pkg#779

Closed

Pipeline metrics enhancements #1441

Closed

ghost mentioned this pull request Dec 20, 2019

Observability aspects of CD pipelines #855

Closed

bobcatfish mentioned this pull request Feb 21, 2020

Document the metrics exposed by the Pipelines Controller #1965

Closed

Adds pipeline metrics 🔭 #1387

Adds pipeline metrics 🔭 #1387

Conversation

hrishin commented Oct 7, 2019

Changes

Submitter Checklist

Release Notes

tekton-robot commented Oct 7, 2019

tekton-robot commented Oct 7, 2019

tekton-robot commented Oct 7, 2019

imjasonh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdemeester left a comment

Choose a reason for hiding this comment

tekton-robot commented Oct 12, 2019

tekton-robot commented Oct 13, 2019

tekton-robot commented Oct 13, 2019

tekton-robot commented Oct 13, 2019

afrittoli commented Oct 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hrishin commented Oct 15, 2019 • edited Loading

hrishin commented Oct 15, 2019

tekton-robot commented Oct 16, 2019

tekton-robot commented Oct 16, 2019

vdemeester left a comment

Choose a reason for hiding this comment

imjasonh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekton-robot commented Oct 17, 2019

tekton-robot commented Oct 17, 2019

tekton-robot commented Oct 17, 2019

tekton-robot commented Oct 17, 2019

vdemeester left a comment

Choose a reason for hiding this comment

tekton-robot commented Oct 17, 2019

vdemeester commented Oct 17, 2019

imjasonh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hrishin commented Oct 19, 2019

hrishin commented Oct 15, 2019 •

edited

Loading