Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change kubelet probe metrics to counter #76074

Merged
merged 2 commits into from Apr 12, 2019

Conversation

@danielqsj
Copy link
Member

commented Apr 3, 2019

What type of PR is this?

/kind feature
/sig instrumentation node

What this PR does / why we need it:

As discussion in #75839, we prefer to using counter type of metrics for kubelet probe rather than gauge type.

Which issue(s) this PR fixes:

Fixes #75839

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Change kubelet probe metrics to counter type.
The metrics `prober_probe_result` is replaced by `prober_probe_total`.

@danielqsj danielqsj force-pushed the danielqsj:probe branch from 4b53d1f to 295d672 Apr 3, 2019

@danielqsj

This comment has been minimized.

Copy link
Member Author

commented Apr 3, 2019

@k8s-ci-robot k8s-ci-robot requested review from brancz and logicalhan Apr 3, 2019

@brancz

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

Given that none of the popular alert/dashboard definitions use this metric, I'm inclined to just remove the old metric.

@logicalhan
Copy link
Contributor

left a comment

I'm okay with just deleting the old metric as well. As mentioned in the issue, I would be surprised if anyone is actually using it for alerting since it is probably pretty noisy.

@@ -98,16 +101,27 @@ func newWorker(
w.initialValue = results.Success
}

w.proberResultsMetricLabels = prometheus.Labels{
"probe_type": w.probeType.String(),
"container_name": w.container.Name,

This comment has been minimized.

Copy link
@logicalhan

logicalhan Apr 3, 2019

Contributor

Woah, this is weird. I realize you didn't write this, but curious if anyone knows why we would have multiple labels on a metric which would always share the same label value?

This comment has been minimized.

Copy link
@brancz

brancz Apr 3, 2019

Member

Are you talking about container and contaiber_name?

This comment has been minimized.

Copy link
@logicalhan

logicalhan Apr 3, 2019

Contributor

Either. The previous implementation of this seems to store the same value for in both the container label and the container_name label. Same thing happens for pod and pod_name, which I find quite odd.

This comment has been minimized.

Copy link
@logicalhan

logicalhan Apr 3, 2019

Contributor

You can verify it by outputting the actual metric output via curl.

prober_probe_result{container="etcd-container",container_name="etcd-container",namespace="kube-system",pod="etcd-server-events-kubernetes-master",pod_name="etcd-server-events-kubernetes-master",pod_uid="1234",probe_type="Liveness"} 0

Why do we have two labels which always have the same value in either of them?

This comment has been minimized.

Copy link
@logicalhan

logicalhan Apr 3, 2019

Contributor

Super weird.

This comment has been minimized.

Copy link
@brancz

brancz Apr 3, 2019

Member

This was for migration purposes. For a long time these and cadvisor metrics used pod_name and container_name label keys, which violate the instrumentation guidelines. For migration purposes in 1.14 both are present.

This comment has been minimized.

Copy link
@logicalhan

logicalhan Apr 4, 2019

Contributor

Ah, that would explain it.

@danielqsj

This comment has been minimized.

Copy link
Member Author

commented Apr 4, 2019

@brancz @logicalhan removed old metrics prober_probe_result. PTAL

@brancz

This comment has been minimized.

Copy link
Member

commented Apr 4, 2019

looks good from instrumentation side

/lgtm

still needs a kubelet approver though @derekwaynecarr @tallclair

@logicalhan
Copy link
Contributor

left a comment

/lgtm

@dashpole, thoughts?

@dashpole

This comment has been minimized.

Copy link
Contributor

commented Apr 5, 2019

/lgtm
This looks reasonable to me.

@danielqsj

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2019

/retest

@danielqsj

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2019

friendly ping @derekwaynecarr @tallclair , could you please help review it?

@tallclair

This comment has been minimized.

Copy link
Member

commented Apr 11, 2019

/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danielqsj, tallclair

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fejta-bot

This comment has been minimized.

Copy link

commented Apr 12, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@danielqsj

This comment has been minimized.

Copy link
Member Author

commented Apr 12, 2019

/retest

@fejta-bot

This comment has been minimized.

Copy link

commented Apr 12, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@brancz

This comment has been minimized.

Copy link
Member

commented Apr 12, 2019

/retest

@k8s-ci-robot k8s-ci-robot merged commit b7858e3 into kubernetes:master Apr 12, 2019

18 checks passed

cla/linuxfoundation danielqsj authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-godeps Context retired. Status moved to "pull-kubernetes-dependencies".
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.