New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add number measurement for bound/unbound pv/pvc #57872

Merged
merged 2 commits into from Feb 6, 2018

Conversation

@mlmhl
Contributor

mlmhl commented Jan 5, 2018

What this PR does / why we need it:

Implement number measurement for bound/unbound pv/pvc defined in the Metrics Spec

ref feature: kubernetes/features#496

Release note:

Intended for post-1.9
@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 5, 2018

/sig storage

@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 5, 2018

/cc @gnufied

@k8s-ci-robot k8s-ci-robot requested a review from gnufied Jan 5, 2018

@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 5, 2018

/assign @thockin

@spiffxp

This comment has been minimized.

Member

spiffxp commented Jan 7, 2018

/ok-to-test

pvControllerSubsystem = "pv_collector"
// Metric names.
boundPvKey = "bound_pv_count"

This comment has been minimized.

@jsafrane

jsafrane Jan 9, 2018

Member

boundPVKey

(and everywhere else Pv -> PV, Pvc -> PVC in indentifiers)

This comment has been minimized.

@mlmhl

mlmhl Jan 10, 2018

Contributor

Done

@jsafrane

This comment has been minimized.

Member

jsafrane commented Jan 9, 2018

/assign @gnufied

I have only annoying comments about variable and function names.

@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 10, 2018

/retest

storageClassName)
}
for storageClassName, number := range unboundNumberByStorageClass {
ch <- prometheus.MustNewConstMetric(

This comment has been minimized.

@gnufied

gnufied Jan 10, 2018

Member

Is there any particular reason you chose to use MustNewConstMetric? The documentation of this function implies that this type is most useful for "throwaway" metrics.

This comment has been minimized.

@mlmhl

mlmhl Jan 11, 2018

Contributor

Use MustNewConstMetric here just to make the code simple as we needn't to handle the error returned by NewConstMetric, but it pushed the volume controller into an unstable state, so I changed all these functions to NewConstMetric, PTAL.

@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 11, 2018

/retest

1 similar comment
@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 11, 2018

/retest

@gnufied

This comment has been minimized.

Member

gnufied commented Jan 11, 2018

@mlmhl can you update existing e2e tests to cover these metrics https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/volume_metrics.go ?

@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 11, 2018

@gnufied OK, I will update e2e tests to cover these metrics.

@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 12, 2018

@gnufied e2e tests already added for these metrics, PTAL.

By the way, I intend to add total provision/deletion time metrics after this PR, but I'm not sure the exact definition of total Provision/Deletion time. According to my understanding, the total provision time starts from the PVC created and end to the PV created, the total deletion time starts from the PVC deleted and end to the PV deleted. Please let me know if I understand something wrong.

pv, err = framework.CreatePV(c, pv)
Expect(err).NotTo(HaveOccurred(), "Error creating pv: %v", err)
waitForPVControllerSync(metricsGrabber, unboundPVKey, classKey)
validator([]map[string]int64{nil, {className: 1}, nil, nil})

This comment has been minimized.

@gnufied

gnufied Jan 23, 2018

Member

These metrics appear to be checking absolute number of bound or unbound PVs. Will this not fail when some other PV might exist in the cluster while this test is running?

This comment has been minimized.

@mlmhl

mlmhl Jan 24, 2018

Contributor

All volume metric e2e tests are labeled as [Serial](see here), so we can consider that no other existing PVs while this test is running.

This comment has been minimized.

@gnufied

gnufied Jan 24, 2018

Member

Yes - I know but reality is stranger than that. I have fixed a number of flakes in this test suite because something else caused metrics to jump around. We have to be careful, and only observe increment in metric values, rather than absolute values - because asserting on absolute values is almost sure to be error prone.

This comment has been minimized.

@mlmhl

mlmhl Jan 24, 2018

Contributor

ACK. This is indeed a problem as some other tests maybe create PV/PVCs and forget to cleanup. I will change to use increment instead of absolute values.

This comment has been minimized.

@mlmhl

mlmhl Jan 24, 2018

Contributor

@gnufied The e2e tests are updated to validate the relative increment value instead of absolute value, PTAL, thanks.

@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 24, 2018

/retest

1 similar comment
@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Jan 24, 2018

/retest

@gnufied

This comment has been minimized.

Member

gnufied commented Feb 2, 2018

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Feb 2, 2018

@mlmhl

This comment has been minimized.

Contributor

mlmhl commented Feb 3, 2018

/retest

@thockin

This comment has been minimized.

Member

thockin commented Feb 6, 2018

/approve

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Feb 6, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gnufied, mlmhl, thockin

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-merge-robot

This comment has been minimized.

Contributor

k8s-merge-robot commented Feb 6, 2018

Automatic merge from submit-queue (batch tested with PRs 58317, 58687, 57872, 59063, 59328). If you want to cherry-pick this change to another branch, please follow the instructions here.

@k8s-merge-robot k8s-merge-robot merged commit 997fe31 into kubernetes:master Feb 6, 2018

15 checks passed

Submit Queue Queued to run github e2e tests a second time.
Details
cla/linuxfoundation mlmhl authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-gke-gci Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce-canary Skipped
pull-kubernetes-kubemark-e2e-gce Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-unit Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details

@mlmhl mlmhl deleted the mlmhl:volume_metric_bound_pvc branch Feb 7, 2018

k8s-merge-robot added a commit that referenced this pull request Feb 14, 2018

Merge pull request #59523 from gnufied/revert-57872-volume_metric_bou…
…nd_pvc

Automatic merge from submit-queue (batch tested with PRs 57445, 59523). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Revert "add number measurement for bound/unbound pv/pvc"

Reverts #57872

Fixes : #59517
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment