csi: Implement NodeServiceCapability_RPC_GET_VOLUME_STATS rpc call #76188

humblec · 2019-04-05T09:52:34Z

Signed-off-by: Humble Chirammal hchiramm@redhat.com

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Expose CSI volume stats via kubelet volume metrics

humblec · 2019-04-05T09:58:55Z

/retest

humblec · 2019-04-05T10:09:03Z

/assign @gnufied

humblec · 2019-04-05T10:10:38Z

/release-note-none

humblec · 2019-04-05T12:00:51Z

@gnufied Can you please take a look at this PR ? :)

humblec · 2019-04-05T14:01:23Z

/assign @rootfs

gnufied · 2019-05-14T17:00:56Z

/assign @msau42

msau42 · 2019-05-14T17:37:32Z

pkg/volume/csi/csi_client.go

@@ -841,3 +849,94 @@ func (c *csiClientGetter) Get() (csiClient, error) {
 	c.csiClient = csi
 	return c.csiClient, nil
 }
+
+func (c *csiDriverClient) NodeSupportsVolumeStats(ctx context.Context) (bool, error) {
+	klog.V(4).Info(log("calling NodeGetCapabilities rpc to determine if NodeSupportsVolumeStats"))


do we really need this log? Can we make it higher like 5?

lifted to V(5).

msau42 · 2019-05-14T17:39:03Z

pkg/volume/csi/csi_client.go

+}
+
+func (c *csiDriverClient) NodeGetVolumeStats(ctx context.Context, volID string, targetPath string) (*volume.Metrics, error) {
+	klog.V(4).Info(log("calling NodeGetVolumeStats rpc: [volid=%s, target_path=%s", volID, targetPath))


How often is this called?

As often as prometheus is configured to scrap the metrics endpoint.

unable to scrap any metric for aws efs csi , not sure whether rpc call was executed or not.

msau42 · 2019-05-14T17:44:53Z

pkg/volume/csi/csi_client.go

+	for _, usage := range usages {
+		unit := usage.GetUnit()
+		switch unit.String() {
+		case "BYTES":


Should this use the consts defined here: https://github.com/kubernetes/kubernetes/blob/ce1e86fb99a8e36912ebb48b226ebb1bc6a99797/vendor/github.com/container-storage-interface/spec/lib/go/csi/csi.pb.go#L235

Converted to consts.

msau42 · 2019-05-14T18:32:40Z

pkg/volume/csi/csi_client.go

+			metrics.Inodes = resource.NewQuantity(usage.GetTotal(), resource.BinarySI)
+			metrics.InodesUsed = resource.NewQuantity(usage.GetUsed(), resource.BinarySI)
+		default:
+			klog.Errorf("unknown key %s in usage", unit.String())


This means that CSI spec added more that we don't support yet?

Should it be a warning, or even Info at level 5 if we expect this to flood logs?

This should ideally never happen and if it does, it is probably an error and worth either fixing k8s or CSI driver.

msau42 · 2019-05-14T18:43:11Z

pkg/volume/csi/csi_plugin.go

@@ -408,6 +408,8 @@ func (p *csiPlugin) NewMounter(
 	}
 	klog.V(4).Info(log("created path successfully [%s]", dataDir))

+	mounter.MetricsProvider = NewMetricsCsi(volumeHandle, dataDir)


Should the path be dir or dataDir? dir is the one we pass as the targetPath to NodePublish?

good catch, this indeed should be dir and not dataDir.

Can we make sure to validate this with a real CSI driver?

Replaced the path.

Can we make sure to validate this with a real CSI driver?

Yes, thats the very next plan. I dont think, the CSI drivers exist which expose the metrics. I have started the work for Ceph-CSI any way.

This can be also tested via CSI mock driver fwiw. Mock driver can implement NodeGetVolumeStats RPC call and it can check for volume path to be same as publish path and return error if it isn't.

This has to go in an e2e though. @humblec can you add a github issue for creating an e2e for this?

Sure @gnufied . I am opening an issue for e2e tracking as discussed earlier..

@gnufied The E2E issue is here #77933 . Thanks !

humblec · 2019-05-15T04:21:58Z

@msau42 the final review comments are addressed. PTAL .. Thanks !!

gnufied · 2019-05-15T18:26:29Z

@humblec this PR was discussed in wg-csi-implementation stadnup and it was agreed that we should try and manually validate if CSI volume stats are being available through kubelet metrics if using a real CSI driver (even mock or hostpath driver would do).

We can do e2e as a follow up but could we do the manual validation for ANY CSI driver?

msau42 · 2019-05-15T23:33:36Z

/approve
/hold
for testing with a real csi driver

k8s-ci-robot · 2019-05-15T23:33:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: humblec, msau42

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/volume/csi/OWNERS~~ [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

humblec · 2019-05-17T04:02:57Z

@humblec this PR was discussed in wg-csi-implementation stadnup and it was agreed that we should try and manually validate if CSI volume stats are being available through kubelet metrics if using a real CSI driver (even mock or hostpath driver would do).
We can do e2e as a follow up but could we do the manual validation for ANY CSI driver?

Sure @gnufied and @msau42 . I patched hostpath driver to return a handcrafted response for this RPC call ( also note/confirm the path /var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount" in the request) and tested out. The final result is that, I could see the metrics in kubelet. Below is an example output for the RPC call and also the metrics in kubelet.

Logs from CSI driver:

I0516 15:55:17.079907       1 server.go:117] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0516 15:55:17.079920       1 server.go:118] GRPC request: {"volume_id":"c0419fd7-77f2-11e9-b47e-c85b7636c232","volume_path":"/var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount"}
I0516 15:55:17.080500       1 mount_linux.go:164] Detected OS without systemd
I0516 15:55:17.080510       1 nodeserver.go:302] VolumeStats: stats on targetpath/volumeID /var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount/c0419fd7-77f2-11e9-b47e-c85b7636c232 has been requested.
E0516 15:55:17.080519       1 nodeserver.go:304] VolumeStats: stats on targetpath/volumeID /var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount/c0419fd7-77f2-11e9-b47e-c85b7636c232 has been requested.
I0516 15:55:17.080784       1 nodeserver.go:316] targetpath /var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount: already mounted
I0516 15:55:17.080794       1 server.go:123] GRPC response: {"usage":[{"total":10,"unit":1}]}

Logs/stats colllected in kubelet!


kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="csi-pvc"} 0
kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="csi-pvc"} 10
kubelet_volume_stats_inodes{namespace="default",persistentvolumeclaim="csi-pvc"} 0
kubelet_volume_stats_inodes_free{namespace="default",persistentvolumeclaim="csi-pvc"} 0
kubelet_volume_stats_inodes_used{namespace="default",persistentvolumeclaim="csi-pvc"} 0
kubelet_volume_stats_used_bytes{namespace="default",persistentvolumeclaim="csi-pvc"} 0

Hope this is good to go in and declare the metrics support in CSI !!.. Thanks a lot!

and implement Metrics Provider for CSI driver Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

gnufied · 2019-05-17T14:16:59Z

/hold cancel
/lgtm

fejta-bot · 2019-05-17T21:09:44Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

Justification: Maintainer/Owner of kubernetes volume plugins: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/glusterfs/OWNERS https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/iscsi/OWNERS Lead contributor @ https://github.com/kubernetes-incubator/external-storage/graphs/contributors Contributed CSI features like metrics support: kubernetes/kubernetes#76188 Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

Davidrjx · 2021-05-31T12:18:18Z

@humblec this PR was discussed in wg-csi-implementation stadnup and it was agreed that we should try and manually validate if CSI volume stats are being available through kubelet metrics if using a real CSI driver (even mock or hostpath driver would do).
We can do e2e as a follow up but could we do the manual validation for ANY CSI driver?

Sure @gnufied and @msau42 . I patched hostpath driver to return a handcrafted response for this RPC call ( also note/confirm the path /var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount" in the request) and tested out. The final result is that, I could see the metrics in kubelet. Below is an example output for the RPC call and also the metrics in kubelet.

Logs from CSI driver:
I0516 15:55:17.079907       1 server.go:117] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0516 15:55:17.079920       1 server.go:118] GRPC request: {"volume_id":"c0419fd7-77f2-11e9-b47e-c85b7636c232","volume_path":"/var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount"}
I0516 15:55:17.080500       1 mount_linux.go:164] Detected OS without systemd
I0516 15:55:17.080510       1 nodeserver.go:302] VolumeStats: stats on targetpath/volumeID /var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount/c0419fd7-77f2-11e9-b47e-c85b7636c232 has been requested.
E0516 15:55:17.080519       1 nodeserver.go:304] VolumeStats: stats on targetpath/volumeID /var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount/c0419fd7-77f2-11e9-b47e-c85b7636c232 has been requested.
I0516 15:55:17.080784       1 nodeserver.go:316] targetpath /var/lib/kubelet/pods/4891f5a7-2c72-4f33-8e4b-88ba63b5eb1a/volumes/kubernetes.io~csi/pvc-b1e01c02-063a-467c-9f88-621cadbf780f/mount: already mounted
I0516 15:55:17.080794       1 server.go:123] GRPC response: {"usage":[{"total":10,"unit":1}]}
Logs/stats colllected in kubelet!
kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="csi-pvc"} 0
kubelet_volume_stats_capacity_bytes{namespace="default",persistentvolumeclaim="csi-pvc"} 10
kubelet_volume_stats_inodes{namespace="default",persistentvolumeclaim="csi-pvc"} 0
kubelet_volume_stats_inodes_free{namespace="default",persistentvolumeclaim="csi-pvc"} 0
kubelet_volume_stats_inodes_used{namespace="default",persistentvolumeclaim="csi-pvc"} 0
kubelet_volume_stats_used_bytes{namespace="default",persistentvolumeclaim="csi-pvc"} 0
Hope this is good to go in and declare the metrics support in CSI !!.. Thanks a lot!

hi @humblec
did you hard-coded specific pv in driver and execute GetVolumeStats rpc call?

k8s-ci-robot requested review from gnufied and rootfs April 5, 2019 09:52

k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Apr 5, 2019

humblec force-pushed the csi-in-m branch from 6a2a39f to c16747d Compare April 5, 2019 10:07

k8s-ci-robot assigned gnufied Apr 5, 2019

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 5, 2019

humblec force-pushed the csi-in-m branch 2 times, most recently from 0e44d82 to 9dca72a Compare April 5, 2019 10:29

humblec changed the title ~~csi: Add capability check for NodeServiceCapability_RPC_GET_VOLUME_STATS~~ csi: Implement NodeServiceCapability_RPC_GET_VOLUME_STATS rpc call Apr 5, 2019

humblec force-pushed the csi-in-m branch from 70bfdd4 to 94b8697 Compare April 5, 2019 11:46

k8s-ci-robot assigned rootfs Apr 5, 2019

humblec force-pushed the csi-in-m branch from 94b8697 to 1f87855 Compare April 5, 2019 14:52

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 5, 2019

humblec force-pushed the csi-in-m branch 3 times, most recently from f6607c0 to e6604b0 Compare April 6, 2019 10:32

k8s-ci-robot assigned msau42 May 14, 2019

msau42 reviewed May 14, 2019

View reviewed changes

humblec force-pushed the csi-in-m branch from ce1e86f to a361a87 Compare May 15, 2019 04:10

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 15, 2019

humblec mentioned this pull request May 15, 2019

Add E2E test for CSI Volume Stats RPC and metrics provider #77933

Closed

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 15, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 15, 2019

humblec force-pushed the csi-in-m branch from a361a87 to 656d83b Compare May 17, 2019 03:57

csi: Implement NodeServiceCapability_RPC_GET_VOLUME_STATS rpc call

c511c90

and implement Metrics Provider for CSI driver Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

humblec force-pushed the csi-in-m branch from 656d83b to c511c90 Compare May 17, 2019 11:44

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels May 17, 2019

k8s-ci-robot merged commit fad34e4 into kubernetes:master May 18, 2019

cloudbehl mentioned this pull request May 31, 2019

Capacity Metrics in PVs and PVCs openshift/console#1258

Closed

anmolsachan mentioned this pull request Jun 4, 2019

Implement metrics for cephfs CSI driver. ceph/ceph-csi#310

Merged

humblec mentioned this pull request Aug 1, 2019

Invoke MetricsProvider in CSI for block Volume Mode PVCs #80865

Closed

Thakurvaibhav mentioned this pull request Dec 16, 2019

Can't get metrics of PersistentVolumeClaim Thakurvaibhav/k8s#13

Closed

humblec mentioned this pull request Apr 1, 2020

Add humblec to kubernetes-csi org kubernetes/org#1766

Merged

ghouscht mentioned this pull request Apr 8, 2020

Expose kubelet_volume* metrics NetApp/trident#134

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

csi: Implement NodeServiceCapability_RPC_GET_VOLUME_STATS rpc call #76188

csi: Implement NodeServiceCapability_RPC_GET_VOLUME_STATS rpc call #76188

humblec commented Apr 5, 2019 •

edited

humblec commented Apr 5, 2019

humblec commented Apr 5, 2019

humblec commented Apr 5, 2019

humblec commented Apr 5, 2019

humblec commented Apr 5, 2019

gnufied commented May 14, 2019

msau42 May 14, 2019

humblec May 15, 2019

msau42 May 14, 2019

gnufied May 14, 2019

Davidrjx May 31, 2021

msau42 May 14, 2019

humblec May 15, 2019

msau42 May 14, 2019

gnufied May 14, 2019

msau42 May 14, 2019

gnufied May 14, 2019

msau42 May 14, 2019

humblec May 15, 2019

gnufied May 15, 2019

humblec May 15, 2019

humblec May 15, 2019

humblec commented May 15, 2019

gnufied commented May 15, 2019

msau42 commented May 15, 2019

k8s-ci-robot commented May 15, 2019

humblec commented May 17, 2019 •

edited

gnufied commented May 17, 2019

fejta-bot commented May 17, 2019

Davidrjx commented May 31, 2021

csi: Implement NodeServiceCapability_RPC_GET_VOLUME_STATS rpc call #76188

csi: Implement NodeServiceCapability_RPC_GET_VOLUME_STATS rpc call #76188

Conversation

humblec commented Apr 5, 2019 • edited

humblec commented Apr 5, 2019

humblec commented Apr 5, 2019

humblec commented Apr 5, 2019

humblec commented Apr 5, 2019

humblec commented Apr 5, 2019

gnufied commented May 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humblec commented May 15, 2019

gnufied commented May 15, 2019

msau42 commented May 15, 2019

k8s-ci-robot commented May 15, 2019

humblec commented May 17, 2019 • edited

gnufied commented May 17, 2019

fejta-bot commented May 17, 2019

Davidrjx commented May 31, 2021

humblec commented Apr 5, 2019 •

edited

humblec commented May 17, 2019 •

edited