Support object count quota for custom resources #64201

nikhita · 2018-05-23T13:53:08Z

Fixes #53777
Fixes #59826
Draws a lot of inspiration from #47665

The resource quota controller now uses a dynamic client and a rest mapper to understand custom resources created via CRDs and aggregated apiservers.

To calculate the quota usage, the quota controller needs informers and listers.

A new shared informer is created for custom resources.
The controller also uses a object count quota evaluator to calculate the usage. The evaluator uses a lister to list the resources. So a new lister is created (by creating an indexer) for the custom resource.

Please note that the resource quota controller syncs every 30 seconds, so it might take some time for the quota status to be updated after the CRD is created. Until the quota status is updated and synced, we cannot create any custom resources.

Also, the custom resources created need to be namespaced to work with the quota controller.

Release note:

Object count quota is now supported for namespaced custom resources using the `count/<resource>.<group>` syntax.

k8s-ci-robot · 2018-05-23T13:53:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nikhita
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: deads2k

Assign the PR to them by writing /assign @deads2k in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nikhita · 2018-05-23T13:53:42Z

In action with kubectl:

# create the object count resource quota
$ kubectl create quota test --hard=count/crontabs.example.com=2
resourcequota/test created

# the usage will not be reflected till the CRD is created
$ kubectl get quota test -oyaml
apiVersion: v1
kind: ResourceQuota
metadata:
  creationTimestamp: 2018-05-23T13:39:25Z
  name: test1
  namespace: default
  resourceVersion: "319"
  selfLink: /api/v1/namespaces/default/resourcequotas/test1
  uid: b5826694-5e8e-11e8-8ec1-54e1ad6c2d05
spec:
  hard:
    count/crontabs.example.com: "2"
status:
  hard:
    count/crontabs.example.com: "2"

# create the CRD and wait for sometime (the crd should be available via discovery and the quota controller needs to sync)
$ kubectl create -f crd.yaml
customresourcedefinition.apiextensions.k8s.io/crontabs.example.com created

# the quota now contains the usage
$ kubectl get quota test -oyaml
apiVersion: v1
kind: ResourceQuota
metadata:
  creationTimestamp: 2018-05-23T13:39:37Z
  name: test
  namespace: default
  resourceVersion: "651"
  selfLink: /api/v1/namespaces/default/resourcequotas/test
  uid: bc8d1a09-5e8e-11e8-8ec1-54e1ad6c2d05
spec:
  hard:
    count/crontabs.example.com: "2"
status:
  hard:
    count/crontabs.example.com: "2"
  used:
    count/crontabs.example.com: "0"

# create custom resources
$ kubectl create -f cr1.yaml
crontab.example.com/my-new-cron-object-1 created

$ kubectl create -f cr2.yaml
crontab.example.com/my-new-cron-object-2 created

$ kubectl create -f cr3.yaml
Error from server (Forbidden): error when creating "cr3.yaml": crontabs.example.com "my-new-cron-object-3" is forbidden: exceeded quota: test, requested: count/crontabs.example.com=1, used: count/crontabs.example.com=2, limited: count/crontabs.example.com=2

nikhita · 2018-05-23T13:54:16Z

/cc @sttts @deads2k @derekwaynecarr @liggitt @ironcladlou @caesarxuchao

nikhita · 2018-05-23T13:58:22Z

/sig api-machinery
/sig scheduling
/area custom-resources

The resource quota controller now uses a dynamic client and a rest mapper to understand custom resources created via CRDs and aggregated apiservers. It now uses shared informers. It uses the existing known informer for known resources and creates a new shared informer for custom resources. To calculate the usage in the quota, it needs to create a new object count evaluator for the custom resource. A evaluator contains a lister. So a new lister is created (by creating an indexer) for the custom resource. This lister is then used to create the evaluator. Please note that the resource quota controller syncs every 30 seconds, so it might take some time for the quota status to be updated after the CRD is created. Until the quota status is updated and synced, we cannot create any custom resources.

deads2k · 2018-05-24T12:10:27Z

pkg/controller/resourcequota/resource_quota_controller.go

+	// like those created via CRDs or aggregated apiservers.
+	DynamicClient dynamic.Interface
+	// RESTMapper can reset itself from discovery.
+	RESTMapper resettableRESTMapper


Is the underlying impl for this threadsafe? We use a generally dynamic restmapper in the controller context, but I don't know if it is designed to tolerate concurrent resets.

Is the underlying impl for this threadsafe?

Yes.

kubernetes/staging/src/k8s.io/client-go/restmapper/discovery.go

Lines 210 to 220 in 28f171b

// Reset resets the internally cached Discovery information and will

// cause the next mapping request to re-discover.

func (d *DeferredDiscoveryRESTMapper) Reset() {

glog.V(5).Info("Invalidating discovery information")

d.initMu.Lock()

defer d.initMu.Unlock()

d.cl.Invalidate()

d.delegate = nil

}

deads2k · 2018-05-24T12:11:32Z

pkg/controller/resourcequota/resource_quota_controller.go

-			replenishmentFunc: rq.replenishQuota,
-			registry:          rq.registry,
+			informersStarted:      options.InformersStarted,
+			ignoredResources:      options.IgnoredResourcesFunc(),


This becomes interesting.

// TODO we either find a way to discover this or find a way to provide it via config

Created #64310 to allow it to be provided via config (same as GC)

deads2k · 2018-05-24T12:12:56Z

pkg/controller/resourcequota/resource_quota_controller.go

+		// Resetting the REST mapper will also invalidate the underlying discovery
+		// client. This is a leaky abstraction and assumes behavior about the REST
+		// mapper, but we'll deal with it for now.
+		rq.restMapper.Reset()


I thought we passed a restmapper that was already being periodically refreshed. Why do we need to punch this out?

I think we do, outside of the consuming controllers.

already being periodically refreshed

The Sync is supposed to periodically refresh the restmapper with new discovery information.

The reset occurs within this Sync function.

deads2k · 2018-05-24T12:15:54Z

pkg/controller/resourcequota/resource_quota_controller.go

-		newResources, err := GetQuotableResources(discoveryFunc)
-		if err != nil {
-			utilruntime.HandleError(err)
+		newResources := GetQuotableResources(discoveryClient)


Yeah, it seems like you really want to flush your discovery cache and it doesn't have to do with the restmapper

deads2k · 2018-05-24T12:17:12Z

pkg/controller/resourcequota/resource_quota_controller_test.go

+}
+
+// ServeHTTP logs the action that occurred and always returns the associated status code
+func (f *fakeActionHandler) ServeHTTP(response http.ResponseWriter, request *http.Request) {


this is a lot of mock I didn't expect to see.

the listWatch test doesn't add anything here. That certainly can go, removing some of the mock code.

Removed the listWatch test, along with some others. Weren't really adding too much. This also removed most of the mock code.

Please see this commit: 9247d03. If removing those tests seems fine, I'll squash.

lavalamp · 2018-05-24T20:05:48Z

cc @yliaog

lavalamp · 2018-05-24T20:06:03Z

cc @mbohlool

sttts · 2018-05-25T11:17:10Z

pkg/controller/resourcequota/resource_quota_controller_test.go

+	}
+}
+
+type fakeServerResources struct {


fakeDiscoveryClient

removed the mock code from the quota test but fixed this in the GC test

sttts · 2018-05-25T12:01:30Z

pkg/controller/resourcequota/resource_quota_controller_test.go

+
+// TestQuotaControllerSync ensures that a discovery client error
+// will not cause the quota controller to block infinitely.
+


not: no new line

k8s-ci-robot · 2018-05-25T12:46:35Z

@nikhita: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-kubemark-e2e-gce-big	`d6d5197`	link	`/test pull-kubernetes-kubemark-e2e-gce-big`
pull-kubernetes-e2e-kops-aws	`d6d5197`	link	`/test pull-kubernetes-e2e-kops-aws`
pull-kubernetes-e2e-gce	`d6d5197`	link	`/test pull-kubernetes-e2e-gce`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

nikhita · 2018-05-25T15:04:05Z

For the record: we are not going to target this for 1.11. Since this ends up creating a new informer and lister for custom resources, we are increasing the number of times we cache resources (GC does the same as well).

This is solvable if we have something like dynamic informers and listers. We will first focus on creating them and then refocus on the quota and GC controller. See #64310 (comment) and #64319 for more details.

k8s-ci-robot · 2018-06-07T21:33:32Z

@nikhita: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fejta-bot · 2018-09-05T22:29:49Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

yliaog · 2018-10-01T21:32:57Z

/cc @yliaog

nikhita · 2018-10-03T07:58:23Z

I'm going to close this PR and reopen a proper fix once @p0lyn0mial's PR (#69308) gets in.

p0lyn0mial · 2018-10-11T12:35:39Z

I'm going to close this PR and reopen a proper fix once @p0lyn0mial's PR (#69308) gets in.

@nikhita done :)

raushan2016 · 2019-05-23T00:32:46Z

Is this feature now supported. Where i want to limit number of instance of a CRD created in a namespace like number of knative services created in a namespace ?

liggitt · 2019-05-23T01:00:02Z

Support is merged in master and will be part of 1.15

raushan2016 · 2019-05-23T05:38:48Z

@liggitt can you point me to sample/doc about how to use it for CRD objects

liggitt · 2019-05-23T15:15:10Z

the docs at https://kubernetes.io/docs/concepts/policy/resource-quotas/#object-count-quota are accurate

in 1.15+, to quota a custom resource "widgets" in the "example.com" API group, you would use count/widgets.example.com

liggitt · 2019-05-23T15:19:49Z

opened doc PR for the 1.15 branch in kubernetes/website#14492

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 23, 2018

k8s-ci-robot requested review from deads2k and derekwaynecarr May 23, 2018 13:53

k8s-ci-robot requested review from caesarxuchao, ironcladlou, liggitt and sttts May 23, 2018 13:54

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. area/custom-resources labels May 23, 2018

nikhita force-pushed the cr-quota branch 2 times, most recently from 0756a8e to 3b592a8 Compare May 24, 2018 06:03

nikhita added 2 commits May 24, 2018 11:36

quota: update bazel

3b592a8

deads2k reviewed May 24, 2018

View reviewed changes

nikhita mentioned this pull request May 24, 2018

Add support for ResourceQuota over ServiceInstances and ServiceInstanceCredentials kubernetes-retired/service-catalog#1197

Closed

sttts reviewed May 25, 2018

View reviewed changes

address comments

9247d03

update bazel

d6d5197

sttts reviewed May 25, 2018

View reviewed changes

This was referenced May 25, 2018

quota: allow ignored resources to be customized #64310

Closed

Create dynamic informers and listers #64319

Closed

liggitt mentioned this pull request May 29, 2018

Limits around the number of CRs kubernetes/community#798

Closed

This was referenced May 30, 2018

Resource Quota controller should allow discover-resources errors #59826

Closed

Umbrella issue for CRDs moving to GA #58682

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 7, 2018

ecordell mentioned this pull request Aug 14, 2018

Support For Cluster Addons operator-framework/operator-sdk#400

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 5, 2018

jingxu97 mentioned this pull request Sep 28, 2018

quota support for custom resources #53777

Closed

k8s-ci-robot requested a review from yliaog October 1, 2018 21:32

nikhita closed this Oct 3, 2018

zhouhaibing089 mentioned this pull request Dec 27, 2018

resourcequota: use dynamic informer #72384

Merged

liggitt mentioned this pull request May 23, 2019

Add custom resource quota example kubernetes/website#14492

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support object count quota for custom resources #64201

Support object count quota for custom resources #64201

nikhita commented May 23, 2018 •

edited

Loading

k8s-ci-robot commented May 23, 2018

nikhita commented May 23, 2018

nikhita commented May 23, 2018

nikhita commented May 23, 2018

deads2k May 24, 2018

nikhita May 25, 2018 •

edited

Loading

deads2k May 24, 2018

nikhita May 25, 2018

nikhita May 25, 2018

deads2k May 24, 2018

sttts May 24, 2018

nikhita May 25, 2018

deads2k May 24, 2018

deads2k May 24, 2018

sttts May 25, 2018

nikhita May 25, 2018 •

edited

Loading

lavalamp commented May 24, 2018

lavalamp commented May 24, 2018

sttts May 25, 2018

nikhita May 25, 2018

sttts May 25, 2018

k8s-ci-robot commented May 25, 2018 •

edited

Loading

nikhita commented May 25, 2018

k8s-ci-robot commented Jun 7, 2018

fejta-bot commented Sep 5, 2018

yliaog commented Oct 1, 2018

nikhita commented Oct 3, 2018

p0lyn0mial commented Oct 11, 2018

raushan2016 commented May 23, 2019

liggitt commented May 23, 2019

raushan2016 commented May 23, 2019

liggitt commented May 23, 2019

liggitt commented May 23, 2019

	// Reset resets the internally cached Discovery information and will
	// cause the next mapping request to re-discover.
	func (d *DeferredDiscoveryRESTMapper) Reset() {
	glog.V(5).Info("Invalidating discovery information")

	d.initMu.Lock()
	defer d.initMu.Unlock()

	d.cl.Invalidate()
	d.delegate = nil
	}


		// TestQuotaControllerSync ensures that a discovery client error
		// will not cause the quota controller to block infinitely.

Support object count quota for custom resources #64201

Support object count quota for custom resources #64201

Conversation

nikhita commented May 23, 2018 • edited Loading

k8s-ci-robot commented May 23, 2018

nikhita commented May 23, 2018

nikhita commented May 23, 2018

nikhita commented May 23, 2018

Choose a reason for hiding this comment

nikhita May 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikhita May 25, 2018 • edited Loading

Choose a reason for hiding this comment

lavalamp commented May 24, 2018

lavalamp commented May 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 25, 2018 • edited Loading

nikhita commented May 25, 2018

k8s-ci-robot commented Jun 7, 2018

fejta-bot commented Sep 5, 2018

yliaog commented Oct 1, 2018

nikhita commented Oct 3, 2018

p0lyn0mial commented Oct 11, 2018

raushan2016 commented May 23, 2019

liggitt commented May 23, 2019

raushan2016 commented May 23, 2019

liggitt commented May 23, 2019

liggitt commented May 23, 2019

nikhita commented May 23, 2018 •

edited

Loading

nikhita May 25, 2018 •

edited

Loading

nikhita May 25, 2018 •

edited

Loading

k8s-ci-robot commented May 25, 2018 •

edited

Loading