Opaque integer resources #76

ConnorDoyle · 2016-08-24T23:00:47Z

This feature provides a simple way to advertise some amount of arbitrary countable resource for a node in a Kubernetes cluster. Users can consume these resources by including them in pod specs, and the scheduler takes them into account when placing pods on nodes.

Issues:

Related:

Cluster Scoped Resources kubernetes#19080

Progress Tracker

FEATURE_STATUS is used for feature tracking and to be updated by @kubernetes/feature-reviewers.
FEATURE_STATUS: IN_DEVELOPMENT

More advice:

Design

Once you get LGTM from a @kubernetes/feature-reviewers member, you can check this checkbox, and the reviewer will apply the "design-complete" label.

Coding

Use as many PRs as you need. Write tests in the same or different PRs, as is convenient for you.
As each PR is merged, add a comment to this issue referencing the PRs. Code goes in the http://github.com/kubernetes/kubernetes repository,
and sometimes http://github.com/kubernetes/contrib, or other repos.
When you are done with the code, apply the "code-complete" label.
When the feature has user docs, please add a comment mentioning @kubernetes/feature-reviewers and they will
check that the code matches the proposed feature and design, and that everything is done, and that there is adequate
testing. They won't do detailed code review: that already happened when your PRs were reviewed.
When that is done, you can check this box and the reviewer will apply the "code-complete" label.

Docs

Write user docs and get them merged in.
User docs go into http://github.com/kubernetes/kubernetes.github.io.
When the feature has user docs, please add a comment mentioning @kubernetes/docs.
When you get LGTM, you can check this checkbox, and the reviewer will apply the "docs-complete" label.

The text was updated successfully, but these errors were encountered:

ConnorDoyle · 2016-08-24T23:04:55Z

cc @davidopp @balajismaniam

ConnorDoyle · 2016-09-15T23:17:38Z

@davidopp could you please mark this for 1.5?

idvoretskyi · 2016-09-29T14:10:24Z

@ConnorDoyle any progress on this feature? Which SIG is responsible for it?

ConnorDoyle · 2016-09-29T15:46:02Z

@idvoretskyi the current progress is mostly discussed in the in-flight PR (31652). The feature spans sig-scheduling and sig-node, it's on the 1.5 roadmaps for both. Since most of the work is in the scheduler @timothysc has volunteered to shepherd the feature. Please let me know if you need more info.

idvoretskyi · 2016-10-06T11:12:48Z

@ConnorDoyle thank you.

@davidopp

Automatic merge from submit-queue [PHASE 1] Opaque integer resource accounting. ## [PHASE 1] Opaque integer resource accounting. This change provides a simple way to advertise some amount of arbitrary countable resource for a node in a Kubernetes cluster. Users can consume these resources by including them in pod specs, and the scheduler takes them into account when placing pods on nodes. See the example at the bottom of the PR description for more info. Summary of changes: - Defines opaque integer resources as any resource with prefix `pod.alpha.kubernetes.io/opaque-int-resource-`. - Prevent kubelet from overwriting capacity. - Handle opaque resources in scheduler. - Validate integer-ness of opaque int quantities in API server. - Tests for above. Feature issue: kubernetes/enhancements#76 Design: http://goo.gl/IoKYP1 Issues: #28312 #19082 Related: #19080 CC @davidopp @timothysc @balajismaniam **Release note**:  ```release-note Added support for accounting opaque integer resources. Allows cluster operators to advertise new node-level resources that would be otherwise unknown to Kubernetes. Users can consume these resources in pod specs just like CPU and memory. The scheduler takes care of the resource accounting so that no more than the available amount is simultaneously allocated to pods. ``` ## Usage example ```sh $ echo '[{"op": "add", "path": "pod.alpha.kubernetes.io~1opaque-int-resource-bananas", "value": "555"}]' | \ > http PATCH http://localhost:8080/api/v1/nodes/localhost.localdomain/status \ > Content-Type:application/json-patch+json ``` ```http HTTP/1.1 200 OK Content-Type: application/json Date: Thu, 11 Aug 2016 16:44:55 GMT Transfer-Encoding: chunked { "apiVersion": "v1", "kind": "Node", "metadata": { "annotations": { "volumes.kubernetes.io/controller-managed-attach-detach": "true" }, "creationTimestamp": "2016-07-12T04:07:43Z", "labels": { "beta.kubernetes.io/arch": "amd64", "beta.kubernetes.io/os": "linux", "kubernetes.io/hostname": "localhost.localdomain" }, "name": "localhost.localdomain", "resourceVersion": "12837", "selfLink": "/api/v1/nodes/localhost.localdomain/status", "uid": "2ee9ea1c-47e6-11e6-9fb4-525400659b2e" }, "spec": { "externalID": "localhost.localdomain" }, "status": { "addresses": [ { "address": "10.0.2.15", "type": "LegacyHostIP" }, { "address": "10.0.2.15", "type": "InternalIP" } ], "allocatable": { "alpha.kubernetes.io/nvidia-gpu": "0", "cpu": "2", "memory": "8175808Ki", "pods": "110" }, "capacity": { "alpha.kubernetes.io/nvidia-gpu": "0", "pod.alpha.kubernetes.io/opaque-int-resource-bananas": "555", "cpu": "2", "memory": "8175808Ki", "pods": "110" }, "conditions": [ { "lastHeartbeatTime": "2016-08-11T16:44:47Z", "lastTransitionTime": "2016-07-12T04:07:43Z", "message": "kubelet has sufficient disk space available", "reason": "KubeletHasSufficientDisk", "status": "False", "type": "OutOfDisk" }, { "lastHeartbeatTime": "2016-08-11T16:44:47Z", "lastTransitionTime": "2016-07-12T04:07:43Z", "message": "kubelet has sufficient memory available", "reason": "KubeletHasSufficientMemory", "status": "False", "type": "MemoryPressure" }, { "lastHeartbeatTime": "2016-08-11T16:44:47Z", "lastTransitionTime": "2016-08-10T06:27:11Z", "message": "kubelet is posting ready status", "reason": "KubeletReady", "status": "True", "type": "Ready" }, { "lastHeartbeatTime": "2016-08-11T16:44:47Z", "lastTransitionTime": "2016-08-10T06:27:01Z", "message": "kubelet has no disk pressure", "reason": "KubeletHasNoDiskPressure", "status": "False", "type": "DiskPressure" } ], "daemonEndpoints": { "kubeletEndpoint": { "Port": 10250 } }, "images": [], "nodeInfo": { "architecture": "amd64", "bootID": "1f7e95ca-a4c2-490e-8ca2-6621ae1eb5f0", "containerRuntimeVersion": "docker://1.10.3", "kernelVersion": "4.5.7-202.fc23.x86_64", "kubeProxyVersion": "v1.3.0-alpha.4.4285+7e4b86c96110d3-dirty", "kubeletVersion": "v1.3.0-alpha.4.4285+7e4b86c96110d3-dirty", "machineID": "cac4063395254bc89d06af5d05322453", "operatingSystem": "linux", "osImage": "Fedora 23 (Cloud Edition)", "systemUUID": "D6EE0782-5DEB-4465-B35D-E54190C5EE96" } } } ``` After patching, the kubelet's next sync fills in allocatable: ``` $ kubectl get node localhost.localdomain -o json | jq .status.allocatable ``` ```json { "alpha.kubernetes.io/nvidia-gpu": "0", "pod.alpha.kubernetes.io/opaque-int-resource-bananas": "555", "cpu": "2", "memory": "8175808Ki", "pods": "110" } ``` Create two pods, one that needs a single banana and another that needs a truck load: ``` $ kubectl create -f chimp.yaml $ kubectl create -f superchimp.yaml ``` Inspect the scheduler result and pod status: ``` $ kubectl describe pods chimp Name: chimp Namespace: default Node: localhost.localdomain/10.0.2.15 Start Time: Thu, 11 Aug 2016 19:58:46 +0000 Labels: <none> Status: Running IP: 172.17.0.2 Controllers: <none> Containers: nginx: Container ID: docker://46ff268f2f9217c59cc49f97cc4f0f085d5ac0e251f508cc08938601117c0cec Image: nginx:1.10 Image ID: docker://sha256:82e97a2b0390a20107ab1310dea17f539ff6034438099384998fd91fc540b128 Port: 80/TCP Limits: cpu: 500m memory: 64Mi pod.alpha.kubernetes.io/opaque-int-resource-bananas: 3 Requests: cpu: 250m memory: 32Mi pod.alpha.kubernetes.io/opaque-int-resource-bananas: 1 State: Running Started: Thu, 11 Aug 2016 19:58:51 +0000 Ready: True Restart Count: 0 Volume Mounts: <none> Environment Variables: <none> Conditions: Type Status Initialized True Ready True PodScheduled True No volumes. QoS Class: Burstable Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 9m 9m 1 {default-scheduler } Normal Scheduled Successfully assigned chimp to localhost.localdomain 9m 9m 2 {kubelet localhost.localdomain} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy. 9m 9m 1 {kubelet localhost.localdomain} spec.containers{nginx} Normal Pulled Container image "nginx:1.10" already present on machine 9m 9m 1 {kubelet localhost.localdomain} spec.containers{nginx} Normal Created Created container with docker id 46ff268f2f92 9m 9m 1 {kubelet localhost.localdomain} spec.containers{nginx} Normal Started Started container with docker id 46ff268f2f92 ``` ``` $ kubectl describe pods superchimp Name: superchimp Namespace: default Node: / Labels: <none> Status: Pending IP: Controllers: <none> Containers: nginx: Image: nginx:1.10 Port: 80/TCP Requests: cpu: 250m memory: 32Mi pod.alpha.kubernetes.io/opaque-int-resource-bananas: 10Ki Volume Mounts: <none> Environment Variables: <none> Conditions: Type Status PodScheduled False No volumes. QoS Class: Burstable Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 1s 15 {default-scheduler } Warning FailedScheduling pod (superchimp) failed to fit in any node fit failure on node (localhost.localdomain): Insufficient pod.alpha.kubernetes.io/opaque-int-resource-bananas ```

spiffxp · 2016-11-18T23:56:54Z

@ConnorDoyle @idvoretskyi this needs an alpha-in-1.5, beta-in-1.5, or stable-in-1.5 label if it's going to be included in release notes for kubernetes 1.5; it's listed as "alpha" in this spreadsheet

spiffxp · 2016-12-01T20:50:17Z

I'm removing the team/SIG-Node label given that SIG-Scheduling is listed as the owner in the 1.5 feature spreadsheet; if we're using labels to describe areas of overlap then we need something else in these issues to identify owner

idvoretskyi · 2016-12-13T18:32:48Z

@ConnorDoyle can you confirm that this item targets beta in 1.6?

ConnorDoyle · 2017-09-05T17:34:13Z

OIR is deprecated as of v1.8 in favor of Extended Resources (this is in the release notes draft already for v1.8).

zhangxiaoyu-zidif · 2017-09-09T03:59:36Z

@ConnorDoyle could you give me some links about Extended Resources.
Many thanks. =)

ConnorDoyle · 2017-09-09T04:49:23Z

@zhangxiaoyu-zidif so far the best source of information is the PR -- docs to come. kubernetes/kubernetes#48922

ConnorDoyle · 2017-09-12T00:12:50Z

@zhangxiaoyu-zidif docs PR is up kubernetes/website#5399

zhangxiaoyu-zidif · 2017-09-12T00:32:02Z

@ConnorDoyle Thanks a lot!
It's much helpful. =)

ConnorDoyle · 2017-11-09T20:00:16Z

As mentioned above, this feature was deprecated in v1.8.
The PR to remove them in v1.9 is here: kubernetes/kubernetes#55103

idvoretskyi · 2018-01-22T13:47:26Z

@ConnorDoyle am I right that we may close this issue now (no further progress on the feature is expected)?

ConnorDoyle · 2018-01-22T14:38:03Z

Correct — closing

…ancements Move the networking enhancements to the right subdir

This was referenced Aug 24, 2016

Create an opaque integer resource kubernetes/kubernetes#19082

Closed

Accounting and feasibility for opaque integer resources in the API server and scheduler. kubernetes/kubernetes#28312

Closed

ConnorDoyle mentioned this issue Aug 29, 2016

[PHASE 1] Opaque integer resource accounting. kubernetes/kubernetes#31652

Merged

davidopp added this to the v1.5 milestone Sep 15, 2016

idvoretskyi assigned ConnorDoyle Sep 29, 2016

davidopp added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label Oct 1, 2016

idvoretskyi added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Oct 13, 2016

idvoretskyi added the alpha-in-1.5 label Nov 23, 2016

ConnorDoyle mentioned this issue Nov 24, 2016

Documentation for opaque integer resources (alpha feature in 1.5) kubernetes/website#1783

Merged

spiffxp removed the sig/node Categorizes an issue or PR as relevant to SIG Node. label Dec 1, 2016

idvoretskyi modified the milestones: next-milestone, v1.5 Dec 13, 2016

This was referenced Apr 5, 2017

Extended pod-level resource isolation #246

Closed

Requirements for opaque integer resources to graduate from Alpha to Beta kubernetes/kubernetes#44181

Closed

ConnorDoyle mentioned this issue Apr 24, 2017

[WIP] Pod and container-level resource isolator interface (plus isolator library, examples, tests) kubernetes/kubernetes#44870

Closed

fabiand mentioned this issue Apr 27, 2017

Resource-dependent pod presets kubernetes/kubernetes#44001

Closed

idvoretskyi removed the (deprecated label - do not use) alpha-in-1.5 label Aug 8, 2017

ConnorDoyle closed this as completed Jan 22, 2018

kasturinarra mentioned this issue Jan 28, 2020

[enterprise-4.3] Edit suggested in file nodes/nodes/nodes-nodes-opaque-resources.adoc openshift/openshift-docs#19347

Closed

This was referenced Feb 6, 2020

Removing Opaque Integer Resources openshift/openshift-docs#19592

Merged

Removing Opaque Integer Resources from 3x openshift/openshift-docs#19593

Merged

ingvagabund pushed a commit to ingvagabund/enhancements that referenced this issue Apr 2, 2020

Merge pull request kubernetes#76 from knobunc/fix/move-networking-enh…

a949874

…ancements Move the networking enhancements to the right subdir

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Opaque integer resources #76

Opaque integer resources #76

ConnorDoyle commented Aug 24, 2016 •

edited

ConnorDoyle commented Aug 24, 2016

ConnorDoyle commented Sep 15, 2016

idvoretskyi commented Sep 29, 2016 •

edited

ConnorDoyle commented Sep 29, 2016

idvoretskyi commented Oct 6, 2016

spiffxp commented Nov 18, 2016

spiffxp commented Dec 1, 2016

idvoretskyi commented Dec 13, 2016

ConnorDoyle commented Sep 5, 2017

zhangxiaoyu-zidif commented Sep 9, 2017

ConnorDoyle commented Sep 9, 2017

ConnorDoyle commented Sep 12, 2017

zhangxiaoyu-zidif commented Sep 12, 2017

ConnorDoyle commented Nov 9, 2017

idvoretskyi commented Jan 22, 2018

ConnorDoyle commented Jan 22, 2018

Opaque integer resources #76

Opaque integer resources #76

Comments

ConnorDoyle commented Aug 24, 2016 • edited

Progress Tracker

ConnorDoyle commented Aug 24, 2016

ConnorDoyle commented Sep 15, 2016

idvoretskyi commented Sep 29, 2016 • edited

ConnorDoyle commented Sep 29, 2016

idvoretskyi commented Oct 6, 2016

spiffxp commented Nov 18, 2016

spiffxp commented Dec 1, 2016

idvoretskyi commented Dec 13, 2016

ConnorDoyle commented Sep 5, 2017

zhangxiaoyu-zidif commented Sep 9, 2017

ConnorDoyle commented Sep 9, 2017

ConnorDoyle commented Sep 12, 2017

zhangxiaoyu-zidif commented Sep 12, 2017

ConnorDoyle commented Nov 9, 2017

idvoretskyi commented Jan 22, 2018

ConnorDoyle commented Jan 22, 2018

ConnorDoyle commented Aug 24, 2016 •

edited

idvoretskyi commented Sep 29, 2016 •

edited