Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CustomResourceDefinitions #95

Closed
deads2k opened this issue Sep 30, 2016 · 127 comments
Closed

CustomResourceDefinitions #95

deads2k opened this issue Sep 30, 2016 · 127 comments
Assignees
Labels
kind/feature Categorizes issue or PR as related to a new feature. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. stage/stable Denotes an issue tracking an enhancement targeted for Stable/GA status
Milestone

Comments

@deads2k
Copy link
Contributor

deads2k commented Sep 30, 2016

Enhancement Description

Scope of work planned for v1.15

Scope of work planned for v1.11

Scope of work planned for v1.10

Scope of work planned for v1.9

Scope of work planned for v1.8

  • Remove deprecated ThirdPartyResource API.
  • Add validation and defaulting for CustomResourceDefinition.
  • Add subresources for CustomResourceDefinition.
    • Support Spec/Status split (/status subresource) on custom resources.
    • Support incrementing object Generation on custom resource data mutation (requires Spec/Status split).
  • Support OwnerReference-based garbage collection with CRD.

Scope of work planned for v1.7

  • Move TPR to a new API group (tentatively called apiextensions) to support deprecation of the extensions group
    • Ideally, implement the new TPR in a separate API server, to be integrated into kube-apiserver via API Aggregation.
  • For now, only allow 1 version at a time per TPR. In the absence of conversion (which is out of scope for this release), this is necessary to remain consistent with the expectations of other components.
    • Support for multiple versions could be added (with or without conversion) in a later release.
  • Fix name conflicts due to lossy conversion of TPR name into resource/kind.
  • Allow TPRs to specify their own names for resources and kinds, rather than tying them to the TPR name.
  • Allow TPRs to register short names that will be discoverable by kubectl.
  • Allow TPRs to optionally be cluster-scoped rather than namespaced.
  • Define and document a process to migrate from extensions/v1beta1 TPR, possibly requiring brief downtime for TPR custom controllers and operators.
    • Where possible, provide automated tools to help with migration.
  • A finalizer ensures CR data is deleted if a CRD is deleted.
  • Fix TPR/CRD data cleanup upon namespace deletion for the 3rd time, this time with a regression test.

Other plans not in scope for this release

  • Support multiple versions at the same time for a given TPR.
    • Other components (e.g. GC, namespace finalizers) expect automatic conversion. TPR currently does not support that.
    • Note that it's possible to change the single registered version of a TPR, but it requires brief downtime for TPR custom controllers and operators.
    • The extensions/v1beta1 TPR gives the appearance of supporting multiple versions, but multiple version support was never implemented.
  • Support customizing where TPR APIs appear in discovery, relative to other TPRs or other APIs.
  • Support namespace-scoped CRD whose CRs are only visible in one namespace.

Plans with unclear status

Still investigating or TBD. Please comment/edit with any updates.

  • Improve the display of TPRs in kubectl/dashboard.
    • There may be other feature trackers addressing this.
@deads2k
Copy link
Contributor Author

deads2k commented Sep 30, 2016

@lavalamp I've created this to try to have a place where we can at least consolidate our thoughts and track progress on third party resources. I've tried to create a list of known shortcomings to be resolved before promotion to stable.

I don't have an owner in mind, but recognition of the problem seems like step 1.

@adohe-zz
Copy link

adohe-zz commented Oct 9, 2016

@deads2k I am learning third party resource recently, also wish to help with something.

@deads2k
Copy link
Contributor Author

deads2k commented Oct 10, 2016

@deads2k I am learning third party resource recently, also wish to help with something.

I've re-ordered the list in terms of what I see as tactical priority. People are trying to use this now and these problems will burn them badly.

If you're comfortable taking the "multiple resources" item, that would be a great start. You could create a separate issue and we can talk about implementation in there.

@adohe-zz
Copy link

adohe-zz commented Oct 23, 2016

@deads2k I spent some time trying to reproduce the first issue:

Multiple Resources, single version, different add times - Adding resource A, waiting for it to appear, then adding resource B fails. Resource B is never added.

but with unluck. Below is my reproduce steps:

  1. create a custom thirdparty resource&wait it to appear
[root@localhost kubernetes]# cat /home/tony/Desktop/debug/lbclaim.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancerclaim.k8s.io
description: "Allow user to claim a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/lbclaim.yaml
thirdpartyresource "loadbalancerclaim.k8s.io" created
[root@localhost kubernetes]# curl  http://localhost:8080/apis/extensions/v1beta1/thirdpartyresources/
{
  "kind": "ThirdPartyResourceList",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/",
    "resourceVersion": "170"
  },
  "items": [
    {
      "metadata": {
        "name": "loadbalancerclaim.k8s.io",
        "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/loadbalancerclaim.k8s.io",
        "uid": "dcb88b3a-9857-11e6-a19b-08002767e1f5",
        "resourceVersion": "146",
        "creationTimestamp": "2016-10-22T13:03:01Z"
      },
      "description": "Allow user to claim a loadbalancer instance",
      "versions": [
        {
          "name": "v1"
        }
      ]
    }
  ]
}
  1. after a moment(more than 10s), create another custom thirdparty resource
[root@localhost kubernetes]# cat /home/tony/Desktop/debug/loadbalancer.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancer.k8s.io
description: "Allow user to curd a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/loadbalancer.yaml
thirdpartyresource "loadbalancer.k8s.io" created
  1. verify both resources exist
[root@localhost kubernetes]# curl http://localhost:8080/apis/k8s.io/v1/
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "k8s.io/v1",
  "resources": [
    {
      "name": "loadbalancerclaims",
      "namespaced": true,
      "kind": "Loadbalancerclaim"
    },
    {
      "name": "loadbalancers",
      "namespaced": true,
      "kind": "Loadbalancer"
    }
  ]
}
[root@localhost kubernetes]# kc get loadbalancers
No resources found.
[root@localhost kubernetes]# kc get loadbalancerclaims
No resources found.

seems we already support multiple resources, single version.

@adohe-zz
Copy link

adohe-zz commented Oct 23, 2016

And I take a deep look at TPR related code. The thirdparty_controller will do periodically sync(every 10 seconds), it will install every new TPR, and also do some deletion job. The ThirdPartyResourceServer contains all installed TPR mappings. As we can see from SyncOneResource and InstallThirdPartyResource, even this this group exists, it will still update the group with the new API.

@adohe-zz
Copy link

adohe-zz commented Oct 23, 2016

Also I found that I am able to delete a TPR schema def even there are TPR instances in the system. I think this should not be allowed.

@deads2k
Copy link
Contributor Author

deads2k commented Oct 24, 2016

@deads2k I spent some time trying to reproduce the first issue:

Try to enable this test: https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 . If it works, we're good. If it fails, something is wrong.

@adohe-zz
Copy link

adohe-zz commented Oct 30, 2016

@deads2k Hi David, please take a look at the message I sent on Slack. Besides, I add a fix to the failed integration test, the third party resource controller will remove the corresponding routes handler when a TPR get deleted, this will help with the integration test, but I am not sure whether this will bring in any other problems.

@brendandburns
Copy link
Contributor

brendandburns commented Nov 1, 2016

For problem #1, it was fixed here:

kubernetes/kubernetes#28414

@adohe-zz
Copy link

adohe-zz commented Nov 1, 2016

@brendandburns actually not, you can run the comment out integration test, and it will fail.

@adohe-zz
Copy link

adohe-zz commented Nov 1, 2016

@brendandburns More correctly, we did support multiple resources, single version, but the deletion logical has some problem.

@brendandburns
Copy link
Contributor

brendandburns commented Nov 1, 2016

@adohe did you file an issue? I can take a look.

@adohe-zz
Copy link

adohe-zz commented Nov 1, 2016

@brendandburns you can see here:

https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 

enable this test, and you will see it will fail. I have tried to fix this on my local, and I will open a PR later today.

@adohe-zz
Copy link

adohe-zz commented Nov 1, 2016

@brendandburns I am afraid I don't file an issue.

@nikhiljindal
Copy link

nikhiljindal commented Nov 1, 2016

Also ref kubernetes/kubernetes#32306 (TPR should be deleted when namespace is deleted)

@grodrigues3
Copy link
Contributor

grodrigues3 commented Nov 9, 2016

@deads2k can you update the checklist ?

@deads2k
Copy link
Contributor Author

deads2k commented Nov 10, 2016

@deads2k can you update the checklist ?

All issues still outstanding. This is actually a feature to track the problems in the (already) beta thirdparyresources implementation from 1.3. We needed a place to keep track of our problems, but had to devote energy to other efforts in 1.5.

@adohe-zz
Copy link

adohe-zz commented Nov 10, 2016

@deads2k I am already working on Multiple Resources, single version and Multiple versions, I think a lot of code need to be update.

@idvoretskyi
Copy link
Member

idvoretskyi commented Nov 16, 2016

@deads2k does still feature still target 1.5?

@idvoretskyi idvoretskyi added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Nov 16, 2016
@kacole2
Copy link
Member

kacole2 commented Apr 12, 2019

Hello @nikhita @deads2k , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. A KEP will need to be merged for 1.15 inclusion as well. Thanks!

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

@liggitt liggitt changed the title CustomResourceDefinitions, née ThirdPartyResources CustomResourceDefinitions Apr 18, 2019
@liggitt
Copy link
Member

liggitt commented Apr 29, 2019

this will remain in beta stage. work on validation, conversion, and OpenAPI publishing is happening in 1.15

@liggitt
Copy link
Member

liggitt commented Apr 29, 2019

updated description with links to relevant KEPs for 1.15

@liggitt liggitt assigned jpbetz and sttts and unassigned deads2k Apr 29, 2019
@kacole2 kacole2 added tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team and removed tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team labels Apr 30, 2019
@kacole2 kacole2 added this to the v1.15 milestone Apr 30, 2019
@daminisatya
Copy link
Member

daminisatya commented May 18, 2019

Hey, @liggitt @deads2k @jpbetz @sttts I'm the v1.15 docs release shadow.

Does this enhancement (or the work planned for v1.15) require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions! 😄

@daminisatya
Copy link
Member

daminisatya commented May 27, 2019

@deads2k @jpbetz @sttts @liggitt

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions! 😄

@sttts
Copy link
Contributor

sttts commented May 28, 2019

Docs PR for 1.15: kubernetes/website#14583

@deads2k can you update the issue description?

@liggitt
Copy link
Member

liggitt commented Jul 8, 2019

/milestone v1.16
/stage stable

@k8s-ci-robot k8s-ci-robot added stage/stable Denotes an issue tracking an enhancement targeted for Stable/GA status and removed stage/beta Denotes an issue tracking an enhancement targeted for Beta status labels Jul 8, 2019
@k8s-ci-robot k8s-ci-robot modified the milestones: v1.15, v1.16 Jul 8, 2019
@simplytunde
Copy link
Member

simplytunde commented Jul 29, 2019

Hey, @liggitt @jpbetz @sttts I'm the v1.16 docs release lead.

Does this enhancement (or the work planned for v1.16) require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday,August 23rd. Let me know if you have any questions!

@sttts
Copy link
Contributor

sttts commented Aug 21, 2019

@simonswine the placeholder PR kubernetes/website#15982

@kacole2
Copy link
Member

kacole2 commented Aug 26, 2019

@liggitt @jpbetz @sttts Thursday is code freeze. Are there any outstanding k/k PRs that will prevent this from graduating to Stable? Everything in the original post for planned 1.15* work looks to be merged.

@jpbetz
Copy link
Contributor

jpbetz commented Aug 26, 2019

I believe the outstanding PRs are just the feature gate version bump (kubernetes/kubernetes#81965) and two outstanding bug fixes that should go in this week: kubernetes/kubernetes#81436, kubernetes/kubernetes#78707

@liggitt
Copy link
Member

liggitt commented Sep 3, 2019

docs ready for review in kubernetes/website#15982

@liggitt
Copy link
Member

liggitt commented Sep 19, 2019

Released as stable in v1.16.0

Post-GA work tracked in https://github.com/orgs/kubernetes/projects/28

/close

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Sep 19, 2019

@liggitt: Closing this issue.

In response to this:

Released as stable in v1.16.0

Post-GA work tracked in https://github.com/orgs/kubernetes/projects/28

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mrbobbytables mrbobbytables removed the tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team label Oct 14, 2019
ingvagabund pushed a commit to ingvagabund/enhancements that referenced this issue Apr 2, 2020
Move machine API proposals into its own repo
howardjohn pushed a commit to howardjohn/enhancements that referenced this issue Oct 21, 2022
* Add revision tags feature progress

* fix template

* Add revision tag feature
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. stage/stable Denotes an issue tracking an enhancement targeted for Stable/GA status
Projects
No open projects
Development

No branches or pull requests