Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CustomResourceDefinitions #95

Open
deads2k opened this issue Sep 30, 2016 · 121 comments

Comments

@deads2k
Copy link
Contributor

commented Sep 30, 2016

Enhancement Description

Scope of work planned for v1.15

  • Define allowed OpenAPI subset (#1002, #692)
  • Define and perform scale testing for CRD (#1015)
  • Bring CRD conversion webhooks to beta (#1004, #598)

Scope of work planned for v1.11

Scope of work planned for v1.10

Scope of work planned for v1.9

Scope of work planned for v1.8

  • Remove deprecated ThirdPartyResource API.
  • Add validation and defaulting for CustomResourceDefinition.
  • Add subresources for CustomResourceDefinition.
    • Support Spec/Status split (/status subresource) on custom resources.
    • Support incrementing object Generation on custom resource data mutation (requires Spec/Status split).
  • Support OwnerReference-based garbage collection with CRD.

Scope of work planned for v1.7

  • Move TPR to a new API group (tentatively called apiextensions) to support deprecation of the extensions group
    • Ideally, implement the new TPR in a separate API server, to be integrated into kube-apiserver via API Aggregation.
  • For now, only allow 1 version at a time per TPR. In the absence of conversion (which is out of scope for this release), this is necessary to remain consistent with the expectations of other components.
    • Support for multiple versions could be added (with or without conversion) in a later release.
  • Fix name conflicts due to lossy conversion of TPR name into resource/kind.
  • Allow TPRs to specify their own names for resources and kinds, rather than tying them to the TPR name.
  • Allow TPRs to register short names that will be discoverable by kubectl.
  • Allow TPRs to optionally be cluster-scoped rather than namespaced.
  • Define and document a process to migrate from extensions/v1beta1 TPR, possibly requiring brief downtime for TPR custom controllers and operators.
    • Where possible, provide automated tools to help with migration.
  • A finalizer ensures CR data is deleted if a CRD is deleted.
  • Fix TPR/CRD data cleanup upon namespace deletion for the 3rd time, this time with a regression test.

Other plans not in scope for this release

  • Support multiple versions at the same time for a given TPR.
    • Other components (e.g. GC, namespace finalizers) expect automatic conversion. TPR currently does not support that.
    • Note that it's possible to change the single registered version of a TPR, but it requires brief downtime for TPR custom controllers and operators.
    • The extensions/v1beta1 TPR gives the appearance of supporting multiple versions, but multiple version support was never implemented.
  • Support customizing where TPR APIs appear in discovery, relative to other TPRs or other APIs.
  • Support namespace-scoped CRD whose CRs are only visible in one namespace.

Plans with unclear status

Still investigating or TBD. Please comment/edit with any updates.

  • Improve the display of TPRs in kubectl/dashboard.
    • There may be other feature trackers addressing this.
@deads2k

This comment has been minimized.

Copy link
Contributor Author

commented Sep 30, 2016

@lavalamp I've created this to try to have a place where we can at least consolidate our thoughts and track progress on third party resources. I've tried to create a list of known shortcomings to be resolved before promotion to stable.

I don't have an owner in mind, but recognition of the problem seems like step 1.

@adohe

This comment has been minimized.

Copy link
Member

commented Oct 9, 2016

@deads2k I am learning third party resource recently, also wish to help with something.

@deads2k

This comment has been minimized.

Copy link
Contributor Author

commented Oct 10, 2016

@deads2k I am learning third party resource recently, also wish to help with something.

I've re-ordered the list in terms of what I see as tactical priority. People are trying to use this now and these problems will burn them badly.

If you're comfortable taking the "multiple resources" item, that would be a great start. You could create a separate issue and we can talk about implementation in there.

@adohe

This comment has been minimized.

Copy link
Member

commented Oct 23, 2016

@deads2k I spent some time trying to reproduce the first issue:

Multiple Resources, single version, different add times - Adding resource A, waiting for it to appear, then adding resource B fails. Resource B is never added.

but with unluck. Below is my reproduce steps:

  1. create a custom thirdparty resource&wait it to appear
[root@localhost kubernetes]# cat /home/tony/Desktop/debug/lbclaim.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancerclaim.k8s.io
description: "Allow user to claim a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/lbclaim.yaml
thirdpartyresource "loadbalancerclaim.k8s.io" created
[root@localhost kubernetes]# curl  http://localhost:8080/apis/extensions/v1beta1/thirdpartyresources/
{
  "kind": "ThirdPartyResourceList",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/",
    "resourceVersion": "170"
  },
  "items": [
    {
      "metadata": {
        "name": "loadbalancerclaim.k8s.io",
        "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/loadbalancerclaim.k8s.io",
        "uid": "dcb88b3a-9857-11e6-a19b-08002767e1f5",
        "resourceVersion": "146",
        "creationTimestamp": "2016-10-22T13:03:01Z"
      },
      "description": "Allow user to claim a loadbalancer instance",
      "versions": [
        {
          "name": "v1"
        }
      ]
    }
  ]
}
  1. after a moment(more than 10s), create another custom thirdparty resource
[root@localhost kubernetes]# cat /home/tony/Desktop/debug/loadbalancer.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancer.k8s.io
description: "Allow user to curd a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/loadbalancer.yaml
thirdpartyresource "loadbalancer.k8s.io" created
  1. verify both resources exist
[root@localhost kubernetes]# curl http://localhost:8080/apis/k8s.io/v1/
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "k8s.io/v1",
  "resources": [
    {
      "name": "loadbalancerclaims",
      "namespaced": true,
      "kind": "Loadbalancerclaim"
    },
    {
      "name": "loadbalancers",
      "namespaced": true,
      "kind": "Loadbalancer"
    }
  ]
}
[root@localhost kubernetes]# kc get loadbalancers
No resources found.
[root@localhost kubernetes]# kc get loadbalancerclaims
No resources found.

seems we already support multiple resources, single version.

@adohe

This comment has been minimized.

Copy link
Member

commented Oct 23, 2016

And I take a deep look at TPR related code. The thirdparty_controller will do periodically sync(every 10 seconds), it will install every new TPR, and also do some deletion job. The ThirdPartyResourceServer contains all installed TPR mappings. As we can see from SyncOneResource and InstallThirdPartyResource, even this this group exists, it will still update the group with the new API.

@adohe

This comment has been minimized.

Copy link
Member

commented Oct 23, 2016

Also I found that I am able to delete a TPR schema def even there are TPR instances in the system. I think this should not be allowed.

@deads2k

This comment has been minimized.

Copy link
Contributor Author

commented Oct 24, 2016

@deads2k I spent some time trying to reproduce the first issue:

Try to enable this test: https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 . If it works, we're good. If it fails, something is wrong.

@adohe

This comment has been minimized.

Copy link
Member

commented Oct 30, 2016

@deads2k Hi David, please take a look at the message I sent on Slack. Besides, I add a fix to the failed integration test, the third party resource controller will remove the corresponding routes handler when a TPR get deleted, this will help with the integration test, but I am not sure whether this will bring in any other problems.

@brendandburns

This comment has been minimized.

Copy link
Contributor

commented Nov 1, 2016

For problem #1, it was fixed here:

kubernetes/kubernetes#28414

@adohe

This comment has been minimized.

Copy link
Member

commented Nov 1, 2016

@brendandburns actually not, you can run the comment out integration test, and it will fail.

@adohe

This comment has been minimized.

Copy link
Member

commented Nov 1, 2016

@brendandburns More correctly, we did support multiple resources, single version, but the deletion logical has some problem.

@brendandburns

This comment has been minimized.

Copy link
Contributor

commented Nov 1, 2016

@adohe did you file an issue? I can take a look.

@adohe

This comment has been minimized.

Copy link
Member

commented Nov 1, 2016

@brendandburns you can see here:

https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 

enable this test, and you will see it will fail. I have tried to fix this on my local, and I will open a PR later today.

@adohe

This comment has been minimized.

Copy link
Member

commented Nov 1, 2016

@brendandburns I am afraid I don't file an issue.

@nikhiljindal

This comment has been minimized.

Copy link
Member

commented Nov 1, 2016

Also ref kubernetes/kubernetes#32306 (TPR should be deleted when namespace is deleted)

@grodrigues3

This comment has been minimized.

Copy link
Contributor

commented Nov 9, 2016

@deads2k can you update the checklist ?

@deads2k

This comment has been minimized.

Copy link
Contributor Author

commented Nov 10, 2016

@deads2k can you update the checklist ?

All issues still outstanding. This is actually a feature to track the problems in the (already) beta thirdparyresources implementation from 1.3. We needed a place to keep track of our problems, but had to devote energy to other efforts in 1.5.

@adohe

This comment has been minimized.

Copy link
Member

commented Nov 10, 2016

@deads2k I am already working on Multiple Resources, single version and Multiple versions, I think a lot of code need to be update.

@idvoretskyi

This comment has been minimized.

Copy link
Member

commented Nov 16, 2016

@deads2k does still feature still target 1.5?

@justaugustus justaugustus removed this from the next-milestone milestone Jul 2, 2018

@kacole2

This comment has been minimized.

Copy link
Member

commented Oct 8, 2018

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

  • Docs (open placeholder PRs): 11/8
  • Code Slush: 11/9
  • Code Freeze Begins: 11/15
  • Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

@nikhita

This comment has been minimized.

Copy link
Member

commented Oct 10, 2018

This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13.

No, there are no plans to graduate this in 1.13. The CRD API will remain in beta.

@fejta-bot

This comment has been minimized.

Copy link

commented Jan 8, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@nikhita

This comment has been minimized.

Copy link
Member

commented Jan 8, 2019

/remove-lifecycle stale

@claurence

This comment has been minimized.

Copy link

commented Jan 16, 2019

@deads2k Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

@nikhita

This comment has been minimized.

Copy link
Member

commented Jan 16, 2019

@claurence The CRD API will remain in beta for 1.14 as well.

@kacole2

This comment has been minimized.

Copy link
Member

commented Apr 12, 2019

Hello @nikhita @deads2k , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. A KEP will need to be merged for 1.15 inclusion as well. Thanks!

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

@liggitt liggitt changed the title CustomResourceDefinitions, née ThirdPartyResources CustomResourceDefinitions Apr 18, 2019

@liggitt

This comment has been minimized.

Copy link
Member

commented Apr 29, 2019

this will remain in beta stage. work on validation, conversion, and OpenAPI publishing is happening in 1.15

@liggitt

This comment has been minimized.

Copy link
Member

commented Apr 29, 2019

updated description with links to relevant KEPs for 1.15

@liggitt liggitt assigned jpbetz and sttts and unassigned deads2k Apr 29, 2019

@kacole2 kacole2 added tracked/yes and removed tracked/no labels Apr 30, 2019

@kacole2 kacole2 added this to the v1.15 milestone Apr 30, 2019

@daminisatya

This comment has been minimized.

Copy link
Member

commented May 18, 2019

Hey, @liggitt @deads2k @jpbetz @sttts I'm the v1.15 docs release shadow.

Does this enhancement (or the work planned for v1.15) require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions! 😄

@daminisatya

This comment has been minimized.

Copy link
Member

commented May 27, 2019

@deads2k @jpbetz @sttts @liggitt

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions! 😄

@sttts

This comment has been minimized.

Copy link
Contributor

commented May 28, 2019

Docs PR for 1.15: kubernetes/website#14583

@deads2k can you update the issue description?

@liggitt

This comment has been minimized.

Copy link
Member

commented Jul 8, 2019

/milestone v1.16
/stage stable

@k8s-ci-robot k8s-ci-robot added stage/stable and removed stage/beta labels Jul 8, 2019

@k8s-ci-robot k8s-ci-robot modified the milestones: v1.15, v1.16 Jul 8, 2019

@simplytunde

This comment has been minimized.

Copy link
Member

commented Jul 29, 2019

Hey, @liggitt @jpbetz @sttts I'm the v1.16 docs release lead.

Does this enhancement (or the work planned for v1.16) require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday,August 23rd. Let me know if you have any questions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.