New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CustomResourceDefinitions, née ThirdPartyResources #95

Open
deads2k opened this Issue Sep 30, 2016 · 109 comments

Comments

Projects
@deads2k

deads2k commented Sep 30, 2016

CustomResourceDefinitions, née Third Party Resources

  • One-line feature description (can be used as a release note):
    • The deprecated ThirdPartyResource API is now removed. If you have any TPRs, you should migrate them to CRD before upgrading to 1.8.
    • Add validation, defaulting, and subresources for CRD.
  • Primary contact (assignee): @deads2k, @enisoc, @sttts
  • Responsible SIGs: sig-api-machinery, sig-apps
  • Design proposal link (community repo):
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target: Already released.
    • Beta release target: released in v1.7
    • Stable release target:

Scope of work planned for v1.11

Scope of work planned for v1.10

Scope of work planned for v1.9

Scope of work planned for v1.8

  • Remove deprecated ThirdPartyResource API.
  • Add validation and defaulting for CustomResourceDefinition.
  • Add subresources for CustomResourceDefinition.
    • Support Spec/Status split (/status subresource) on custom resources.
    • Support incrementing object Generation on custom resource data mutation (requires Spec/Status split).
  • Support OwnerReference-based garbage collection with CRD.

Scope of work planned for v1.7

  • Move TPR to a new API group (tentatively called apiextensions) to support deprecation of the extensions group
    • Ideally, implement the new TPR in a separate API server, to be integrated into kube-apiserver via API Aggregation.
  • For now, only allow 1 version at a time per TPR. In the absence of conversion (which is out of scope for this release), this is necessary to remain consistent with the expectations of other components.
    • Support for multiple versions could be added (with or without conversion) in a later release.
  • Fix name conflicts due to lossy conversion of TPR name into resource/kind.
  • Allow TPRs to specify their own names for resources and kinds, rather than tying them to the TPR name.
  • Allow TPRs to register short names that will be discoverable by kubectl.
  • Allow TPRs to optionally be cluster-scoped rather than namespaced.
  • Define and document a process to migrate from extensions/v1beta1 TPR, possibly requiring brief downtime for TPR custom controllers and operators.
    • Where possible, provide automated tools to help with migration.
  • A finalizer ensures CR data is deleted if a CRD is deleted.
  • Fix TPR/CRD data cleanup upon namespace deletion for the 3rd time, this time with a regression test.

Other plans not in scope for this release

  • Support multiple versions at the same time for a given TPR.
    • Other components (e.g. GC, namespace finalizers) expect automatic conversion. TPR currently does not support that.
    • Note that it's possible to change the single registered version of a TPR, but it requires brief downtime for TPR custom controllers and operators.
    • The extensions/v1beta1 TPR gives the appearance of supporting multiple versions, but multiple version support was never implemented.
  • Support customizing where TPR APIs appear in discovery, relative to other TPRs or other APIs.
  • Support namespace-scoped CRD whose CRs are only visible in one namespace.

Plans with unclear status

Still investigating or TBD. Please comment/edit with any updates.

  • Improve the display of TPRs in kubectl/dashboard.
    • There may be other feature trackers addressing this.
@deads2k

This comment has been minimized.

deads2k commented Sep 30, 2016

@lavalamp I've created this to try to have a place where we can at least consolidate our thoughts and track progress on third party resources. I've tried to create a list of known shortcomings to be resolved before promotion to stable.

I don't have an owner in mind, but recognition of the problem seems like step 1.

@adohe

This comment has been minimized.

Member

adohe commented Oct 9, 2016

@deads2k I am learning third party resource recently, also wish to help with something.

@deads2k

This comment has been minimized.

deads2k commented Oct 10, 2016

@deads2k I am learning third party resource recently, also wish to help with something.

I've re-ordered the list in terms of what I see as tactical priority. People are trying to use this now and these problems will burn them badly.

If you're comfortable taking the "multiple resources" item, that would be a great start. You could create a separate issue and we can talk about implementation in there.

@adohe

This comment has been minimized.

Member

adohe commented Oct 23, 2016

@deads2k I spent some time trying to reproduce the first issue:

Multiple Resources, single version, different add times - Adding resource A, waiting for it to appear, then adding resource B fails. Resource B is never added.

but with unluck. Below is my reproduce steps:

  1. create a custom thirdparty resource&wait it to appear
[root@localhost kubernetes]# cat /home/tony/Desktop/debug/lbclaim.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancerclaim.k8s.io
description: "Allow user to claim a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/lbclaim.yaml
thirdpartyresource "loadbalancerclaim.k8s.io" created
[root@localhost kubernetes]# curl  http://localhost:8080/apis/extensions/v1beta1/thirdpartyresources/
{
  "kind": "ThirdPartyResourceList",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/",
    "resourceVersion": "170"
  },
  "items": [
    {
      "metadata": {
        "name": "loadbalancerclaim.k8s.io",
        "selfLink": "/apis/extensions/v1beta1/thirdpartyresources/loadbalancerclaim.k8s.io",
        "uid": "dcb88b3a-9857-11e6-a19b-08002767e1f5",
        "resourceVersion": "146",
        "creationTimestamp": "2016-10-22T13:03:01Z"
      },
      "description": "Allow user to claim a loadbalancer instance",
      "versions": [
        {
          "name": "v1"
        }
      ]
    }
  ]
}
  1. after a moment(more than 10s), create another custom thirdparty resource
[root@localhost kubernetes]# cat /home/tony/Desktop/debug/loadbalancer.yaml
kind: ThirdPartyResource
apiVersion: extensions/v1beta1
metadata:
  name: loadbalancer.k8s.io
description: "Allow user to curd a loadbalancer instance"
versions:
- name: v1
[root@localhost kubernetes]# kc create -f /home/tony/Desktop/debug/loadbalancer.yaml
thirdpartyresource "loadbalancer.k8s.io" created
  1. verify both resources exist
[root@localhost kubernetes]# curl http://localhost:8080/apis/k8s.io/v1/
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "k8s.io/v1",
  "resources": [
    {
      "name": "loadbalancerclaims",
      "namespaced": true,
      "kind": "Loadbalancerclaim"
    },
    {
      "name": "loadbalancers",
      "namespaced": true,
      "kind": "Loadbalancer"
    }
  ]
}
[root@localhost kubernetes]# kc get loadbalancers
No resources found.
[root@localhost kubernetes]# kc get loadbalancerclaims
No resources found.

seems we already support multiple resources, single version.

@adohe

This comment has been minimized.

Member

adohe commented Oct 23, 2016

And I take a deep look at TPR related code. The thirdparty_controller will do periodically sync(every 10 seconds), it will install every new TPR, and also do some deletion job. The ThirdPartyResourceServer contains all installed TPR mappings. As we can see from SyncOneResource and InstallThirdPartyResource, even this this group exists, it will still update the group with the new API.

@adohe

This comment has been minimized.

Member

adohe commented Oct 23, 2016

Also I found that I am able to delete a TPR schema def even there are TPR instances in the system. I think this should not be allowed.

@deads2k

This comment has been minimized.

deads2k commented Oct 24, 2016

@deads2k I spent some time trying to reproduce the first issue:

Try to enable this test: https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 . If it works, we're good. If it fails, something is wrong.

@adohe

This comment has been minimized.

Member

adohe commented Oct 30, 2016

@deads2k Hi David, please take a look at the message I sent on Slack. Besides, I add a fix to the failed integration test, the third party resource controller will remove the corresponding routes handler when a TPR get deleted, this will help with the integration test, but I am not sure whether this will bring in any other problems.

@brendandburns

This comment has been minimized.

brendandburns commented Nov 1, 2016

For problem #1, it was fixed here:

kubernetes/kubernetes#28414

@adohe

This comment has been minimized.

Member

adohe commented Nov 1, 2016

@brendandburns actually not, you can run the comment out integration test, and it will fail.

@adohe

This comment has been minimized.

Member

adohe commented Nov 1, 2016

@brendandburns More correctly, we did support multiple resources, single version, but the deletion logical has some problem.

@brendandburns

This comment has been minimized.

brendandburns commented Nov 1, 2016

@adohe did you file an issue? I can take a look.

@adohe

This comment has been minimized.

Member

adohe commented Nov 1, 2016

@brendandburns you can see here:

https://github.com/kubernetes/kubernetes/blob/master/test/integration/thirdparty/thirdparty_test.go#L137 

enable this test, and you will see it will fail. I have tried to fix this on my local, and I will open a PR later today.

@adohe

This comment has been minimized.

Member

adohe commented Nov 1, 2016

@brendandburns I am afraid I don't file an issue.

@nikhiljindal

This comment has been minimized.

Member

nikhiljindal commented Nov 1, 2016

Also ref kubernetes/kubernetes#32306 (TPR should be deleted when namespace is deleted)

@grodrigues3

This comment has been minimized.

Contributor

grodrigues3 commented Nov 9, 2016

@deads2k can you update the checklist ?

@deads2k

This comment has been minimized.

deads2k commented Nov 10, 2016

@deads2k can you update the checklist ?

All issues still outstanding. This is actually a feature to track the problems in the (already) beta thirdparyresources implementation from 1.3. We needed a place to keep track of our problems, but had to devote energy to other efforts in 1.5.

@adohe

This comment has been minimized.

Member

adohe commented Nov 10, 2016

@deads2k I am already working on Multiple Resources, single version and Multiple versions, I think a lot of code need to be update.

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Nov 16, 2016

@deads2k does still feature still target 1.5?

@nikhita nikhita referenced this issue Jan 23, 2018

Open

Umbrella issue for CRDs moving to GA #58682

17 of 53 tasks complete
@nikhita

This comment has been minimized.

Member

nikhita commented Jan 23, 2018

List of tasks for CRDs moving to GA: kubernetes/kubernetes#58682

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Jan 23, 2018

@nikhita does it mean that entire CRD feature is moving to GA?

@sttts

This comment has been minimized.

sttts commented Jan 23, 2018

does it mean that entire CRD feature is moving to GA?

The API will move to GA, i.e. to v1, possibly with some beta/alpha sub-features though. It is not terminated though when this will happen, i.e. whether 1.10 is feasible.

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Jan 23, 2018

@sttts @nikhita can you define the feature roadmap more precisely?

@nikhita

This comment has been minimized.

Member

nikhita commented Jan 23, 2018

can you define the feature roadmap more precisely?

For 1.10:

There is no exact set of deliverables planned for the next releases but the plan is to go GA by the end of the year (https://groups.google.com/forum/#!topic/kubernetes-sig-api-machinery/07JKqCzQKsc).

We will go to GA once all the issues that are not crossed out in kubernetes/kubernetes#58682 will be complete.

When the CRD api goes GA, there might be features in it (example: CustomResourceValidation https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiextensions-apiserver/pkg/features/kube_features.go#L35) that could be in alpha/beta.

@justaugustus

This comment has been minimized.

Member

justaugustus commented Apr 17, 2018

@sttts @nikhita @deads2k
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

  • Description
  • Milestone
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

cc @idvoretskyi

@nikhita

This comment has been minimized.

Member

nikhita commented Apr 17, 2018

Any plans for this in 1.11?

I don't have permissions to edit the PR body (if someone can do that, it'd be great!). But the plan is:

If so, can you please ensure the feature is up-to-date with the appropriate:
Description

The one-line description should be updated to include "Add validation, defaulting, subresources and versioning for CRDs".

Design proposals mentioned in the description needs to include:

Can someone please add these in the PR body as well?

Labels:

/kind feature

@nikhita

This comment has been minimized.

Member

nikhita commented Apr 17, 2018

/cc @mbohlool

@sttts

This comment has been minimized.

sttts commented Apr 17, 2018

Can someone please add these in the PR body as well?

done

@justaugustus justaugustus modified the milestones: v1.9, v1.11 Apr 17, 2018

@justaugustus

This comment has been minimized.

Member

justaugustus commented Apr 17, 2018

@nikhita @sttts @mbohlool -- just to clarify, are we targeting beta for the 1.11 cycle?

@justaugustus

This comment has been minimized.

Member

justaugustus commented Apr 24, 2018

@nikhita @sttts @mbohlool -- pinging again on this...
Are we targeting beta for 1.11? Just want to make sure as feature freeze is today.

@sttts

This comment has been minimized.

sttts commented Apr 24, 2018

@justaugustus CRDs are beta already. GA is not planned for 1.11.

All listed features/extensions (pruning, defaulting, versioning) will probably start as alpha.

@justaugustus

This comment has been minimized.

Member

justaugustus commented Apr 24, 2018

@sttts Hmmm, in that case, should we have separate issues to track those features / extensions and their stages independently?

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Jun 1, 2018

To record - @nikhita has created an issue for the subfeature #571

@sttts @justaugustus

@nikhita

This comment has been minimized.

Member

nikhita commented Jun 21, 2018

Defaulting and Pruning sub-feature issue: #575

@nikhita

This comment has been minimized.

Member

nikhita commented Jun 22, 2018

@justaugustus @idvoretskyi for 1.12 tracking purposes: there will be additions and maybe bug fixes but this will stay in beta for 1.12 (so no change from the features perspective).

There is a new sub-feature which is planned as alpha, but it is created as a separate issue: #575.

@justaugustus justaugustus removed this from the next-milestone milestone Jul 2, 2018

@kacole2

This comment has been minimized.

Contributor

kacole2 commented Oct 8, 2018

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

  • Docs (open placeholder PRs): 11/8
  • Code Slush: 11/9
  • Code Freeze Begins: 11/15
  • Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

@nikhita

This comment has been minimized.

Member

nikhita commented Oct 10, 2018

This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13.

No, there are no plans to graduate this in 1.13. The CRD API will remain in beta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment