Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CRD age does not change from <invalid> as soon as it becomes Established #57042

Closed
vadimeisenbergibm opened this issue Dec 11, 2017 · 34 comments
Closed

Comments

@vadimeisenbergibm
Copy link

@vadimeisenbergibm vadimeisenbergibm commented Dec 11, 2017

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
When deploying https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml on minikube, I get the following errors:

unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=attributemanifest

for each of the custom resources specified in https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml

Running kubernetes get crd returns:

NAME                                  AGE
attributemanifests.config.istio.io    <invalid>

After several seconds, kubernetes get crd returns valid age for the CRDs. Then the custom resources are added successfully.

What you expected to happen:
I expect that it should be explicitly documented: can custom resources be added immediately after their CRDs? Or the user should wait until kubernetes get crd returns valid age.

How to reproduce it (as minimally and precisely as possible):
Deploy https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml on a minikube with limited resources, so it will take time for the CRDs to become valid.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): 1.8.0
  • Cloud provider or hardware configuration: Minikube 0.23.0
  • OS (e.g. from /etc/os-release): Minikube on Virtual Box on Mac OS
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:
@vadimeisenbergibm
Copy link
Author

@vadimeisenbergibm vadimeisenbergibm commented Dec 11, 2017

/sig api-machinery

@ncdc
Copy link
Member

@ncdc ncdc commented Dec 11, 2017

It typically takes a short amount of time for the server to process a new CRD before you can begin creating resources. You can check the CRD's status.conditions for an entry in the array where type equals Established.

cc @deads2k @sttts @enisoc @nikhita - do we have this documented anywhere?

@vadimeisenbergibm
Copy link
Author

@vadimeisenbergibm vadimeisenbergibm commented Dec 11, 2017

@ncdc So what are the users of kubectl advised to do with regard to new CRDs and their resources? Define CRDs first, wait several seconds and then add resources? Or just add new CRDS and their resources, and in case of failure retry? I would like this advice to be documented somewhere.

You can check the CRD's status.conditions for an entry in the array where type equals Established

By which command can I check this?

@ncdc
Copy link
Member

@ncdc ncdc commented Dec 11, 2017

@vadimeisenbergibm

So what are the users of kubectl advised to do with regard to new CRDs and their resources? Define CRDs first, wait several seconds and then add resources? Or just add new CRDS and their resources, and in case of failure retry? I would like this advice to be documented somewhere.

I'm not sure what our guidance is beyond waiting. You could also programmatically poll/watch the CRD and wait for it to be established. cc @kubernetes/sig-api-machinery-misc for increased visibility & guidance

By which command can I check this?

Here is a sample I just ran:

$ get crd/backups.ark.heptio.com -o jsonpath='{.status.conditions[?(.type == "Established")].status}{"\n"}'
True
@sttts
Copy link
Contributor

@sttts sttts commented Dec 11, 2017

@sttts
Copy link
Contributor

@sttts sttts commented Dec 11, 2017

And of course there is no connnection to Minikube, but it is like that by design on every cluster.

@nikhita
Copy link
Member

@nikhita nikhita commented Dec 12, 2017

For the docs: kubernetes/website#6660.

@nikhita
Copy link
Member

@nikhita nikhita commented Dec 12, 2017

/assign

@vadimeisenbergibm
Copy link
Author

@vadimeisenbergibm vadimeisenbergibm commented Dec 12, 2017

@ncdc I have the output of kubectl get crd/backups.ark.heptio.com -o jsonpath='{.status.conditions[?(.type == "Established")].status}{"\n"}' as True, despite the fact that kubectl get crd returns AGE <invalid>. I think the simplest way is just to run kubectl get crd and to wait until all the CRDs have non-invalid age.

@ncdc
Copy link
Member

@ncdc ncdc commented Dec 12, 2017

@deads2k @nikhita @sttts does the age matter at all?

@ncdc
Copy link
Member

@ncdc ncdc commented Dec 12, 2017

From what I'm reading, you'll see <invalid> if the object's lifespan appears to be negative. I think you can ignore that. The definitive green-light is the Established condition and/or the availability of the custom api group+resource in discovery.

@vadimeisenbergibm
Copy link
Author

@vadimeisenbergibm vadimeisenbergibm commented Dec 12, 2017

I experience that adding resources fails until the age becomes 0s. I mean while the age is <invalid>, adding resources fails. Established condition is True immediately, even when the age is <invalide> and the resources fail to be added.

@ncdc
Copy link
Member

@ncdc ncdc commented Dec 12, 2017

Can you share the logs from the apiserver around the time of the failures?

@sttts
Copy link
Contributor

@sttts sttts commented Dec 13, 2017

We might have a race-condition between the update of the Established condition (https://github.com/sttts/kubernetes/blob/1161561ee12eea5ba28eb5efa9b384ec732def0e/staging/src/k8s.io/apiextensions-apiserver/pkg/controller/status/naming_controller.go#L253) and the HTTP handler of the API server (same process) actually picking in up (through informers, https://github.com/sttts/kubernetes/blob/b5b62c68318be79a665257c260ea9f9bbb6d6318/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go#L116). We could synchronize these two, but this won't help in HA setups either.

@vadimeisenbergibm
Copy link
Author

@vadimeisenbergibm vadimeisenbergibm commented Dec 13, 2017

@ncdc Here is the audit log
audit_log.txt

See for example attributemanifests.config.istio.io .

Do you want me to produce any other log?

@roycaihw
Copy link
Member

@roycaihw roycaihw commented Dec 14, 2017

@nikhita
Copy link
Member

@nikhita nikhita commented Dec 23, 2017

/reopen

@k8s-ci-robot k8s-ci-robot reopened this Dec 23, 2017
@venkatsc
Copy link
Contributor

@venkatsc venkatsc commented Jan 11, 2018

Facing the same issue on 1.9.1. CRD AGE never changed from <invalid>.

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.1+coreos.0", GitCommit:"49020061462317aa0d54c2577f5bfb257e9cf58e", GitTreeState:"clean", BuildDate:"2018-01-08T18:21:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

@nikhita
Copy link
Member

@nikhita nikhita commented Jan 23, 2018

Looks like the real issue here is that CRD age does not change from <invalid> as soon as it becomes Established.

Can someone update the PR title to reflect the same? Thanks!

@nikhita
Copy link
Member

@nikhita nikhita commented Jun 13, 2018

The age problem could be there, but I'm not sure what's the root of the problem there. I have left some comments on this earlier, in this issue.

This is still pending.

I'm going to explicitly unassign myself here, since it has the help-wanted label (and could use some help for the above). Will be involved in this though.
/unassign

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Sep 11, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@xmudrii
Copy link
Member

@xmudrii xmudrii commented Sep 11, 2018

/remove-lifecycle stale

The issue #65517 describes similar problem, so maybe one can be closed.

@nikhita
Copy link
Member

@nikhita nikhita commented Sep 13, 2018

The issue #65517 describes similar problem, so maybe one can be closed.

I think the underlying issue should be the same, but let's keep it open until we confirm this.

Removing help because we don't have explicit directions about solving this yet.
/remove-help

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Dec 12, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@nikhita
Copy link
Member

@nikhita nikhita commented Dec 12, 2018

/remove-lifecycle stale

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Mar 12, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot
Copy link

@fejta-bot fejta-bot commented Apr 11, 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@liggitt liggitt added this to Required for GA, in progress in Custom Resource Definitions Apr 18, 2019
@roycaihw
Copy link
Member

@roycaihw roycaihw commented Apr 18, 2019

/remove-lifecycle rotten

@liggitt
Copy link
Member

@liggitt liggitt commented Apr 24, 2019

#69654 is tracking the established condition transition time

Age: <invalid> means that you are using a version of kubectl older than 1.11, and that your client clock is skewed from your server clock.

Supported versions of kubectl (1.12+) get display text from the server and do not have this issue.

/close

@k8s-ci-robot
Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Apr 24, 2019

@liggitt: Closing this issue.

In response to this:

#69654 is tracking the established condition transition time

Age: <invalid> means that you are using a version of kubectl older than 1.11, and that your client clock is skewed from your server clock.

Supported versions of kubectl (1.12+) get display text from the server and do not have this issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@liggitt liggitt removed this from Required for GA, in progress in Custom Resource Definitions May 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet