CRD age does not change from <invalid> as soon as it becomes Established

[vadimeisenbergibm]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
When deploying https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml on minikube, I get the following errors:

unable to recognize "install/kubernetes/istio.yaml": no matches for config.istio.io/, Kind=attributemanifest

for each of the custom resources specified in https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml

Running kubernetes get crd returns:

NAME                                  AGE
attributemanifests.config.istio.io    <invalid>

After several seconds, kubernetes get crd returns valid age for the CRDs. Then the custom resources are added successfully.

What you expected to happen:
I expect that it should be explicitly documented: can custom resources be added immediately after their CRDs? Or the user should wait until kubernetes get crd returns valid age.

How to reproduce it (as minimally and precisely as possible):
Deploy https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml on a minikube with limited resources, so it will take time for the CRDs to become valid.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.8.0
• Cloud provider or hardware configuration: Minikube 0.23.0
• OS (e.g. from /etc/os-release): Minikube on Virtual Box on Mac OS
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[vadimeisenbergibm]

/sig api-machinery

[ncdc]

It typically takes a short amount of time for the server to process a new CRD before you can begin creating resources. You can check the CRD's status.conditions for an entry in the array where type equals Established.

cc @deads2k @sttts @enisoc @nikhita - do we have this documented anywhere?

[vadimeisenbergibm]

@ncdc So what are the users of kubectl advised to do with regard to new CRDs and their resources? Define CRDs first, wait several seconds and then add resources? Or just add new CRDS and their resources, and in case of failure retry? I would like this advice to be documented somewhere.

You can check the CRD's status.conditions for an entry in the array where type equals Established

By which command can I check this?

[ncdc]

@vadimeisenbergibm

So what are the users of kubectl advised to do with regard to new CRDs and their resources? Define CRDs first, wait several seconds and then add resources? Or just add new CRDS and their resources, and in case of failure retry? I would like this advice to be documented somewhere.

I'm not sure what our guidance is beyond waiting. You could also programmatically poll/watch the CRD and wait for it to be established. cc @kubernetes/sig-api-machinery-misc for increased visibility & guidance

By which command can I check this?

Here is a sample I just ran:

$ get crd/backups.ark.heptio.com -o jsonpath='{.status.conditions[?(.type == "Established")].status}{"\n"}'
True

[sttts]

https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/ does not document this, but should. PR welcome.

[sttts]

And of course there is no connnection to Minikube, but it is like that by design on every cluster.

[nikhita]

For the docs: kubernetes/website#6660.

[nikhita]

/assign

[vadimeisenbergibm]

@ncdc I have the output of kubectl get crd/backups.ark.heptio.com -o jsonpath='{.status.conditions[?(.type == "Established")].status}{"\n"}' as True, despite the fact that kubectl get crd returns AGE <invalid>. I think the simplest way is just to run kubectl get crd and to wait until all the CRDs have non-invalid age.

[ncdc]

@deads2k @nikhita @sttts does the age matter at all?

[ncdc]

From what I'm reading, you'll see <invalid> if the object's lifespan appears to be negative. I think you can ignore that. The definitive green-light is the Established condition and/or the availability of the custom api group+resource in discovery.

[vadimeisenbergibm]

I experience that adding resources fails until the age becomes 0s. I mean while the age is <invalid>, adding resources fails. Established condition is True immediately, even when the age is <invalide> and the resources fail to be added.

[ncdc]

Can you share the logs from the apiserver around the time of the failures?

[sttts]

We might have a race-condition between the update of the Established condition (https://github.com/sttts/kubernetes/blob/1161561ee12eea5ba28eb5efa9b384ec732def0e/staging/src/k8s.io/apiextensions-apiserver/pkg/controller/status/naming_controller.go#L253) and the HTTP handler of the API server (same process) actually picking in up (through informers, https://github.com/sttts/kubernetes/blob/b5b62c68318be79a665257c260ea9f9bbb6d6318/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go#L116). We could synchronize these two, but this won't help in HA setups either.

[vadimeisenbergibm]

@ncdc Here is the audit log
audit_log.txt

See for example attributemanifests.config.istio.io .

Do you want me to produce any other log?

[roycaihw]

cc @jennybuckley @cheftako @yliaog

[nikhita]

/reopen

[venkatsc]

Facing the same issue on 1.9.1. CRD AGE never changed from <invalid>.

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.1+coreos.0", GitCommit:"49020061462317aa0d54c2577f5bfb257e9cf58e", GitTreeState:"clean", BuildDate:"2018-01-08T18:21:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

[nikhita]

Looks like the real issue here is that CRD age does not change from <invalid> as soon as it becomes Established.

Can someone update the PR title to reflect the same? Thanks!

[nikhita]

The age problem could be there, but I'm not sure what's the root of the problem there. I have left some comments on this earlier, in this issue.

This is still pending.

I'm going to explicitly unassign myself here, since it has the help-wanted label (and could use some help for the above). Will be involved in this though.
/unassign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[xmudrii]

/remove-lifecycle stale

The issue #65517 describes similar problem, so maybe one can be closed.

[nikhita]

The issue #65517 describes similar problem, so maybe one can be closed.

I think the underlying issue should be the same, but let's keep it open until we confirm this.

Removing help because we don't have explicit directions about solving this yet.
/remove-help

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[roycaihw]

/remove-lifecycle rotten

[liggitt]

#69654 is tracking the established condition transition time

Age: <invalid> means that you are using a version of kubectl older than 1.11, and that your client clock is skewed from your server clock.

Supported versions of kubectl (1.12+) get display text from the server and do not have this issue.

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

#69654 is tracking the established condition transition time

Age: <invalid> means that you are using a version of kubectl older than 1.11, and that your client clock is skewed from your server clock.

Supported versions of kubectl (1.12+) get display text from the server and do not have this issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.