Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1beta1 failed mutation requests #4533

Closed
dgerd opened this issue Jun 25, 2019 · 12 comments

Comments

Projects
None yet
5 participants
@dgerd
Copy link
Contributor

commented Jun 25, 2019

In what area(s)?

/area API

What version of Knative?

0.7
HEAD

Expected Behavior

Updating Knative Services through the v1beta1 endpoint works.

Actual Behavior

Editing a Knative Service objects returns an error from the API Server.

$ kubectl edit ksvc runtime-test-image
error: services.serving.knative.dev "runtime-test-image" could not be patched: Internal error occurred: no kind "Service" is registered for version "serving.knative.dev/v1beta1" in scheme "k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:51"

Steps to Reproduce the Problem

Versions:

Client Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.8-dispatcher", GitCommit:"1215389331387f57594b42c5dd024a2fe27334f8", GitTreeState:"clean", BuildDate:"2019-05-13T18:09:56Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-gke.24", GitCommit:"8d9b8641e72cf7c96efa61421e87f96387242ba1", GitTreeState:"clean", BuildDate:"2019-06-22T02:44:12Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

This is how my cluster got into this state. Not sure which parts are relevant.

  1. Create Knative Cluster with just the v1alpha1 endpoint (such as Knative 0.6)
  2. Create Knative Services in the cluster
  3. Update cluster to either HEAD or Knative 0.7 which enables the v1beta1 endpoint
  4. Attempt to edit existing services

Workaround

  1. Use Kubernetes 1.14+ (This has not been verified)
  2. On your first edit of your Knative Service also update the apiVersion to v1alpha1. After performing this once future edits should work correctly.
apiVersion: serving.knative.dev/v1beta1 <--- Change this to v1alpha1
...
spec:
  template:
    spec:
      containers:
      - image: gcr.io/dangerd-dev/runtime:v2.0 <--- While updating your spec

Related

kubernetes/kubernetes#73752
kubernetes/kubernetes#74154

@dgerd dgerd added the kind/bug label Jun 25, 2019

@mattmoor mattmoor added this to the Serving 0.8 milestone Jun 25, 2019

@dgerd

This comment has been minimized.

Copy link
Contributor Author

commented Jun 25, 2019

Easiest way I have reproduced this.

  1. Create a new Service
  2. Run the command below (Removes v1beta1 endpoint and then adds it back)
sed -e '28,30d' config/300-service.yaml | kubectl apply -f - && kubectl apply -f config/300-service.yaml
  1. Attempt to edit your Service
@dgerd

This comment has been minimized.

Copy link
Contributor Author

commented Jun 26, 2019

Validated that I cannot reproduce the bug in a 1.14 cluster

@mattmoor

This comment has been minimized.

Copy link
Member

commented Jun 26, 2019

@dgerd We should talk about this at the API WG tomorrow. Hopefully we'll know some more by then, but we should try to make some sort of determination on this by next Tuesday in case we want to patch 0.7

@mattmoor

This comment has been minimized.

Copy link
Member

commented Jun 26, 2019

Here's how this is shaping up in my head....

By dark launching v1beta1 in 0.7 we bumped our minimum required Kubernetes version from 1.11 to 1.14 (the version where the fix landed), this was unintentional.

The "fix" for this is a relatively simple config-only change: strip these lines, and disable some tests. In a perfect world we'd have the operator already, and we could optionally enable v1beta1 in 1.14+ clusters, but alas that's not where we are today.

So I think we have a few options:

  1. Rollback the endpoint changes and disable tests,
  2. Release two version of the YAML (pre-/post-1.14)
  3. Move our minimum version to 1.14

At this point, I think 1.14 is far too new for 3. I don't like 1. because I think the messaging should be that things are beta and should be treated as such, even if Kubernetes struggles to support it 🙄.

So I think 2. is probably my favorite of bad options, even though it adds some debt for us to carry until we have an operator or a minimum version of 1.14.

🤷‍♂

@dgerd

This comment has been minimized.

Copy link
Contributor Author

commented Jun 26, 2019

WG Notes:

  • Versioned clients -- what happens?
  • Hard cut over upgrade -- Is it even possible? Does it work?
@steren

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2019

Can you detail what "2. Release two version of the YAML (pre-/post-1.14)" means exactly?

@dgerd

This comment has been minimized.

Copy link
Contributor Author

commented Jun 26, 2019

  1. Release two version of the YAML (pre-/post-1.14)

The idea would be to release something like:

  • release-0.7-pre1.14.yaml
  • release-0.7-1.14.yaml

instead of our normal:

release-0.7.yaml

The diff between the two release YAMLs would be the addition of the "v1beta1" endpoint in the 1.14 version. The 1.14 version would have these three lines (L28-30) in each of the CRD objects to enable the endpoint: https://github.com/knative/serving/blob/master/config/300-service.yaml#L28

The idea is that:

  1. We can continue to use the v1beta1 endpoint in our e2e and conformance tests to gain confidence as we develop Knative 0.8 by setting up a 1.14 test environment.
  2. Users and vendors can opt to use the v1beta1 endpoint on newer clusters.
  3. K8s 1.11-1.13 clusters will receive all the new features minus the new endpoint

The downsides as discussed in the API working group meeting is:

  1. Complicates documentation as your experience depends now on Knative + K8s version
  2. Behavior might be unexpected when using samples or different clients

We did not reach a conclusion between option 1 and option 2 in the meeting. There is also likely some space between these options (i.e. setup 1.14 test cluster with v1beta1, but don't create a release yaml for it).

@dgerd

This comment has been minimized.

Copy link
Contributor Author

commented Jun 26, 2019

Versioned clients -- what happens?

I tested using kn to update a broken service. Using kn works to update the Service and 'fixes' it so future kubectl edits work. This seems to confirm our hypothesis that using versioned clients succeeds and is very likely why we did not catch this earlier in our upgrade tests.

Example:

$ kubectl edit ksvc
error: services.serving.knative.dev "runtime-test-image" could not be patched: Internal error occurred: no kind "Service" is registered for version "serving.knative.dev/v1beta1" in scheme "k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:51"

$ ./kn service update runtime-test-image --limits-cpu 1000m
Service 'runtime-test-image' updated in namespace 'default'.

$ kubectl edit ksvc
service.serving.knative.dev/runtime-test-image edited
@dprotaso

This comment has been minimized.

Copy link
Member

commented Jun 26, 2019

@dgerd Out of curiosity does a versioned v1beta1 client work? (Assuming no)

@dgerd

This comment has been minimized.

Copy link
Contributor Author

commented Jun 26, 2019

mattmoor added a commit to mattmoor/serving that referenced this issue Jun 27, 2019

Separately release our CRDs.
This change has two main parts:
1. Create separate release artifacts for our CRDs,
1. Create two versions of our CRD definitions, with and without v1beta1.

The `v1alpha1` CRDs are generated during `./hack/update-codegen.sh` from the
`v1beta1` CRDs, so we still have a single source of truth (e.g. for columns
and whatnot).

For now, `config/` contains symlinks of the `v1alpha1` CRDs, which are the most
broadly compatible, but we symlink the `v1beta1` CRDs into `test/config` so they
are available to run the `./test/conformance/api/v1beta1` tests.  We can explore
more exotic variations in the future, but this seemed the simplest short-term
change.

Related to: knative#4533

mattmoor added a commit to mattmoor/serving that referenced this issue Jun 27, 2019

Separately release our CRDs.
This change has two main parts:
1. Create separate release artifacts for our CRDs,
1. Create two versions of our CRD definitions, with and without v1beta1.

The `v1alpha1` CRDs are generated during `./hack/update-codegen.sh` from the
`v1beta1` CRDs, so we still have a single source of truth (e.g. for columns
and whatnot).

For now, `config/` contains symlinks of the `v1alpha1` CRDs, which are the most
broadly compatible, but we symlink the `v1beta1` CRDs into `test/config` so they
are available to run the `./test/conformance/api/v1beta1` tests.  We can explore
more exotic variations in the future, but this seemed the simplest short-term
change.

Related to: knative#4533
@dgerd

This comment has been minimized.

Copy link
Contributor Author

commented Jun 27, 2019

The current thought on this is:

  1. Release both a 0.7.1 "beta" yaml and an "alpha" yaml. This lets users and vendors choose which API versions to enable. The "beta" yaml will have both v1alpha1 and v1beta1 endpoints enabled. The "alpha" yaml should work on all K8s versions 1.11+ and "beta" yaml should work on all K8s versions 1.14+.
  2. The "default" CRDs (i.e. ko apply -f config/) will be the v1alpha1 crds
  3. We will revert all knative documentation and samples to use v1alpha1 as the "apiVersion". This provides the most interoperability and portability as it will work against all Knative 0.7.1 installs.
  4. We will add additional information about the "alpha" and "beta" endpoints and yamls in the Knative Serving documentation.

As we approach the release of Knative 0.8 we should re-evaluate the need to ship separate YAMLs based upon the adoption of Kubernetes 1.14 across users and cloud vendors.

mattmoor added a commit to mattmoor/serving that referenced this issue Jun 28, 2019

Separately release our CRDs.
This change has two main parts:
1. Create separate release artifacts for our CRDs,
1. Create two versions of our CRD definitions, with and without v1beta1.

The `v1alpha1` CRDs are generated during `./hack/update-codegen.sh` from the
`v1beta1` CRDs, so we still have a single source of truth (e.g. for columns
and whatnot).

`ko apply -f config` is no longer sufficient to install Knative.  You need
to install either `ko apply -f config/v1alpha1` of `ko apply -f config/v1beta1`.
I have added a flag to select the install for e2e, which defaults to beta
to maintain our current test coverage.  We should add a leg that just runs with
alpha to ensure we don't regress alpha-only installs.

Related to: knative#4533

mattmoor added a commit to mattmoor/serving that referenced this issue Jun 28, 2019

Separately release our CRDs.
This change has two main parts:
1. Create separate release artifacts for our CRDs,
1. Create two versions of our CRD definitions, with and without v1beta1.

The `v1alpha1` CRDs are generated during `./hack/update-codegen.sh` from the
`v1beta1` CRDs, so we still have a single source of truth (e.g. for columns
and whatnot).

`ko apply -f config` is no longer sufficient to install Knative.  You need
to install either `ko apply -f config/v1alpha1` of `ko apply -f config/v1beta1`.
I have added a flag to select the install for e2e, which defaults to beta
to maintain our current test coverage.  We should add a leg that just runs with
alpha to ensure we don't regress alpha-only installs.

Related to: knative#4533

mattmoor added a commit to mattmoor/serving that referenced this issue Jun 28, 2019

Separately release our CRDs.
This change has two main parts:
1. Create separate release artifacts for our CRDs,
1. Create two versions of our CRD definitions, with and without v1beta1.

The `v1alpha1` CRDs are generated during `./hack/update-codegen.sh` from the
`v1beta1` CRDs, so we still have a single source of truth (e.g. for columns
and whatnot).

`ko apply -f config` is no longer sufficient to install Knative.  You need
to install either `ko apply -f config/v1alpha1` of `ko apply -f config/v1beta1`.
I have added a flag to select the install for e2e, which defaults to beta
to maintain our current test coverage.  We should add a leg that just runs with
alpha to ensure we don't regress alpha-only installs.

Related to: knative#4533

knative-prow-robot added a commit that referenced this issue Jun 28, 2019

Separately release our CRDs. (#4552)
This change has two main parts:
1. Create separate release artifacts for our CRDs,
1. Create two versions of our CRD definitions, with and without v1beta1.

The `v1alpha1` CRDs are generated during `./hack/update-codegen.sh` from the
`v1beta1` CRDs, so we still have a single source of truth (e.g. for columns
and whatnot).

`ko apply -f config` is no longer sufficient to install Knative.  You need
to install either `ko apply -f config/v1alpha1` of `ko apply -f config/v1beta1`.
I have added a flag to select the install for e2e, which defaults to beta
to maintain our current test coverage.  We should add a leg that just runs with
alpha to ensure we don't regress alpha-only installs.

Related to: #4533

mattmoor added a commit to mattmoor/serving that referenced this issue Jun 28, 2019

Separately release our CRDs. (knative#4552)
This change has two main parts:
1. Create separate release artifacts for our CRDs,
1. Create two versions of our CRD definitions, with and without v1beta1.

The `v1alpha1` CRDs are generated during `./hack/update-codegen.sh` from the
`v1beta1` CRDs, so we still have a single source of truth (e.g. for columns
and whatnot).

`ko apply -f config` is no longer sufficient to install Knative.  You need
to install either `ko apply -f config/v1alpha1` of `ko apply -f config/v1beta1`.
I have added a flag to select the install for e2e, which defaults to beta
to maintain our current test coverage.  We should add a leg that just runs with
alpha to ensure we don't regress alpha-only installs.

Related to: knative#4533

knative-prow-robot added a commit that referenced this issue Jun 29, 2019

Separately release our CRDs. (#4552) (#4570)
This change has two main parts:
1. Create separate release artifacts for our CRDs,
1. Create two versions of our CRD definitions, with and without v1beta1.

The `v1alpha1` CRDs are generated during `./hack/update-codegen.sh` from the
`v1beta1` CRDs, so we still have a single source of truth (e.g. for columns
and whatnot).

`ko apply -f config` is no longer sufficient to install Knative.  You need
to install either `ko apply -f config/v1alpha1` of `ko apply -f config/v1beta1`.
I have added a flag to select the install for e2e, which defaults to beta
to maintain our current test coverage.  We should add a leg that just runs with
alpha to ensure we don't regress alpha-only installs.

Related to: #4533
@dgerd

This comment has been minimized.

Copy link
Contributor Author

commented Jul 2, 2019

0.7.1 has been cut. Closing this issue.

@dgerd dgerd closed this Jul 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.