`kubectl apply deployment -f` doesn't accept label/selector changes #26202

lavalamp · 2016-05-24T19:43:37Z

I got errors complaining about the selector and labels not matching even if I explicitly specified both. Showed @janetkuo.

janetkuo · 2016-05-24T22:29:19Z

To reproduce:

First create a Deployment via apply -f

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
        test: abcd
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Then remove any of the labels in the file (and optionally specify the selector), and then apply -f

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Error message:

proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: tag has too few fields: "-"
proto: no coders for struct *reflect.rtype
proto: no encoder for sec int64 [GetProperties]
proto: no encoder for nsec int32 [GetProperties]
proto: no encoder for loc *time.Location [GetProperties]
proto: no encoder for Time time.Time [GetProperties]
proto: no encoder for InitContainers []v1.Container [GetProperties]
proto: no coders for intstr.Type
proto: no encoder for Type intstr.Type [GetProperties]
The Deployment "nginx-deployment" is invalid.
spec.template.metadata.labels: Invalid value: {"app":"nginx"}: `selector` does not match template `labels`

@kubernetes/kubectl @adohe

smarterclayton · 2016-05-24T22:50:47Z

Is there a bug already open for all that proto crap? I had not seen that
showing up before - I would not expect it.

On May 24, 2016, at 6:29 PM, Janet Kuo notifications@github.com wrote:

To reproduce:

First create a Deployment via apply -f

apiVersion: extensions/v1beta1kind: Deploymentmetadata:
name: nginx-deploymentspec:
replicas: 3
template:
metadata:
labels:
app: nginx
test: abcd
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80

Then remove any of the labels in the file, and apply -f

$ kubectl apply -f docs/user-guide/nginx-deployment.yaml proto: no
encoder for TypeMeta unversioned.TypeMeta [GetProperties]proto: tag
has too few fields: "-"proto: no coders for struct
*reflect.rtypeproto: no encoder for sec int64 [GetProperties]proto: no
encoder for nsec int32 [GetProperties]proto: no encoder for loc
*time.Location [GetProperties]proto: no encoder for Time time.Time
[GetProperties]proto: no encoder for InitContainers []v1.Container
[GetProperties]proto: no coders for intstr.Typeproto: no encoder for
Type intstr.Type [GetProperties]The Deployment "nginx-deployment" is
invalid.spec.template.metadata.labels: Invalid value: {"test":"abcd"}:
selector does not match template labels

@kubernetes/kubectl https://github.com/orgs/kubernetes/teams/kubectl
@adohe https://github.com/AdoHe

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly or view it on GitHub
#26202 (comment)

janetkuo · 2016-05-24T22:56:12Z

The problem is that the apply annotation didn't record .spec.selector when it's defaulted from .spec.template.labels and it caused problems when calculating diff:

kubectl.kubernetes.io/last-applied-configuration: '{"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"nginx-deployment","creationTimestamp":null},"spec":{"replicas":3,"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"nginx","test":"abcd"}},"spec":{"containers":[{"name":"nginx","image":"nginx:1.7.9","ports":[{"containerPort":80}],"resources":{}}]}},"strategy":{}},"status":{}}'

adohe-zz · 2016-05-25T04:00:28Z

That's because the last-applied-configuration is just the resource content read from user input when we use apply to create resource. Also the .spec.selector is derived from .spec.template.labels rule is not used in client side.

adohe-zz · 2016-05-25T06:04:39Z

@janetkuo I am considering adding the default .spec.selector to last-applied-configuration annotation WDYT?

therc · 2016-05-26T17:06:17Z

We've been bitten by this for a service:

The Service "labeler" is invalid.

* metadata.resourceVersion: Invalid value: "": must be specified for an update
* spec.clusterIP: Invalid value: "": field is immutable

I just nuked the last-applied-configuration annotation and the change applied successfully.

therc · 2016-05-26T17:07:10Z

And before that error, there was the very same sequence of proto errors.

lavalamp · 2016-05-26T17:52:25Z

Please note if you want to get defaulted stuff, it's super important to let the server do the defaulting.

adohe-zz · 2016-05-27T02:29:47Z

@lavalamp yes, we should put this in server side.

adohe-zz · 2016-05-27T08:05:19Z

@bgrant0607 @janetkuo @lavalamp I thought about this carefully, I think the root cause of both this issue and #24198 is last-applied-configuration annotation doesn't include fields which has default value, e.g spec.selector and spec.strategy. Users may not specify these fields in the original resource file, but we will set in the client side. I have opened a PR #26405 to indicate this problem.

bgrant0607 · 2016-05-27T14:44:11Z

@adohe We should not add default values to last-applied-configuration. Unfortunately the design hasn't been written up yet, but there's some discussion on #1702, I think.

bgrant0607 · 2016-05-27T14:45:44Z

Changing selectors has other problems, also. For instance, it's a map, and maps are merged by default.

Also note that a default selector won't be changed automatically when pod template labels are changed.

bgrant0607 · 2016-05-27T16:38:11Z

I wrote a brief explanation here: #15894 (comment)

metral · 2016-05-27T17:56:46Z

Ran into this myself. In the meantime is there a work around that doesn't require blowing away last-applied-configuration?

bgrant0607 · 2016-05-27T21:51:27Z

Accidental orphaning is another problem: #24888

bgrant0607 · 2016-05-27T22:14:04Z

@metral What problem did you encounter?

bgrant0607 · 2016-05-27T22:14:57Z

@lavalamp We capture the last-applied-configuration prior to defaulting, so it shouldn't matter whether the defaulting happens in the client or server, though obviously I'd prefer the latter.

bgrant0607 · 2016-05-27T22:22:50Z

Controlled pods are identified by labels, not by name.

In general, changing the labels and selector on a controller isn't expected to work. To do it properly requires a multi-step dance. In any case, controllers don't update running pods -- for the most part, not even Deployment does that.

We should strongly recommend that users shouldn't do that, unless they use kubectl replace --force, which does a cascading delete and re-create.

metral · 2016-05-27T22:23:36Z

@bgrant0607 I'm hitting the same issue as @janetkuo described when I try changing a label. In my case it's a label I use for the revision/commit of the current build which gets altered in the deployment manifest upon a successful build so that I can apply the new deployment.

Oddly enough, this same strategy is working on 2 other projects / deployments that I have in the system just fine, but for some reason on my 3rd one I'm hitting this issue. I'm sure it's something on my end that I haven't caught onto, but strange nevertheless that updating the rev & applying the new deployment works for some and not others

bgrant0607 · 2016-05-27T22:25:08Z

@metral Specify a selector that omits the label you plan to change.

metral · 2016-05-27T22:28:03Z

@bgrant0607 that makes sense - I'll give that a shot when I'm back at my computer.

Just odd that I've never specified the selector in my working deployments, just the labels in the metadata and changing the revs on those has never caused an issue until now with my new projects deployment

bgrant0607 · 2016-05-28T00:01:43Z

I think the longer-term solution will be to move to a defaulting approach similar to what we use for Job:
https://github.com/kubernetes/kubernetes/blob/master/docs/design/selector-generation.md

ltupin · 2018-10-24T15:07:22Z

disallowing mutation on those fields

Why do we want this immutable ?

janetkuo · 2018-10-26T21:14:26Z

@ltupin because it's not safe to mutate a controller's selector. For example, deployment finds its children resources (replicasets) by label/selector. Changing a deployment's selector is dangerous. It could cause the deployment to be unable to find its existing children and therefore cause disruptions to your running workloads. We don't want to encourage that.

Sergey80 · 2018-12-29T21:48:21Z

still reproducible in apps/v1beta2

apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: $APP_NAME-deployment
labels:
app: $APP_NAME-k8s ...

Or maybe I did not get @liggitt - what exactly I should do with the fields..

liggitt · 2018-12-29T22:00:41Z

Mutation of label selectors on deployments, replicasets, and daemonsets is not allowed in apps/v1beta2 and forward

liggitt · 2018-12-29T22:01:26Z

See #50808 for details

In certain situations (see details and repro below), the deployment to Kubernetes fails with "The Deployment [DEPLOYMENT_OBJECT] is invalid: [...] `selector` does not match template `labels`". This is caused by the K8S Deployment manifests missing an explicit `selector` value. This commit adds explicit `selector` values for all Deployment objects. It also bumps the K8S API from the now deprecated `extensions/v1beta1` version to the current stable `apps/v1` version. This version made the `selector` property of the Deployment a required value, preventing any further issues with missing selectors in the future. This change is backwards compatible with existing deployments of the microservices demo app. I.e. you should be able to pull this change and run `skaffold run` against an existing deployment of the app without issues. This will not however fix the damage done by the defaulted Deployment selectors in existing deployments of the app (see below for an explanation of the issue). Selectors are immutable and will therefore retain their current defaulted value. Which means that you'll still run into the error above. You should run `skaffold delete` followed by `skaffold run` after having pulled this change to do a clean re-deployment of the app, which will resolve the issue. In the `extensions/v1beta1` version of K8S API (the version that was used by this project), the `selector` property of a Deployment object is optional and is defaulted to the labels used in the pod template. This can cause subtle issues leading to deployment failures. This project, where Deployment selectors were omitted, is a good example of what can go wrong with defaulted selectors. Consider this: 1. Run `skaffold run` to build locally with Docker and deploy. Since the Deployment specs don't have explict selectors, they will be defaulted to the pod template labels. And since skaffold adds additional labels to the pod template like `skaffold-builder` and `skaffold-deployer`, the end-result will be a selector that looks like this: ``` app=cartservice,cleanup=true,docker-api-version=1.39,skaffold-builder=local,skaffold-deployer=kubectl,skaffold-tag-policy=git-commit,tail=true ``` So far, so good. 2. Now run `skaffold run -p gcb --default-repo=your-gcr-repo` to build on Google Cloud Build instead of building locally. This will blow up when attempting to deploy to Kubernetes with an error similar to: ``` The Deployment "cartservice" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"skaffold-builder":"google-cloud-build", "profiles"="gcb", "skaffold-deployer":"kubectl", "skaffold-tag-policy":"git-commit", "docker-api-version":"1.39", "tail":"true", "app":"cartservice", "cleanup":"true"}: `selector` does not match template `labels` ``` This is because the skaffold labels that were automatically added to the pod template have changed to include references to Google Cloud Build. That normally shouldn't be an issue. But without explicit Deployment selectors, this results in the defaulted selectors for our Deployment objects to have also changed. Which means that the new version of our Deployment objects are now managing different sets of Pods. Which is thankfully caught by kubectl before the deployment happens (otherwise this would have resulted in orphaned pods). In this commit, we explicitely set the `selector` value of all Deployment objects, which fixes this issue. We also bump the K8S API version to the stable `apps/v1`, which makes the `selector` property a required value and will avoid accidently forgetting selectors in the future. More details if your're curious: * Why defaulted Deployment selectors cause problems: kubernetes/kubernetes#26202 * Why Deployment selectors should be (and were made) immutable: kubernetes/kubernetes#50808

In certain situations (see details below), the deployment to Kubernetes fails with: > "The Deployment [DEPLOYMENT_OBJECT] is invalid: [...] `selector` does not match template `labels`". This is caused by the K8S Deployment manifests missing an explicit `selector` value. This commit: * adds explicit `selector` values for all Deployment objects. * bumps the K8S API from the deprecated `extensions/v1beta1` version to the stable `apps/v1` version. This version made the `selector` property of the Deployment a required value, preventing any further issues with missing selectors in the future. This change is backwards compatible with existing deployments of the microservices demo app. I.e. you should be able to pull this change and run `skaffold run` against an existing deployment of the app without issues. This will not however resolve the issue for existing deployments of the app. Selectors are immutable and will therefore retain their current defaulted value. You should run `skaffold delete` followed by `skaffold run` after having pulled this change to do a clean re-deployment of the app, which will resolve the issue. **The nitty-gritty details** In the `extensions/v1beta1` version of K8S API (the version that was used by this project), the `selector` property of a Deployment object is optional and is defaulted to the labels used in the pod template. This can cause subtle issues leading to deployment failures. This project, where Deployment selectors were omitted, is a good example of what can go wrong with defaulted selectors. Consider this: 1. Run `skaffold run` to build locally with Docker and deploy. Since the Deployment specs don't have explict selectors, they will be defaulted to the pod template labels. And since skaffold adds additional labels to the pod template like `skaffold-builder` and `skaffold-deployer`, the end-result will be a selector that looks like this: > app=cartservice,cleanup=true,docker-api-version=1.39,skaffold-builder=local,skaffold-deployer=kubectl,skaffold-tag-policy=git-commit,tail=true So far, so good. 2. Now run `skaffold run -p gcb --default-repo=your-gcr-repo` to build on Google Cloud Build instead of building locally. This will blow up when attempting to deploy to Kubernetes with an error similar to: > The Deployment "cartservice" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"skaffold-builder":"google-cloud-build", "profiles"="gcb", "skaffold-deployer":"kubectl", "skaffold-tag-policy":"git-commit", "docker-api-version":"1.39", "tail":"true", "app":"cartservice", "cleanup":"true"}: `selector` does not match template `labels` This is because the skaffold labels that were automatically added to the pod template have changed to include references to Google Cloud Build. That normally shouldn't be an issue. But without explicit Deployment selectors, this results in the defaulted selectors for our Deployment objects to have also changed. Which means that the new version of our Deployment objects are now managing different sets of Pods. Which is thankfully caught by kubectl before the deployment happens (otherwise this would have resulted in orphaned pods). In this commit, we explicitely set the `selector` value of all Deployment objects, which fixes this issue. We also bump the K8S API version to the stable `apps/v1`, which makes the `selector` property a required value and will avoid accidently forgetting selectors in the future. More details if your're curious: * Why defaulted Deployment selectors cause problems: kubernetes/kubernetes#26202 * Why Deployment selectors should be (and were made) immutable: kubernetes/kubernetes#50808

In certain situations (see details below), the deployment to Kubernetes fails with: > "The Deployment [DEPLOYMENT_OBJECT] is invalid: [...] `selector` does not match template `labels`". This is caused by the K8S Deployment manifests missing an explicit `selector` value. This commit: * adds explicit `selector` values for all Deployment objects. * bumps the K8S API from the deprecated `extensions/v1beta1` version to the stable `apps/v1` version. This version made the `selector` property of the Deployment a required value, preventing any further issues with missing selectors in the future. This change is backwards compatible with existing deployments of the microservices demo app. I.e. you should be able to pull this change and run `skaffold run` against an existing deployment of the app without issues. This will not however resolve the issue for existing deployments of the app. Selectors are immutable and will therefore retain their current defaulted value. You should run `skaffold delete` followed by `skaffold run` after having pulled this change to do a clean re-deployment of the app, which will resolve the issue. In the `extensions/v1beta1` version of K8S API (the version that was used by this project), the `selector` property of a Deployment object is optional and is defaulted to the labels used in the pod template. This can cause subtle issues leading to deployment failures. This project, where Deployment selectors were omitted, is a good example of what can go wrong with defaulted selectors. Consider this: 1. Run `skaffold run` to build locally with Docker and deploy. Since the Deployment specs don't have explict selectors, they will be defaulted to the pod template labels. And since skaffold adds additional labels to the pod template like `skaffold-builder` and `skaffold-deployer`, the end-result will be a selector that looks like this: > app=cartservice,cleanup=true,docker-api-version=1.39,skaffold-builder=local,skaffold-deployer=kubectl,skaffold-tag-policy=git-commit,tail=true So far, so good. 2. Now run `skaffold run -p gcb --default-repo=your-gcr-repo` to build on Google Cloud Build instead of building locally. This will blow up when attempting to deploy to Kubernetes with an error similar to: > The Deployment "cartservice" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"skaffold-builder":"google-cloud-build", "profiles"="gcb", "skaffold-deployer":"kubectl", "skaffold-tag-policy":"git-commit", "docker-api-version":"1.39", "tail":"true", "app":"cartservice", "cleanup":"true"}: `selector` does not match template `labels` (and the same error for every other deployment object) This is because the skaffold labels that were automatically added to the pod template have changed to include references to Google Cloud Build. That normally shouldn't be an issue. But without explicit Deployment selectors, this results in the defaulted selectors for our Deployment objects to have also changed. Which means that the new version of our Deployment objects are now managing different sets of Pods. Which is thankfully caught by kubectl before the deployment happens (otherwise this would have resulted in orphaned pods). In this commit, we explicitely set the `selector` value of all Deployment objects, which fixes this issue. We also bump the K8S API version to the stable `apps/v1`, which makes the `selector` property a required value and will avoid accidently forgetting selectors in the future. More details if your're curious: * Why defaulted Deployment selectors cause problems: kubernetes/kubernetes#26202 * Why Deployment selectors should be (and were made) immutable: kubernetes/kubernetes#50808

In certain situations (see details below), the deployment to Kubernetes fails with: > "The Deployment [DEPLOYMENT_OBJECT] is invalid: [...] `selector` does not match template `labels`". This is caused by the K8S Deployment manifests missing an explicit `selector` value. This commit: * adds explicit `selector` values for all Deployment objects. * bumps the K8S API from the deprecated `extensions/v1beta1` version to the stable `apps/v1` version. This version made the `selector` property of the Deployment a required value, preventing any further issues with missing selectors in the future. This change is backwards compatible with existing deployments of the microservices demo app. I.e. you should be able to pull this change and run `skaffold run` against an existing deployment of the app without issues. This will not however resolve the issue for existing deployments of the app. Selectors are immutable and will therefore retain their current defaulted value. You should run `skaffold delete` followed by `skaffold run` after having pulled this change to do a clean re-deployment of the app, which will resolve the issue. |**The nitty-gritty details** In the `extensions/v1beta1` version of K8S API (the version that was used by this project), the `selector` property of a Deployment object is optional and is defaulted to the labels used in the pod template. This can cause subtle issues leading to deployment failures. This project, where Deployment selectors were omitted, is a good example of what can go wrong with defaulted selectors. Consider this: 1. Run `skaffold run` to build locally with Docker and deploy. Since the Deployment specs don't have explict selectors, they will be defaulted to the pod template labels. And since skaffold adds additional labels to the pod template like `skaffold-builder` and `skaffold-deployer`, the end-result will be a selector that looks like this: > app=cartservice,cleanup=true,docker-api-version=1.39,skaffold-builder=local,skaffold-deployer=kubectl,skaffold-tag-policy=git-commit,tail=true So far, so good. 2. Now run `skaffold run -p gcb --default-repo=your-gcr-repo` to build on Google Cloud Build instead of building locally. This will blow up when attempting to deploy to Kubernetes with an error similar to: > The Deployment "cartservice" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"skaffold-builder":"google-cloud-build", "profiles"="gcb", "skaffold-deployer":"kubectl", "skaffold-tag-policy":"git-commit", "docker-api-version":"1.39", "tail":"true", "app":"cartservice", "cleanup":"true"}: `selector` does not match template `labels` (and the same error for every other deployment object) This is because the skaffold labels that were automatically added to the pod template have changed to include references to Google Cloud Build. That normally shouldn't be an issue. But without explicit Deployment selectors, this results in the defaulted selectors for our Deployment objects to have also changed. Which means that the new version of our Deployment objects are now managing different sets of Pods. Which is thankfully caught by kubectl before the deployment happens (otherwise this would have resulted in orphaned pods). In this commit, we explicitely set the `selector` value of all Deployment objects, which fixes this issue. We also bump the K8S API version to the stable `apps/v1`, which makes the `selector` property a required value and will avoid accidently forgetting selectors in the future. More details if your're curious: * Why defaulted Deployment selectors cause problems: kubernetes/kubernetes#26202 * Why Deployment selectors should be (and were made) immutable: kubernetes/kubernetes#50808

In certain situations (see details below), the deployment to Kubernetes fails with: > "The Deployment [DEPLOYMENT_OBJECT] is invalid: [...] `selector` does not match template `labels`". This is caused by the K8S Deployment manifests missing an explicit `selector` value. This commit: * adds explicit `selector` values for all Deployment objects. * bumps the K8S API from the deprecated `extensions/v1beta1` version to the stable `apps/v1` version. This version made the `selector` property of the Deployment a required value, preventing any issues with missing selectors in the future. This change is backwards compatible with existing deployments of the microservices demo app. I.e. you should be able to pull this change and run `skaffold run` against an existing deployment of the app without issues. This will not however resolve the issue for existing deployments. Selectors are immutable and will therefore retain their current defaulted value. You should run `skaffold delete` followed by `skaffold run` after having pulled this change to do a clean re-deployment of the app, which will resolve the issue. **The nitty-gritty details** In the `extensions/v1beta1` version of K8S API (the version that was used by this project), the `selector` property of a Deployment object is optional and is defaulted to the labels used in the pod template. This can cause subtle issues leading to deployment failures. This project, where Deployment selectors were omitted, is a good example of what can go wrong with defaulted selectors. Consider this: 1. Run `skaffold run` to build locally with Docker and deploy. Since the Deployment specs don't have explict selectors, they will be defaulted to the pod template labels. And since skaffold adds additional labels to the pod template like `skaffold-builder` and `skaffold-deployer`, the end-result will be a selector that looks like this: > app=cartservice,cleanup=true,docker-api-version=1.39,skaffold-builder=local,skaffold-deployer=kubectl,skaffold-tag-policy=git-commit,tail=true So far, so good. 2. Now run `skaffold run -p gcb --default-repo=your-gcr-repo` to build on Google Cloud Build instead of building locally. This will blow up when attempting to deploy to Kubernetes with an error similar to: > The Deployment "cartservice" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"skaffold-builder":"google-cloud-build", "profiles"="gcb", "skaffold-deployer":"kubectl", "skaffold-tag-policy":"git-commit", "docker-api-version":"1.39", "tail":"true", "app":"cartservice", "cleanup":"true"}: `selector` does not match template `labels` (and the same error for every other deployment object) This is because the skaffold labels that were automatically added to the pod template have changed to include references to Google Cloud Build. That normally shouldn't be an issue. But without explicit Deployment selectors, this results in the defaulted selectors for our Deployment objects to have also changed. Which means that the new version of our Deployment objects are now managing different sets of Pods. Which is thankfully caught by kubectl before the deployment happens (otherwise this would have resulted in orphaned pods). In this commit, we explicitely set the `selector` value of all Deployment objects, which fixes this issue. We also bump the K8S API version to the stable `apps/v1`, which makes the `selector` property a required value and will avoid accidently forgetting selectors in the future. More details if you're curious: * Why defaulted Deployment selectors cause problems: kubernetes/kubernetes#26202 * Why Deployment selectors should be (and were made) immutable: kubernetes/kubernetes#50808

quinn · 2019-07-24T16:51:04Z

All i'm seeing is k8s members saying this is "discouraged". What if i have to do this? In production? I have to allocate downtime to destroy the deployment and replace with a new one?

cyrilthank · 2019-09-30T04:40:22Z

faced this issue in kubeflow/kubeflow#4184
kubeflow/kubeflow#4184 (comment)

Please help

hmtai · 2020-11-20T08:05:59Z

I encountered the same problem when rollback the chart from the selector manully added chart to no selector chart:
stderr: 'Error: Failed to recreate resource: DaemonSet.apps "internal-management-ingress" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"internal-management-ingress", "chart":"icp-management-ingress", "component":"internal-management-ingress", "heritage":"Tiller", "k8s-app":"internal-management-ingress", "release":"internal-management-ingress"}:selectordoes not match templatelabels'

abhinav054 · 2020-11-20T08:17:28Z

Can you post the yaml file you are trying to apply for?

…

On Fri, Nov 20, 2020 at 1:37 PM huimin ***@***.***> wrote: I encountered the same problem when rollback the chart from the selector manully added chart to no selector chart: stderr: 'Error: Failed to recreate resource: DaemonSet.apps "internal-management-ingress" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"internal-management-ingress", "chart":"icp-management-ingress", "component":"internal-management-ingress", "heritage":"Tiller", "k8s-app":"internal-management-ingress", "release":"internal-management-ingress"}:selectordoes not match template labels' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#26202 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRUTWXRIA5GVUUH3ZNCVULSQYPTNANCNFSM4CEXNUVQ> .

hmtai · 2020-11-20T08:43:32Z

@abhinav054 This is helm rollback process:

helm rollback --debug --tls   --force internal-management-ingress 2
[debug] Created tunnel using local port: '36208'

[debug] SERVER: "127.0.0.1:36208"

[debug] Host="", Key="/root/.helm/key.pem", Cert="/root/.helm/cert.pem", CA="/root/.helm/ca.pem"

Error: failed to create resource: DaemonSet.apps "internal-management-ingress" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"internal-management-ingress", "chart":"icp-management-ingress", "component":"internal-management-ingress", "heritage":"Tiller", "k8s-app":"internal-management-ingress", "release":"internal-management-ingress"}:selectordoes not match templatelabels
My original daemonset yaml didn't include the .spec.selector.My upgrade version include the .spec.selector:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  creationTimestamp: "2020-11-19T07:05:06Z"
  generation: 1
  labels:
    app: internal-management-ingress
    chart: icp-management-ingress-3.3.1910
    component: internal-management-ingress
    heritage: Tiller
    release: internal-management-ingress
  name: internal-management-ingress
  namespace: kube-system
  resourceVersion: "2489"
  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/internal-management-ingress
  uid: 61b47ed7-7f01-407a-8317-3fbba2783af0
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: internal-management-ingress
      chart: icp-management-ingress
      component: internal-management-ingress
      heritage: Tiller
      k8s-app: internal-management-ingress
      release: internal-management-ingress
  template:
    metadata:
      annotations:
        productID: icp-management-ingress_3.3.1910_00000
        productName: icp-management-ingress
        productVersion: 3.3.1910
        seccomp.security.alpha.kubernetes.io/pod: docker/default
      creationTimestamp: null
      labels:
        app: internal-management-ingress
        chart: icp-management-ingress
        component: internal-management-ingress
        heritage: Tiller
        k8s-app: internal-management-ingress
        release: internal-management-ingress

abhinav054 · 2020-11-20T09:00:16Z

Do you have any app with the given template label running.

…

On Fri, 20 Nov 2020, 14:13 huimin, ***@***.***> wrote: @abhinav054 <https://github.com/Abhinav054> This is helm rollback process: helm rollback --debug --tls --force internal-management-ingress 2 [debug] Created tunnel using local port: '36208' [debug] SERVER: "127.0.0.1:36208" [debug] Host="", Key="/root/.helm/key.pem", Cert="/root/.helm/cert.pem", CA="/root/.helm/ca.pem" Error: failed to create resource: DaemonSet.apps "internal-management-ingress" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"internal-management-ingress", "chart":"icp-management-ingress", "component":"internal-management-ingress", "heritage":"Tiller", "k8s-app":"internal-management-ingress", "release":"internal-management-ingress"}: `selector` does not match template `labels` — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26202 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRUTWUFYS7EXN7HSMNCSIDSQYT4RANCNFSM4CEXNUVQ> .

Bjohnson131 · 2021-11-10T07:50:56Z

All i'm seeing is k8s members saying this is "discouraged". What if i have to do this? In production? I have to allocate downtime to destroy the deployment and replace with a new one?

So far the answer seems to be a tentative yes?

In certain situations (see details below), the deployment to Kubernetes fails with: > "The Deployment [DEPLOYMENT_OBJECT] is invalid: [...] `selector` does not match template `labels`". This is caused by the K8S Deployment manifests missing an explicit `selector` value. This commit: * adds explicit `selector` values for all Deployment objects. * bumps the K8S API from the deprecated `extensions/v1beta1` version to the stable `apps/v1` version. This version made the `selector` property of the Deployment a required value, preventing any issues with missing selectors in the future. This change is backwards compatible with existing deployments of the microservices demo app. I.e. you should be able to pull this change and run `skaffold run` against an existing deployment of the app without issues. This will not however resolve the issue for existing deployments. Selectors are immutable and will therefore retain their current defaulted value. You should run `skaffold delete` followed by `skaffold run` after having pulled this change to do a clean re-deployment of the app, which will resolve the issue. **The nitty-gritty details** In the `extensions/v1beta1` version of K8S API (the version that was used by this project), the `selector` property of a Deployment object is optional and is defaulted to the labels used in the pod template. This can cause subtle issues leading to deployment failures. This project, where Deployment selectors were omitted, is a good example of what can go wrong with defaulted selectors. Consider this: 1. Run `skaffold run` to build locally with Docker and deploy. Since the Deployment specs don't have explict selectors, they will be defaulted to the pod template labels. And since skaffold adds additional labels to the pod template like `skaffold-builder` and `skaffold-deployer`, the end-result will be a selector that looks like this: > app=cartservice,cleanup=true,docker-api-version=1.39,skaffold-builder=local,skaffold-deployer=kubectl,skaffold-tag-policy=git-commit,tail=true So far, so good. 2. Now run `skaffold run -p gcb --default-repo=your-gcr-repo` to build on Google Cloud Build instead of building locally. This will blow up when attempting to deploy to Kubernetes with an error similar to: > The Deployment "cartservice" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"skaffold-builder":"google-cloud-build", "profiles"="gcb", "skaffold-deployer":"kubectl", "skaffold-tag-policy":"git-commit", "docker-api-version":"1.39", "tail":"true", "app":"cartservice", "cleanup":"true"}: `selector` does not match template `labels` (and the same error for every other deployment object) This is because the skaffold labels that were automatically added to the pod template have changed to include references to Google Cloud Build. That normally shouldn't be an issue. But without explicit Deployment selectors, this results in the defaulted selectors for our Deployment objects to have also changed. Which means that the new version of our Deployment objects are now managing different sets of Pods. Which is thankfully caught by kubectl before the deployment happens (otherwise this would have resulted in orphaned pods). In this commit, we explicitely set the `selector` value of all Deployment objects, which fixes this issue. We also bump the K8S API version to the stable `apps/v1`, which makes the `selector` property a required value and will avoid accidently forgetting selectors in the future. More details if you're curious: * Why defaulted Deployment selectors cause problems: kubernetes/kubernetes#26202 * Why Deployment selectors should be (and were made) immutable: kubernetes/kubernetes#50808

lavalamp added the team/ux label May 24, 2016

janetkuo added area/app-lifecycle area/kubectl kind/bug Categorizes issue or PR as related to a bug. labels May 24, 2016

pwittrock added the 0 - Backlog label May 24, 2016

pwittrock added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label May 25, 2016

pwittrock assigned adohe-zz May 25, 2016

adohe-zz mentioned this issue May 26, 2016

changing update strategy and "applying" is not allowed #24198

Closed

ayj mentioned this issue Oct 30, 2018

fix 1.0.3 => 1.1.0 upgrade with default helm options istio/istio#9569

Merged

maiamcc mentioned this issue Nov 7, 2018

words: re-add words service tilt-dev/servantes#49

Merged

timcrall added a commit to IGNW/hello-nodejs that referenced this issue Feb 5, 2019

https://github.com/kubernetes/kubernetes/issues/26202

432184b

mehdime mentioned this issue Feb 25, 2019

Fix for "selector does not match template labels" GoogleCloudPlatform/microservices-demo#165

Closed

mehdime mentioned this issue Feb 25, 2019

Fix for "selector does not match template labels" GoogleCloudPlatform/microservices-demo#166

Merged

harshanarayana added a commit to harshanarayana/go-webhook that referenced this issue Sep 14, 2019

🐛 fix kubernetes/kubernetes#26202

b2c29b9

avloss mentioned this issue Oct 27, 2019

Updating Skaffold breaks redeploys since it changes labels GoogleContainerTools/skaffold#3133

Closed

xoen mentioned this issue Jan 9, 2020

App build fails with message about the label ministryofjustice/analytics-platform#115

Closed

Pothulapati mentioned this issue Apr 15, 2020

charts: Using downwardAPI to mount labels to the proxy container linkerd/linkerd2#4199

Merged

paultiplady mentioned this issue May 11, 2020

Error from server (Invalid): error when applying patch: Job.batch "migration-elasticsearch" is invalid: GoogleContainerTools/skaffold#3171

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`kubectl apply deployment -f` doesn't accept label/selector changes #26202

`kubectl apply deployment -f` doesn't accept label/selector changes #26202

lavalamp commented May 24, 2016

janetkuo commented May 24, 2016 •

edited

Loading

smarterclayton commented May 24, 2016

janetkuo commented May 24, 2016

adohe-zz commented May 25, 2016

adohe-zz commented May 25, 2016 •

edited

Loading

therc commented May 26, 2016

therc commented May 26, 2016

lavalamp commented May 26, 2016

adohe-zz commented May 27, 2016

adohe-zz commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

metral commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

metral commented May 27, 2016

bgrant0607 commented May 27, 2016

metral commented May 27, 2016 •

edited

Loading

bgrant0607 commented May 28, 2016

ltupin commented Oct 24, 2018

janetkuo commented Oct 26, 2018

Sergey80 commented Dec 29, 2018 •

edited

Loading

liggitt commented Dec 29, 2018

liggitt commented Dec 29, 2018

quinn commented Jul 24, 2019

cyrilthank commented Sep 30, 2019 •

edited

Loading

hmtai commented Nov 20, 2020

abhinav054 commented Nov 20, 2020 via email

hmtai commented Nov 20, 2020 •

edited

Loading

abhinav054 commented Nov 20, 2020 via email

Bjohnson131 commented Nov 10, 2021

kubectl apply deployment -f doesn't accept label/selector changes #26202

kubectl apply deployment -f doesn't accept label/selector changes #26202

Comments

lavalamp commented May 24, 2016

janetkuo commented May 24, 2016 • edited Loading

smarterclayton commented May 24, 2016

janetkuo commented May 24, 2016

adohe-zz commented May 25, 2016

adohe-zz commented May 25, 2016 • edited Loading

therc commented May 26, 2016

therc commented May 26, 2016

lavalamp commented May 26, 2016

adohe-zz commented May 27, 2016

adohe-zz commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

metral commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

bgrant0607 commented May 27, 2016

metral commented May 27, 2016

bgrant0607 commented May 27, 2016

metral commented May 27, 2016 • edited Loading

bgrant0607 commented May 28, 2016

ltupin commented Oct 24, 2018

janetkuo commented Oct 26, 2018

Sergey80 commented Dec 29, 2018 • edited Loading

liggitt commented Dec 29, 2018

liggitt commented Dec 29, 2018

quinn commented Jul 24, 2019

cyrilthank commented Sep 30, 2019 • edited Loading

hmtai commented Nov 20, 2020

abhinav054 commented Nov 20, 2020 via email

hmtai commented Nov 20, 2020 • edited Loading

abhinav054 commented Nov 20, 2020 via email

Bjohnson131 commented Nov 10, 2021

`kubectl apply deployment -f` doesn't accept label/selector changes #26202

`kubectl apply deployment -f` doesn't accept label/selector changes #26202

janetkuo commented May 24, 2016 •

edited

Loading

adohe-zz commented May 25, 2016 •

edited

Loading

metral commented May 27, 2016 •

edited

Loading

Sergey80 commented Dec 29, 2018 •

edited

Loading

cyrilthank commented Sep 30, 2019 •

edited

Loading

hmtai commented Nov 20, 2020 •

edited

Loading