Update to namespace resource fails with: "the namespace of the object (default) does not match the namespace on the request (foo)"

[geekflyer]

Hi,

found a very strange bug in newer pulumi versions (I'm not 100% sure, but I believe this used to work in earlier pulumi versions). This bug prevents any updates / patch operations to namespace objects when using them in conjunction with a first class provider. Small noteworthy detail: When instantiating the provider itself I'm explicitly setting the optional namespace property on it.

Here are the versions being used:

  "@pulumi/gcp": "0.16.6",
        "@pulumi/kubernetes": "0.20.0",
        "@pulumi/pulumi": "0.16.14"

runtime: nodejs
CLI version: v0.16.14

Reproduction:

1. Create a k8s namespace on a GKE cluster using the code below. You should replace clusterName project and zone with values of an existing GKE cluster. You can probably reproduce the bug with clusters other than GKE as long as you know how to instantiate a first class k8s provider.

import * as k8s from '@pulumi/kubernetes';
import * as gcp from '@pulumi/gcp';

const namespace = 'foo';
const clusterName = 'my-cluster';
const project = 'my-project';
const zone = 'us-west1-b';

const k8sProvider = new k8s.Provider(clusterName, {
  namespace: namespace,
  kubeconfig: gcp.container
    .getCluster({
      name: clusterName,
      project,
      zone
    })
    .then(({ endpoint, masterAuths }) => {
      const context = `dynamic-context`;
      return `apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${masterAuths[0].clusterCaCertificate}
    server: https://${endpoint}
  name: ${context}
contexts:
- context:
    cluster: ${context}
    user: ${context}
  name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp
`;
    })
});

new k8s.core.v1.Namespace(
  namespace,
  {
    metadata: {
      name: namespace
    }
  },
  { provider: k8sProvider }
);

2. Add a label to the namespace resource:

new k8s.core.v1.Namespace(
  namespace,
  {
    metadata: {
      name: namespace,
     // BELOW IS A CHANGE TO THE NAMESPACE OBJECT
      labels: {
         hello: 'world'
      }
    }
  },
  { provider: k8sProvider }
);

3. run pulumi up . It will fail with this message:

Updating (mycorp/pulumi-bug-foo):

     Type                          Name                       Status                  Info
     pulumi:pulumi:Stack           pulumi-bug-pulumi-bug-foo
 ~   └─ kubernetes:core:Namespace  foo                        **updating failed**     [diff: ~metadata]; 1 error

Diagnostics:
  kubernetes:core:Namespace (foo):
    error: Plan apply failed: 1 error occurred:

    * the namespace of the object (default) does not match the namespace on the request (foo)

[hausdorff]

Thanks for the report! This error is basicaIIy saying that the REST request path (which contains the namespace, like /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}) and the REST request body have different namespaces.

Wild guess: IIRC client-go requires us to specify both a resource type and a namespace type, and when the namespace changes, the path and the request conflict. Hence the error.

@lblackstone, thoughts? I'll assign to you since you're doing the refactoring stuff -- lmk if you need to load balance it.

[lblackstone]

I was able to reproduce with an ever simpler program:

import * as k8s from "@pulumi/kubernetes";

new k8s.core.v1.Namespace("ns");

Updated to:

import * as k8s from "@pulumi/kubernetes";

new k8s.core.v1.Namespace("ns", {
    metadata: {
        labels: {
            hello: "world"
        }
    }
});

Updating (pulumi-k8s-test-dev):

     Type                          Name                                 Status                  Info
     pulumi:pulumi:Stack           pulumi-k8s-test-pulumi-k8s-test-dev
 ~   └─ kubernetes:core:Namespace  ns                                   **updating failed**     [diff: ~metadata]; 1 error

Diagnostics:
  kubernetes:core:Namespace (ns):
    error: Plan apply failed: 1 error occurred:

    * the namespace of the object (default) does not match the namespace on the request (ns-eui85cmq)

I'll work on tracking this down today.

[lblackstone]

Working on cutting a bugfix release now (0.20.1). It should be out by EOD today.

[farnoy]

I'm seeing this again on v1.1.0, is this a regression? My resource is:

# Source: cert-manager/charts/webhook/templates/apiservice.yaml
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.admission.certmanager.k8s.io
  labels:
    app: webhook
    chart: webhook-v0.8.1
    release: cert-manager
    heritage: Tiller
  annotations:
    certmanager.k8s.io/inject-ca-from: "cert-manager/cert-manager-webhook-webhook-tls"
spec:
  group: admission.certmanager.k8s.io
  groupPriorityMinimum: 1000
  versionPriority: 15
  service:
    name: cert-manager-webhook
    namespace: "cert-manager"
  version: v1beta1

imported through k8s.yaml.ConfigFile

Does not matter if I specify metadata.namespace explicitly or not, once the resource is created, it throws an error trying to update it:

  kubernetes:apiregistration:APIService (v1beta1.admission.certmanager.k8s.io):
    error: failed to determine if the following GVK is namespaced: apiregistration/v1beta1, Kind=APIService

I was trying to update from v0.17.21 where I do not have this problem.