Cannot update apiVersion of apiextensions.CustomResource

[j-maxi]

I couldn't update an apiVersion of apiextensions.CustomResource for a Kubernetes API that supports in-place updates.
Specifically, I couldn't update apiVersion: cloud.google.com/v1beta1 to cloud.google.com/v1 for kind BackendConfig, and I am wondering this could be a general issue for CustomResource.

Steps to reproduce

1. Create a Custom Resource.

new kubernetes.apiextensions.CustomResource("test", {
  apiVersion: "cloud.google.com/v1beta1",
  kind: "BackendConfig",
  metadata: {
    name: "backendconfig",
    namespace: "test-namespace",
  },
  spec: {
    timeoutSec: 300,
  },
}, {})

2. Update apiVersion with specifying type alias.

new kubernetes.apiextensions.CustomResource("test", {
  apiVersion: "cloud.google.com/v1",
  kind: "BackendConfig",
  metadata: {
    name: "backendconfig",
    namespace: "test-namespace",
  },
  spec: {
    timeoutSec: 300,
  },
}, {
  aliases: [{ type: "kubernetes:cloud.google.com/v1beta1:BackendConfig" }],
})

3. The error I got on pulumi up.

Updating (mysatck):
     Type                                             Name        Status                  Info
     pulumi:pulumi:Stack                              local-hoge  **failed**              1 error; 3 messages
 ~   └─ kubernetes:cloud.google.com/v1:BackendConfig  test        **updating failed**     [diff: ~apiVersion]; 1 error

  kubernetes:cloud.google.com/v1:BackendConfig (test):
    error: 1 error occurred:
        * the Kubernetes API server reported that "testnamespace/backendconfig" failed to fully initialize or become live: BackendConfig.cloud.google.com "backendconfig" is invalid: apiVersion: Invalid value: "cloud.google.com/v1": must be cloud.google.com/v1beta1

Expected: Update succeeded.
Actual: Update failed.

Additional analysis

I tested with v2.9.1 and v3.6.0, and both failed.

I have checked Kubernetes API log and figured out Pulumi called PATCH API for cloud.google.com/v1beta1.
Looks Kubernetes Provider would use a k8s client created from previous input to patch an object: in my case, the Kubernetes Provider used v1beta1 client as the previous input was v1beta1.
https://github.com/pulumi/pulumi-kubernetes/blob/v2.9.1/provider/pkg/await/await.go#L376-L379

I could successfully run the same procedure with kubectl (apply v1beta1 manifest and v1 manifest). For this case, the Kubernetes API log showed that kubectl called PATCH API for cloud.google.com/v1.

I am wondering if we should use a client from the new Inputs to call an API for the new apiVersion.

[lblackstone]

Thanks for the detailed investigation here. It seems like we may need to special case the behavior for CustomResources because I suspect the api server isn't auto-converting between versions like it does for built in resources. Another possibility would be that we could use the target client version for all resources rather than special casing.

As a workaround in the meantime, I think you could run the update using kubectl and then run pulumi refresh to reconcile the change.

[j-maxi]

@lblackstone Thanks for taking care of this.

As a workaround in the meantime, I think you could run the update using kubectl and then run pulumi refresh to reconcile the change.

I am afraid this wouldn't work. The BackendConfig was actually converted to v1 at the API server. And, even we run pulumi refresh, the state of the previous input remained v1beta1 and we would still use v1beta1 client.

Another possibility would be that we could use the target client version for all resources rather than special casing.

I don't see any drawback to doing this.

[j-maxi]

Another possibility would be that we could use the target client version for all resources rather than special casing.

Many Kubernetes APIs support in-place updates and they may introduce some changes to the spec.
https://kubernetes.io/docs/reference/using-api/deprecation-guide/
We already auto-alias some Kubernetes APIs behind the scene.
#798

For a migration scenario, users would change apiVersion and follow the new API spec. We need to use a new API client here to comply with the new API spec. So, I rather see it is natural to use a new client without special casing.

[lblackstone]

Sorry for the delay here @j-maxi. I confirmed that #1758 fixes this issue, so this should be resolved on the next release.