crossplane v1.14, aws-contrib provider pod showing Unauthorized error

[Swapnil-CSI]

What happened?

I am getting below error in aws contrib pod after upgrading aws contrib provider version in crossplane v1.14.0

W1116 12:52:04.413588       1 reflector.go:535] k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha1.ComputeEnvironment: Unauthorized
E1116 12:52:04.413625       1 reflector.go:147] k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha1.ComputeEnvironment: failed to list *v1alpha1.ComputeEnvironment: Unauthorized
W1116 12:52:04.481294       1 reflector.go:535] k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha1.Domain: Unauthorized
E1116 12:52:04.481329       1 reflector.go:147] k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: Failed to watch *v1alpha1.Domain: failed to list *v1alpha1.Domain: Unauthorized
W1116 12:52:04.895203       1 reflector.go:535] k8s.io/client-go@v0.28.3/tools/cache/reflector.go:229: failed to list *v1alpha1.FileSystem: Unauthorized

following is the DeploymentRuntimeConfig, Provider yaml files.

[swapnil@bharshankar aws-contrib]$ cat 1.deployment-runtime-config.yaml
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: aws-contrib
  namespace: crossplane-system
spec:
  deploymentTemplate:
    spec:
      selector: {}
      template:
        spec:
          serviceAccountName: aws-provider-contrib
          securityContext:
            runAsUser: 2000
            runAsGroup: 2000
            fsGroup: 2000
          containers:
            - name: package-runtime
              args:
                - --debug
                - --enable-management-policies
              resources:
                limits:
                  cpu: 400m
                  memory: 1Gi
                requests:
                  cpu: 400m
                  memory: 1Gi
              securityContext:
                allowPrivilegeEscalation: false
                capabilities:
                  drop:
                  - ALL
                readOnlyRootFilesystem: true
                runAsNonRoot: true
  serviceTemplate: {}
  serviceAccountTemplate:
    metadata:
      name: aws-provider-contrib

[swapnil@bharshankar aws-contrib]$



[swapnil@bharshankar aws-contrib]$ cat 2.provider.yaml
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: aws-provider-contrib
spec:
  runtimeConfigRef:
    apiVersion: pkg.crossplane.io/v1beta1
    kind: DeploymentRuntimeConfig
    name: aws-contrib
  package: xpkg.upbound.io/crossplane-contrib/provider-aws:v0.45.0
  #controllerConfigRef:
  #  name: aws-contrib
[swapnil@bharshankar aws-contrib]$

provider package and revision are in healthy state.

[swapnil@bharshankar aws-contrib]$ kubectl get provider.pkg
NAME                                INSTALLED   HEALTHY   PACKAGE                                                         AGE
aws-provider-contrib                True        True      xpkg.upbound.io/crossplane-contrib/provider-aws:v0.45.0         246d



[swapnil@bharshankar aws-contrib]$ kubectl get providerrevision
NAME                                             HEALTHY   REVISION   IMAGE                                                           STATE      DEP-FOUND   DEP-INSTALLED   AGE
aws-provider-contrib-29c442e72f5f                True      28         xpkg.upbound.io/crossplane-contrib/provider-aws:v0.44.2         Inactive                               3h18m
aws-provider-contrib-9c8a43141871                True      29         xpkg.upbound.io/crossplane-contrib/provider-aws:v0.45.0         Active                                 28m

cluster has 2 clusterrolebinding for contrib provider.

[swapnil@bharshankar ~]$ kubectl get clusterrolebinding crossplane:provider:aws-provider-contrib-9c8a43141871:system -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2023-11-16T12:33:29Z"
  name: crossplane:provider:aws-provider-contrib-9c8a43141871:system
  ownerReferences:
  - apiVersion: pkg.crossplane.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: ProviderRevision
    name: aws-provider-contrib-9c8a43141871
    uid: a4ce5d95-ac34-4cbc-9d65-827e7bcbe6b8
  resourceVersion: "1132178600"
  uid: ecabdd50-46c1-477a-98b8-b913428519e3
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: crossplane:provider:aws-provider-contrib-9c8a43141871:system
subjects:
- kind: ServiceAccount
  name: aws-provider-contrib
  namespace: crossplane-system
[swapnil@bharshankar ~]$


[swapnil@bharshankar ~]$ kubectl get clusterrolebinding crossplane:provider:aws-provider-contrib-29c442e72f5f:system -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2023-11-16T09:43:56Z"
  name: crossplane:provider:aws-provider-contrib-29c442e72f5f:system
  ownerReferences:
  - apiVersion: pkg.crossplane.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: ProviderRevision
    name: aws-provider-contrib-29c442e72f5f
    uid: 2a9d7749-eddf-48f4-b306-74cbcf541a9f
  resourceVersion: "1132178111"
  uid: 03701940-4977-4562-bc7c-7242a5151f33
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: crossplane:provider:aws-provider-contrib-29c442e72f5f:system
[swapnil@bharshankar ~]$

provider is recreating serviceaccount after few mins.

[swapnil@bharshankar ~]$ kubectl get pods -n crossplane-system aws-provider-contrib-9c8a43141871-857784df8-h62pc
NAME                                                READY   STATUS    RESTARTS   AGE
aws-provider-contrib-9c8a43141871-857784df8-h62pc   1/1     Running   0          61m
[swapnil@bharshankar ~]$ kubectl get sa -n crossplane-system aws-provider-contrib
NAME                   SECRETS   AGE
aws-provider-contrib   0         8m57s
[swapnil@bharshankar ~]$

Seen following entries in events.

0s          Warning   SyncPackage                    providerrevision/aws-provider-contrib-9c8a43141871                                         post establish runtime hook failed for package: cannot apply provider package service account: cannot patch object: serviceaccounts "aws-provider-contrib" not found
1s          Normal    SyncPackage                    providerrevision/aws-provider-contrib-29c442e72f5f                                         Successfully configured package revision
1s          Warning   InstallPackageRevision         provider/aws-provider-contrib                                                              current package revision is unhealthy
0s          Warning   InstallPackageRevision         provider/aws-provider-contrib                                                              current package revision is unhealthy
0s          Normal    SyncPackage                    providerrevision/aws-provider-contrib-9c8a43141871                                         Successfully configured package revision
0s          Normal    InstallPackageRevision         provider/aws-provider-contrib                                                              Successfully installed package revision

How can we reproduce it?

1. Install crossplane
2. Configure DeploymentRuntimeConfig, aws contrib Provider v0.44.2 & providerconfig
3. Deploy demo vpc creation resource.
4. Upgrade aws contrib Provider to version v0.45.0
5. Check newly deployed pod logs.

I tried same on 2nd cluster and received same error. Please find the attachment for logs from 2nd cluster.

crossplane-log.txt

What environment did it happen in?

Crossplane version: v1.14.0

[haarchri]

Can you Check your ServiceAccounts ? Possible that these ServiceAccounts recreated over time ? - what happen If you restart the provider ?

[haarchri]

looks like related to: #4071

cc @turkenh @stevendborrelli

[stevendborrelli]

I am seeing this issue too, but only on providers that have a ControllerConfig that defines a serviceAccountName. My understanding is this is affecting many users.

Workarounds:

1. Delete any inactive providerRevisions
2. I haven't seen this behavior with DeploymentRuntimeConfig, so try migrating off ControllerConfig. The crossplane-migrator can help.

[jbw976]

Thanks for the investigations so far everyone, as well as the workaround ideas! 🙇‍♂️

From what we've seen so far in this thread, this looks like something we need to fix and backport to a v1.14.2 patch release as soon as we can - i've triaged as such. Thanks for the patience!