[Web Bug] - GCP Quickstart - GCP CRDs not installed and only one provider installed

[mattwelke]

I'm having trouble completing the GCP quickstart tutorial. I'm able to make a kind cluster and use Helm to deploy Crossplane to it. I'm also able to run the following command to create a provider in the cluster.

cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-gcp-storage
spec:
  package: xpkg.crossplane.io/crossplane-contrib/provider-gcp-storage:v1
EOF

But, when I run kubectl get providers, I only see one provider in that list. The tutorial shows two at this point in the example output and mentions that the first provider is supposed to install the second.

$ k get providers
NAME                   INSTALLED   HEALTHY   PACKAGE                                                         AGE
provider-gcp-storage   False                 xpkg.crossplane.io/crossplane-contrib/provider-gcp-storage:v1   16m

I continued with the tutorial and ran kubectl get crds, expecting to see GCP related CRDs. Instead, I see only Crossplane related CRDs.

$ kubectl get crds
NAME                                                       CREATED AT
compositeresourcedefinitions.apiextensions.crossplane.io   2025-03-05T23:42:05Z
compositionrevisions.apiextensions.crossplane.io           2025-03-05T23:42:05Z
compositions.apiextensions.crossplane.io                   2025-03-05T23:42:05Z
configurationrevisions.pkg.crossplane.io                   2025-03-05T23:42:05Z
configurations.pkg.crossplane.io                           2025-03-05T23:42:05Z
controllerconfigs.pkg.crossplane.io                        2025-03-05T23:42:05Z
deploymentruntimeconfigs.pkg.crossplane.io                 2025-03-05T23:42:06Z
environmentconfigs.apiextensions.crossplane.io             2025-03-05T23:42:05Z
functionrevisions.pkg.crossplane.io                        2025-03-05T23:42:06Z
functions.pkg.crossplane.io                                2025-03-05T23:42:06Z
imageconfigs.pkg.crossplane.io                             2025-03-05T23:42:06Z
locks.pkg.crossplane.io                                    2025-03-05T23:42:06Z
providerrevisions.pkg.crossplane.io                        2025-03-05T23:42:06Z
providers.pkg.crossplane.io                                2025-03-05T23:42:06Z
storeconfigs.secrets.crossplane.io                         2025-03-05T23:42:06Z
usages.apiextensions.crossplane.io                         2025-03-05T23:42:05Z

I proceeded through the step where you create a Kubernetes secret with the contents of the service account key file, and had no problems with that.

But then, when I tried creating the ProviderConfig, it said there was no CRD matched (makes sense).

cat <<EOF | kubectl apply -f -
apiVersion: gcp.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  projectID: crossplane-test-452823 
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: gcp-secret
      key: creds
EOF

So I wasn't able to progress further.

I checked the pod logs of the two pods in the crossplane-system namespace and didn't see anything alarming.

$ k logs crossplane-654d5644f4-4x9cg -n crossplane-system
Defaulted container "crossplane" out of: crossplane, crossplane-init (init)
{"level":"info","ts":"2025-03-05T23:42:10Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaCompositionWebhookSchemaValidation"}
{"level":"info","ts":"2025-03-05T23:42:10Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaUsages"}
{"level":"info","ts":"2025-03-05T23:42:10Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaDeploymentRuntimeConfigs"}
{"level":"info","ts":"2025-03-05T23:42:10Z","logger":"crossplane","msg":"Beta feature enabled","flag":"EnableBetaClaimSSA"}
Warning: ControllerConfig.pkg.crossplane.io/v1alpha1 is deprecated. Use DeploymentRuntimeConfig from pkg.crossplane.io/v1beta1 instead.

$ k logs crossplane-rbac-manager-59d8fcb968-rcqhc -n crossplane-system
Defaulted container "crossplane" out of: crossplane, crossplane-init (init)

Is there a step I missed or some prerequisite I didn't fulfill? Thanks!

More info:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.2 LTS
Release:        24.04
Codename:       noble

$ kubectl version
Client Version: v1.31.4
Kustomize Version: v5.4.2
Server Version: v1.32.2

$ kind version
kind v0.27.0 go1.23.6 linux/amd64

$ helm version
version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"}

URL: https://docs.crossplane.io/latest/getting-started/provider-gcp/

[jbw976]

good catch @mattwelke, thanks for sharing this! Looks like the GCP quickstart is instructing you to install a v1 with no further version information. I don't think that version tag exists 😇. Compare that to the AWS and Azure quickstarts which use a full version number, like v1.20.1.

We should get that updated to point to the latest version of xpkg.crossplane.io/crossplane-contrib/provider-gcp-storage, which appears to be v1.11.4 according to https://github.com/crossplane-contrib/provider-upjet-gcp/pkgs/container/provider-gcp-storage/versions.

If you're up for making a PR with that change, we'll be happy to review it 🤩

[mattwelke]

Thanks for the response. That makes sense. I think we can't avoid hard-coding a particular version in the quickstart for now, but I could at least get it updated. I tried with the latest right now, which is 1.20.0. I get further now but I still get an error. The bucket applied reaches "synced: false" and there's an error reported in a condition.

status:
  atProvider: {}
  conditions:
  - lastTransitionTime: "2025-03-06T21:48:50Z"
    message: 'connect failed: cannot initialize the Terraform plugin SDK async external
      client: cannot get terraform setup: cannot get referenced ProviderConfig: Timeout:
      failed waiting for *v1beta1.ProviderConfig Informer to sync'
    reason: ReconcileError
    status: "False"
    type: Synced

I'm wondering if the provider the quickstart instructs me to create is somehow incompatible. If I search the web for this error, I find stuff like crossplane-contrib/provider-aws#2006 which talks about API versions being incompatible. But pkg.crossplane.io/v1 seems pretty authoritative and up to date. Taking a quick glance at the AWS quickstart, it seems to work this way too.

Do you know what issue I'm running into now?

I'm happy to open a PR once I've successfully gotten through the quickstart and have a good idea of how Provider, ProviderConfig, and Bucket are meant to be used together.

[mattwelke]

Tried the AWS quickstart too and I'm getting the same error. The error message looks identical.

status:
  atProvider: {}
  conditions:
  - lastTransitionTime: "2025-03-07T00:23:11Z"
    message: 'connect failed: cannot initialize the Terraform plugin SDK async external
      client: cannot get terraform setup: cannot get referenced ProviderConfig: "default":
      Timeout: failed waiting for *v1beta1.ProviderConfig Informer to sync'
    reason: ReconcileError
    status: "False"
    type: Synced

For AWS, I created a secret with a key pair that has AmazonS3FullAccess.

[jbw976]

Timeout: failed waiting for *v1beta1.ProviderConfig Informer to sync

hmm, this type of error in general usually indicates a resource constrained control plane. The controllers in the crossplane provider pods are having a hard time connecting to the API server and getting data back to populate their caches.

• Can you mention more about your K8s cluster? is it a managed service, is it local, how much resources (mem/cpu) does it have?
• can you do a kubectl get pkg and kubectl get providerrevision?
• API server logs could be helpful also to show other slowdowns

[mattwelke]

I was using kind. But I was using WSL 2 Ubuntu, so I decided to repeat this test using kind on regular Ubuntu. Same problem. I used the GCP quickstart just now.

I'll try to provide more info about this setup.

I create my kind cluster using the latest version of kind installed, so that it deploys a k8s 1.32.2 cluster.

There is a single node.

$ k get node
NAME                            STATUS   ROLES           AGE   VERSION
crossplane-test-control-plane   Ready    control-plane   31m   v1.32.2

And it doesn't appear to be resource constrained right now.

$ k describe node crossplane-test-control-plane
Name:               crossplane-test-control-plane
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=crossplane-test-control-plane
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 07 Mar 2025 11:54:52 -0500
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  crossplane-test-control-plane
  AcquireTime:     <unset>
  RenewTime:       Fri, 07 Mar 2025 12:23:34 -0500
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 07 Mar 2025 12:23:01 -0500   Fri, 07 Mar 2025 11:54:50 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 07 Mar 2025 12:23:01 -0500   Fri, 07 Mar 2025 11:54:50 -0500   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 07 Mar 2025 12:23:01 -0500   Fri, 07 Mar 2025 11:54:50 -0500   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 07 Mar 2025 12:23:01 -0500   Fri, 07 Mar 2025 11:55:16 -0500   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.18.0.2
  Hostname:    crossplane-test-control-plane
Capacity:
  cpu:                20
  ephemeral-storage:  487968864Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16035540Ki
  pods:               110
Allocatable:
  cpu:                20
  ephemeral-storage:  487968864Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16035540Ki
  pods:               110
System Info:
  Machine ID:                 ddb058e6152b4108992a968481212580
  System UUID:                2c9b3e42-2bc0-4139-b2f9-424db03fbd59
  Boot ID:                    7ee53cae-a730-4274-b4f9-71fab6634d81
  Kernel Version:             6.11.0-19-generic
  OS Image:                   Debian GNU/Linux 12 (bookworm)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://2.0.2
  Kubelet Version:            v1.32.2
  Kube-Proxy Version:         v1.32.2
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
ProviderID:                   kind://docker/crossplane-test/crossplane-test-control-plane
Non-terminated Pods:          (13 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  crossplane-system           crossplane-654d5644f4-fwnw9                                        100m (0%)     500m (2%)   256Mi (1%)       1Gi (6%)       23m
  crossplane-system           crossplane-contrib-provider-family-gcp-12b9aeb46d01-5d6d87xxnn5    0 (0%)        0 (0%)      0 (0%)           0 (0%)         22m
  crossplane-system           crossplane-rbac-manager-59d8fcb968-r2wrp                           100m (0%)     100m (0%)   256Mi (1%)       512Mi (3%)     23m
  crossplane-system           provider-gcp-storage-2b0d7d770d05-f58fc7595-clhsk                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         22m
  kube-system                 coredns-668d6bf9bc-lsx2z                                           100m (0%)     0 (0%)      70Mi (0%)        170Mi (1%)     28m
  kube-system                 coredns-668d6bf9bc-wtcpr                                           100m (0%)     0 (0%)      70Mi (0%)        170Mi (1%)     28m
  kube-system                 etcd-crossplane-test-control-plane                                 100m (0%)     0 (0%)      100Mi (0%)       0 (0%)         28m
  kube-system                 kindnet-gspfm                                                      100m (0%)     100m (0%)   50Mi (0%)        50Mi (0%)      28m
  kube-system                 kube-apiserver-crossplane-test-control-plane                       250m (1%)     0 (0%)      0 (0%)           0 (0%)         28m
  kube-system                 kube-controller-manager-crossplane-test-control-plane              200m (1%)     0 (0%)      0 (0%)           0 (0%)         28m
  kube-system                 kube-proxy-4cdcr                                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         28m
  kube-system                 kube-scheduler-crossplane-test-control-plane                       100m (0%)     0 (0%)      0 (0%)           0 (0%)         28m
  local-path-storage          local-path-provisioner-7dc846544d-jfkbm                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         28m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                1150m (5%)  700m (3%)
  memory             802Mi (5%)  1926Mi (12%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type    Reason                   Age                From             Message
  ----    ------                   ----               ----             -------
  Normal  Starting                 28m                kube-proxy       
  Normal  Starting                 28m                kubelet          Starting kubelet.
  Normal  NodeHasSufficientMemory  28m (x9 over 28m)  kubelet          Node crossplane-test-control-plane status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    28m (x7 over 28m)  kubelet          Node crossplane-test-control-plane status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     28m (x7 over 28m)  kubelet          Node crossplane-test-control-plane status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  28m                kubelet          Updated Node Allocatable limit across pods
  Normal  Starting                 28m                kubelet          Starting kubelet.
  Normal  NodeAllocatableEnforced  28m                kubelet          Updated Node Allocatable limit across pods
  Normal  NodeHasSufficientMemory  28m                kubelet          Node crossplane-test-control-plane status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    28m                kubelet          Node crossplane-test-control-plane status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     28m                kubelet          Node crossplane-test-control-plane status is now: NodeHasSufficientPID
  Normal  RegisteredNode           28m                node-controller  Node crossplane-test-control-plane event: Registered Node crossplane-test-control-plane in Controller
  Normal  NodeReady                28m                kubelet          Node crossplane-test-control-plane status is now: NodeReady

The other resources you mentioned...

$ k get pkg
NAME                                                                INSTALLED   HEALTHY   PACKAGE                                                              AGE
provider.pkg.crossplane.io/crossplane-contrib-provider-family-gcp   True        True      ghcr.io/crossplane-contrib/provider-family-gcp:v1.12.0               23m
provider.pkg.crossplane.io/provider-gcp-storage                     True        True      xpkg.crossplane.io/crossplane-contrib/provider-gcp-storage:v1.12.0   23m

NAME                                                  HEALTHY   REVISION   IMAGE                                                                STATE    DEP-FOUND   DEP-INSTALLED   AGE
crossplane-contrib-provider-family-gcp-12b9aeb46d01   True      1          ghcr.io/crossplane-contrib/provider-family-gcp:v1.12.0               Active                               24m
provider-gcp-storage-2b0d7d770d05                     True      1          xpkg.crossplane.io/crossplane-contrib/provider-gcp-storage:v1.12.0   Active   1           1               24m

I tried getting the API server logs by getting logs from the Docker container, but didn't see anything.

docker logs crossplane-test-control-plane --follow --since=32m | grep control-plane

[mattwelke]

We think we've found the root cause now. After following the quickstart, the S3 provider pod logs an RBAC error. Manually adding a ClusterRoleBinding for role cluster-admin fixes it, but a more specific RBAC binding should be added instead. The quickstarts should include this step.

The error:

k8s.io/client-go@v0.30.0/tools/cache/reflector.go:232: failed to list *v1beta1.ProviderConfig: providerconfigs.aws.upbound.io is forbidden: User "system:serviceaccount:crossplane-system:provider-aws-s3-f388c2201fc3" cannot list resource "providerconfigs" in API group "aws.upbound.io" at the cluster scope

[jbw976]

aha, great debugging @mattwelke!! really appreciate your continued attention to this matter, that is 💯 the root cause here and it's a regression from the new publishing we are doing of all the Crossplane community extensions to xpkg.crossplane.io/ghcr.io. There's clearly a few kinks here to iron out, so I'll try to rally folks to get this sorted out ASAP. Thanks for your patience and investigation here! 🙇‍♂

[mattwelke]

No problem! I'll submit a PR to do the quick GCP version fix, but I'll leave the rest of the tutorial fixing to you folks since you have much more context than me.

Actually, no I won't, because it's already fixed in master.

spec:
  package: xpkg.crossplane.io/crossplane-contrib/provider-gcp-storage:v1.11.4

Looks like we're all set!

[mattwelke]

Closing in favor of #888 to track RBAC issue.