CannotConnectToProvider while trying to create BucketVersioning using Composition

[BartoszZawadzki]

What happened?

I've created a Composition consisting of Bucket, BucketVersioning and BucketServerSideEncryptionConfiguration
Both Bucket and BucketServerSideEncryptionConfiguration are READY and SYNCED, while BucketVersioning is not.

When I describe BucketVersioning I get:

Status:
  At Provider:
  Conditions:
    Last Transition Time:  2024-02-16T16:19:36Z
    Message:               connect failed: cannot initialize the Terraform plugin SDK async external client: cannot get terraform setup: failed to retrieve aws credentials from aws config: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 0, RequestID: , request send failed, Post "https://sts..amazonaws.com/": dial tcp: lookup sts..amazonaws.com: no such host
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
Events:
  Type     Reason                   Age                   From                                                      Message
  ----     ------                   ----                  ----                                                      -------
  Warning  CannotConnectToProvider  38m (x4292 over 3d)   managed/s3.aws.upbound.io/v1beta1, kind=bucketversioning  cannot initialize the Terraform plugin SDK async external client: cannot get terraform setup: failed to retrieve aws credentials from aws config: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 0, RequestID: , request send failed, Post "https://sts..amazonaws.com/": dial tcp: lookup sts..amazonaws.com: no such host
  Warning  CannotConnectToProvider  2m27s (x41 over 37m)  managed/s3.aws.upbound.io/v1beta1, kind=bucketversioning  cannot initialize the Terraform plugin SDK async external client: cannot get terraform setup: failed to retrieve aws credentials from aws config: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, https response error StatusCode: 0, RequestID: , request send failed, Post "https://sts..amazonaws.com/": dial tcp: lookup sts..amazonaws.com: no such host

I've currently installed only two Providers:

kubectl get providers
NAME                      INSTALLED   HEALTHY   PACKAGE                                              AGE
default                   True        True      xpkg.upbound.io/upbound/provider-family-aws:v1.0.0   10d
upbound-provider-aws-s3   True        True      xpkg.upbound.io/upbound/provider-aws-s3:v1.0.0       17d

I'm using IRSA, but it's only configured with default (provider-family-aws) Provider (in accordance with documentation), so I do not uderstand why upbound-provider-aws-s3 is trying to fetch the credentials.

Moreover when I create Bucket and BucketVersioning directly - not using Composition everything works as expected.

How can we reproduce it?

Apply a Composition:

---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: s3
spec:
  writeConnectionSecretsToNamespace: crossplane
  patchSets:
    - name: s3
      patches:
        - type: FromCompositeFieldPath
          fromFieldPath: "spec.region"
          toFieldPath: "spec.forProvider.region"
  resources:
    - name: S3Bucket
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: Bucket
        metadata:
          name: ""
        spec:
          forProvider:
            forceDestroy: false
            region: ""
            tags: {}
          writeConnectionSecretToRef:
            name: ""
            namespace: crossplane
      patches:
        - type: PatchSet
          patchSetName: s3
        - type: FromCompositeFieldPath
          fromFieldPath: "spec.forceDestroy"
          toFieldPath: "spec.forProvider.forceDestroy"
        - type: FromCompositeFieldPath
          fromFieldPath: "spec.name"
          toFieldPath: "metadata.name"
        - type: FromCompositeFieldPath
          fromFieldPath: "spec.name"
          toFieldPath: "spec.writeConnectionSecretToRef.name"
        - type: FromCompositeFieldPath
          fromFieldPath: "spec.tags"
          toFieldPath: "spec.forProvider.tags"
    - name: S3BucketVersioning
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketVersioning
        spec:
          forProvider:
            bucketSelector:
              matchControllerRef: true
            region: ""
            versioningConfiguration:
              - status: Enabled
        patches:
          - type: PatchSet
            patchSetName: s3
    - name: S3BucketSSE
      base:
        apiVersion: s3.aws.upbound.io/v1beta1
        kind: BucketServerSideEncryptionConfiguration
        spec:
          forProvider:
            bucketSelector:
              matchControllerRef: true
            region: ""
            rule:
              - applyServerSideEncryptionByDefault:
                  - sseAlgorithm: AES256
      patches:
        - type: PatchSet
          patchSetName: s3
  compositeTypeRef:
    apiVersion: example.com/v1alpha1
    kind: XObjectStorage

+XRD

---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xobjectstorages.example.com
spec:
  defaultCompositeDeletePolicy: Foreground
  group: example.com
  names:
    categories:
      - crossplane
    kind: XObjectStorage
    plural: xobjectstorages
    shortNames:
      - xs
  claimNames:
    kind: ObjectStorage
    plural: objectstorages
  versions:
    - name: v1alpha1
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                forceDestroy:
                  type: boolean
                name:
                  type: string
                region:
                  type: string
                tags:
                  type: object
                  properties:
                    KubernetesCluster:
                      type: string
                    Environment:
                      type: string
              required:
                - forceDestroy
                - name
                - region
                - tags
      referenceable: true
      served: true

+Claim

---
apiVersion: example.com/v1alpha1
kind: ObjectStorage
metadata:
  name: example-bundles-poc-dev
  namespace: crossplane
spec:
  forceDestroy: true
  name: "example-bundles-poc-dev"
  region: "eu-west-1"
  tags:
    KubernetesCluster: "dev.example.com"
    Environment: "poc-dev"

What environment did it happen in?

Crossplane version: v1.14.5

• Cloud provider or hardware configuration AWS
• Kubernetes version (use kubectl version) v1.23.17
• Kubernetes distribution (e.g. Tectonic, GKE, OpenShift): k8s cluster built and managed by kops

[MisterMX]

This is the wrong provider. It does not use Terraform but the AWS SDK underneath. The API Group of this provider is aws.crossplane.io. Yours is aws.upbound.io.

You want to look at https://github.com/upbound/provider-aws.