Investigate why claims becoming available and deleted taking longer than expected in E2E

[turkenh]

What happened?

We recently observed some flakiness with e2e tests, resulting in a false negative rate of 50%, and wanted to avoid that by bumping the relevant timeouts.

However, we believe it is worth further investigation, especially why we observe failures while waiting for claims to become available.

An example run: https://github.com/crossplane/crossplane/actions/runs/6084680018/job/16507125031

=== RUN   TestCrossplaneLifecycle/TestCrossplaneLifecycleUpgrade/ClaimIsAvailable
    feature.go:213: Waiting 2m0s for NopResource default/lifecycle-upgrade to become Available...
    feature.go:232: resources did not have desired conditions: Available: client rate limiter Wait returned an error: context deadline exceeded:
        
        - apiVersion: nop.example.org/v1alpha1
          kind: NopResource
          metadata:
            creationTimestamp: "2023-09-05T12:45:00Z"
            finalizers:
            - finalizer.apiextensions.crossplane.io
            generation: 3
            managedFields:
            - apiVersion: nop.example.org/v1alpha1
              fieldsType: FieldsV1
              fieldsV1:
                f:spec:
                  f:compositeDeletePolicy: {}
                  f:coolField: {}
              manager: crossplane-e2e-tests
              operation: Apply
              time: "2023-09-05T12:45:00Z"
            - apiVersion: nop.example.org/v1alpha1
              fieldsType: FieldsV1
              fieldsV1:
                f:metadata:
                  f:finalizers:
                    .: {}
                    v:"finalizer.apiextensions.crossplane.io": {}
                f:spec:
                  f:compositionRef:
                    .: {}
                    f:name: {}
                  f:compositionRevisionRef:
                    .: {}
                    f:name: {}
                  f:compositionUpdatePolicy: {}
                  f:resourceRef:
                    .: {}
                    f:apiVersion: {}
                    f:kind: {}
                    f:name: {}
              manager: Go-http-client
              operation: Update
              time: "2023-09-05T12:45:00Z"
            - apiVersion: nop.example.org/v1alpha1
              fieldsType: FieldsV1
              fieldsV1:
                f:status:
                  .: {}
                  f:conditions: {}
              manager: Go-http-client
              operation: Update
              subresource: status
              time: "2023-09-05T12:45:00Z"
            name: lifecycle-upgrade
            namespace: default
            resourceVersion: "2592"
            uid: 8b0b01ba-084f-4b7e-822a-a38b1d60ab4d
          spec:
            compositeDeletePolicy: Foreground
            compositionRef:
              name: xnopresources.nop.example.org
            compositionRevisionRef:
              name: xnopresources.nop.example.org-73f483f
            compositionUpdatePolicy: Automatic
            coolField: I'm cool!
            resourceRef:
              apiVersion: nop.example.org/v1alpha1
              kind: XNopResource
              name: lifecycle-upgrade-tczp2
          status:
            conditions:
            - lastTransitionTime: "2023-09-05T12:45:00Z"
              reason: ReconcileSuccess
              status: "True"
              type: Synced
            - lastTransitionTime: "2023-09-05T12:45:00Z"
              message: Composite resource claim is waiting for composite resource to become
                Ready
              reason: Waiting
              status: "False"
              type: Ready

How can we reproduce it?

Unfortunately it is not easily reproducable and happens sporatically with GitHub actions.

What environment did it happen in?

On master with CI.

[phisco]

Related PRs:

• E2E flakiness experiments. #4745 DRAFT, will fork dedicated PRs for each improvement
• feat: add readiness and liveness probes to crossplane #4748
• tests(e2e): always delete resources and check their status on creation #4749
• tests(e2e): add events to related objects and report also on deletion #4752
• tests(e2e): export kind logs on failure, enable audit logs and fail fast #4750

[phisco]

The e2es are much more stable now, we can close this