[ec2/vpcpeeringconnection] Invalid memory address Provider AWS crashes with CrashLoopBackOff

[carlostaffe]

There is an existent VPCPeeringconnection resource imported with crossplane.io/external-name annotation.
It works fine for provider-aws V0.40.0. Crossplane v1.12.0. Condition is synced with a ReconcilieSucccess reason.
This is resource definition yaml file

apiVersion: ec2.aws.crossplane.io/v1alpha1
kind: VPCPeeringConnection
metadata:
  name: persistent-storage-to-rosa
  labels:
    name: persistent-storage-to-rosa
  annotations:
    crossplane.io/external-name: pcx-0057985aa978778fe
spec:
  deletionPolicy: Orphan
  forProvider:
    region: us-west-2
    acceptRequest: true
    vpcID : vpc-0099ad18597d94449
    peerVPCID: vpc-0b3ba69d55be9b962
    tags:
      - key: Name
        value: persistent-storage-to-rosa
  providerConfigRef:
    name: provider-aws

Once updated provider-aws to v0.41.0 version, ( same resource definition yaml file ), pod always shows this error and go to a CrashLoopBackOff status.

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x36f6fa8]

goroutine 4504 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:119 +0x1fa
panic({0x5c15580, 0xa03d6e0})
	runtime/panic.go:838 +0x207
github.com/crossplane-contrib/provider-aws/pkg/controller/ec2/vpcpeeringconnection.(*custom).postObserve(0xc000d7ec40, {0x71a59c0?, 0xc0065894a0?}, 0xc001e2ad80, 0xc00764be40, {0x1, 0x1, 0x0, 0x0, {0x0, ...}}, ...)
	github.com/crossplane-contrib/provider-aws/pkg/controller/ec2/vpcpeeringconnection/setup.go:92 +0x88
github.com/crossplane-contrib/provider-aws/pkg/controller/ec2/vpcpeeringconnection.(*external).Observe(0xc00057e850, {0x71a59c0, 0xc0065894a0}, {0x71d9110?, 0xc001e2ad80})
	github.com/crossplane-contrib/provider-aws/pkg/controller/ec2/vpcpeeringconnection/zz_controller.go:100 +0x477
github.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile(0xc000169e40, {0x71a59f8, 0xc001416ba0}, {{{0x0?, 0x0?}, {0xc000de6120?, 0x4?}}})
	github.com/crossplane/crossplane-runtime@v0.20.0-rc.0.0.20230320143010-c424c4aca5b0/pkg/reconciler/managed/reconciler.go:780 +0x289f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x71a59f8?, {0x71a59f8?, 0xc001416ba0?}, {{{0x0?, 0x56b1340?}, {0xc000de6120?, 0x413fb4?}}})
	sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:122 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00016f9a0, {0x71a5950, 0xc000e8b7c0}, {0x5eda2e0?, 0xc0001644c0?})
	sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:323 +0x386
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00016f9a0, {0x71a5950, 0xc000e8b7c0})
	sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:231 +0x325

We noticed issue is some conflict with external-name annotation. If we remove this annotation from yaml file, a new Custom Resource Instance (VPCPeeringConnection) is created. Only problem here is AWS refuses to create a new VPCPeeringConnection, because there is already an existent one. Condition is synced with a ReconcileError reason. Message is create failed: cannot create VPCPeeringConnection in AWS: InvalidParameterValue: A matching peering exists with different tags.

So, we need to import this external entity so that crossplane can manage it.

Info about versions:

• crossplane version V1.12.0
• provider-aws version v0.41.0
• Cloud provider AWS
• Kubernetes versionv 1.25.12+ba5cc25
• Kubernetes distribution OpenShift 4.12

[carlostaffe]

Updated to latest provider-aws vesion (v0.46.0) and issue was fixed.

[JaniszM]

Same here, thanks for the ticket @carlostaffe !