[EKS] Adopting existing managed addon causes nil pointer panic

[andrewjeffree]

Describe the bug
I have an existing managed vpc-cni addon setup on the cluster with an IAM role and some config.
logs are pointing me at https://github.com/aws-controllers-k8s/eks-controller/blob/main/pkg/resource/addon/sdk.go#L452

I am trying to change the tags to be different to what's on the addon currently but even if I change the tags to be the same as what's on the addon currently I get the same error... I think what's happening is it's expecting the desired object to have the Addon ARN even when it hasn't stored it on the object yet and as such it's unset resulting in the error.

logs:

ack-eks-controller-568b85568b-xf685 controller {"level":"info","ts":"2025-09-04T10:07:06.688Z","logger":"ackrt","msg":"desired resource state has changed","kind":"Addon","namespace":"kube-system","name":"vpc-cni","account":"1234567890","role":"","region":"us-east-1","is_adopted":false,"generation":4,"diff":[{"Path":{"Parts":["Spec","ConfigurationValues"]},"A":"{\"env\":{\"AWS_VPC_K8S_CNI_EXTERNALSNAT\":\"true\",\"AWS_VPC_K8S_CNI_LOG_FILE\":\"stdout\",\"AWS_VPC_K8S_PLUGIN_LOG_FILE\":\"stderr\",\"AWS_VPC_K8S_PLUGIN_LOG_LEVEL\":\"DEBUG\",\"ENABLE_PREFIX_DELEGATION\":\"true\",\"ENABLE_POD_ENI\":\"true\",\"POD_SECURITY_GROUP_ENFORCING_MODE\":\"standard\",\"ENABLE_BANDWIDTH_PLUGIN\":\"true\"}}","B":"{\"env\":{\"AWS_VPC_K8S_CNI_EXTERNALSNAT\":\"true\",\"AWS_VPC_K8S_CNI_LOG_FILE\":\"stdout\",\"AWS_VPC_K8S_PLUGIN_LOG_FILE\":\"stderr\",\"AWS_VPC_K8S_PLUGIN_LOG_LEVEL\":\"DEBUG\",\"ENABLE_BANDWIDTH_PLUGIN\":\"true\",\"ENABLE_POD_ENI\":\"true\",\"ENABLE_PREFIX_DELEGATION\":\"true\"}}"},{"Path":{"Parts":["Spec","Tags"]},"A":{"environment":"development","services.k8s.aws/controller-version":"eks-1.9.0","services.k8s.aws/namespace":"kube-system","team":"cloud-infrastructure-engineering","tribe":"foundational-engineering"},"B":{"environment":"development","team":"cloud-infrastructure-engineering","tribe":"foundational-engineering"}}]}
ack-eks-controller-568b85568b-xf685 controller {"level":"error","ts":"2025-09-04T10:07:06.688Z","msg":"Observed a panic","controller":"addon","controllerGroup":"eks.services.k8s.aws","controllerKind":"Addon","Addon":{"name":"vpc-cni","namespace":"kube-system"},"namespace":"kube-system","name":"vpc-cni","reconcileID":"d5a04f15-775a-44b1-b4d4-8f9958531c12","panic":"runtime error: invalid memory address or nil pointer dereference","panicGoValue":"\"invalid memory address or nil pointer dereference\"","stacktrace":"goroutine 301 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x2907878, 0xc0011eefc0}, {0x2131c40, 0x3c60820})\n\t/go/pkg/mod/k8s.io/apimachinery@v0.32.1/pkg/util/runtime/runtime.go:107 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:108 +0x112\npanic({0x2131c40?, 0x3c60820?})\n\t/usr/local/go/src/runtime/panic.go:792 +0x132\ngithub.com/aws-controllers-k8s/eks-controller/pkg/resource/addon.(*resourceManager).sdkUpdate(0xc0012f2908, {0x2907878, 0xc000eaccc0}, 0xc000780168, 0xc000614348, 0xc000ece5d0)\n\t/github.com/aws-controllers-k8s/eks-controller/pkg/resource/addon/sdk.go:452 +0xd4f\ngithub.com/aws-controllers-k8s/eks-controller/pkg/resource/addon.(*resourceManager).Update(0xc0012f2908, {0x2907878?, 0xc000eaccc0?}, {0x2915fd0?, 0xc000780168?}, {0x2915fd0?, 0xc000614348}, 0x0?)\n\t/github.com/aws-controllers-k8s/eks-controller/pkg/resource/addon/manager.go:157 +0x77\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).updateResource(0xc000c62c00, {0x2907878, 0xc000eaccc0}, {0x2915f60, 0xc0012f2908}, {0x2915fd0, 0xc000780168}, {0x2915fd0, 0xc000614348})\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:765 +0x38c\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync(0xc000c62c00, {0x2907878, 0xc000eaccc0}, {0x2915f60, 0xc0012f2908}, {0x2915fd0, 0xc000780128})\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:498 +0xf53\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile(0xc000c62c00, {0x2907878, 0xc000eaccc0}, {0x2915f60, 0xc0012f2908}, {0x2915fd0, 0xc000780128})\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:377 +0x265\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile(0xc000c62c00, {0x2907878, 0xc0011eefc0}, {{{0xc000bcfa40?, 0x24b32d4?}, {0xc000bcfa30?, 0x100?}}})\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:284 +0xa3d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc0011eef30?, {0x2907878?, 0xc0011eefc0?}, {{{0xc000bcfa40?, 0x0?}, {0xc000bcfa30?, 0x0?}}})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x292e1c0, {0x29078b0, 0xc000558eb0}, {{{0xc000bcfa40, 0xb}, {0xc000bcfa30, 0x7}}}, 0x0)\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334 +0x3ad\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x292e1c0, {0x29078b0, 0xc000558eb0})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294 +0x21b\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 90\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:251 +0x6b5\n","stacktrace":"runtime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:925\ngithub.com/aws-controllers-k8s/eks-controller/pkg/resource/addon.(*resourceManager).sdkUpdate\n\t/github.com/aws-controllers-k8s/eks-controller/pkg/resource/addon/sdk.go:452\ngithub.com/aws-controllers-k8s/eks-controller/pkg/resource/addon.(*resourceManager).Update\n\t/github.com/aws-controllers-k8s/eks-controller/pkg/resource/addon/manager.go:157\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).updateResource\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:765\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:498\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:377\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:284\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}
ack-eks-controller-568b85568b-xf685 controller {"level":"error","ts":"2025-09-04T10:07:06.688Z","msg":"Reconciler error","controller":"addon","controllerGroup":"eks.services.k8s.aws","controllerKind":"Addon","Addon":{"name":"vpc-cni","namespace":"kube-system"},"namespace":"kube-system","name":"vpc-cni","reconcileID":"d5a04f15-775a-44b1-b4d4-8f9958531c12","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}

object:

apiVersion: eks.services.k8s.aws/v1alpha1
kind: Addon
metadata:
  annotations:
    services.k8s.aws/adoption-policy: adopt-or-create
  labels:
    app: vpc-cni
    app.kubernetes.io/instance: vpc-cni
    aws_region: us-east-1
    group: platform
    team: cloud-infrastructure-engineering
    tribe: foundational-engineering
  name: vpc-cni
  namespace: kube-system
  addonVersion: v1.19.5-eksbuild.3
  clusterName: eks01-us-east-1-development
  configurationValues: '{"env":{"AWS_VPC_K8S_CNI_EXTERNALSNAT":"true","AWS_VPC_K8S_CNI_LOG_FILE":"stdout","AWS_VPC_K8S_PLUGIN_LOG_FILE":"stderr","AWS_VPC_K8S_PLUGIN_LOG_LEVEL":"DEBUG","ENABLE_PREFIX_DELEGATION":"true","ENABLE_POD_ENI":"true","POD_SECURITY_GROUP_ENFORCING_MODE":"standard","ENABLE_BANDWIDTH_PLUGIN":"true"}}'
  name: vpc-cni
  resolveConflicts: OVERWRITE
  serviceAccountRoleARN: arn:aws:iam::1234567890:role/eks01-us-dev-aws-node
  tags:
    environment: development
    team: cloud-infrastructure-engineering
    tribe: foundational-engineering

Steps to reproduce
Create Managed Addon with tags and config on an EKS cluster and then attempt to adopt it.

Expected outcome
Addon is adopted successfully and managed going forward.

Environment

• Controller Version: v1.9.0
• Kubernetes version: 1.32
• Using EKS (yes/no), if so version? 1.32
• AWS service targeted (S3, RDS, etc.): EKS

[github-actions]

Hello @andrewjeffree 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[lonelyelk]

We had the same issue on a different resource. Tags seem to be the culprit during the adoption:

Object:

apiVersion: lambda.services.k8s.aws/v1alpha1
kind: Function
metadata:
  name: wolfi-package-index
  annotations:
    services.k8s.aws/region: ${cluster_region}
    services.k8s.aws/adoption-policy: adopt-or-create
    services.k8s.aws/deletion-policy: retain
spec:
  name: wolfi-package-index-${cluster_env}
  architectures:
    - arm64
  packageType: Image
  code:
    imageURI: XXX.dkr.ecr.us-west-2.amazonaws.com/some/path:1.0.0
  role: arn:aws:iam::${cluster_account_id}:role/service-role/some-role
  description: Does wolfi imdex search.
  timeout: 240
  memorySize: 128
  tags:
    terraform:source: "main.tf"
    terraform:workspace: ${cluster_env}

Stacktrace:

goroutine 393 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2992898, 0xc00129a2a0}, {0x21a5760, 0x3d0fb90})
	/go/pkg/mod/k8s.io/apimachinery@v0.32.1/pkg/util/runtime/runtime.go:107 +0xbc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:108 +0x112
panic({0x21a5760?, 0x3d0fb90?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
github.com/aws-controllers-k8s/lambda-controller/pkg/resource/function.(*resourceManager).updateFunctionTags(0xc0010a5688, {0x2992898, 0xc00129a8a0}, 0xc00079c0d8, 0xc0004cc150)
	/github.com/aws-controllers-k8s/lambda-controller/pkg/resource/function/hooks.go:395 +0x6c8
github.com/aws-controllers-k8s/lambda-controller/pkg/resource/function.(*resourceManager).customUpdateFunction(0xc0010a5688, {0x2992898, 0xc00129a8a0}, 0xc0004cc150, 0xc00079c0d8, 0xc000f90360)
	/github.com/aws-controllers-k8s/lambda-controller/pkg/resource/function/hooks.go:79 +0x305
github.com/aws-controllers-k8s/lambda-controller/pkg/resource/function.(*resourceManager).sdkUpdate(...)
	/github.com/aws-controllers-k8s/lambda-controller/pkg/resource/function/sdk.go:939
github.com/aws-controllers-k8s/lambda-controller/pkg/resource/function.(*resourceManager).Update(0xc0010a5688, {0x2992898?, 0xc00129a8a0?}, {0x29a16f0?, 0xc0004cc150?}, {0x29a16f0?, 0xc00079c0d8}, 0x100?)
	/github.com/aws-controllers-k8s/lambda-controller/pkg/resource/function/manager.go:157 +0x78
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).updateResource(0xc000ebc800, {0x2992898, 0xc00129a8a0}, {0x29a1680, 0xc0010a5688}, {0x29a16f0, 0xc0004cc150}, {0x29a16f0, 0xc00079c0d8})
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:765 +0x38c
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync(0xc000ebc800, {0x2992898, 0xc00129a8a0}, {0x29a1680, 0xc0010a5688}, {0x29a16f0, 0xc0004cc110})
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:498 +0xf33
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile(0xc000ebc800, {0x2992898, 0xc00129a8a0}, {0x29a1680, 0xc0010a5688}, {0x29a16f0, 0xc0004cc110})
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:377 +0x265
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile(0xc000ebc800, {0x2992898, 0xc00129a2a0}, {{{0xc0004bf680?, 0x13?}, {0xc0004bf668?, 0x13?}}})
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.51.0/pkg/runtime/reconciler.go:284 +0xa3d
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc00129a210?, {0x2992898?, 0xc00129a2a0?}, {{{0xc0004bf680?, 0x0?}, {0xc0004bf668?, 0x0?}}})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119 +0xbf
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x29ba2e0, {0x29928d0, 0xc0005f89b0}, {{{0xc0004bf680, 0x13}, {0xc0004bf668, 0x13}}}, 0x0)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334 +0x3ad
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x29ba2e0, {0x29928d0, 0xc0005f89b0})
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294 +0x21b
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 218
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:251 +0x6b5

The final line to panic is this one: https://github.com/aws-controllers-k8s/lambda-controller/blob/v1.9.0/pkg/resource/function/hooks.go#L395

Resource: (*string)(desired.ko.Status.ACKResourceMetadata.ARN),

We think it's because it has no Status. Which most probably leads us here: https://github.com/aws-controllers-k8s/runtime/blob/main/pkg/runtime/reconciler.go#L903-L906

	// The call to Patch() above ends up setting the latest variable's Status
	// to the value of the desired variable's Status. We do not want this
	// behaviour; instead, we want to keep latest's original Status value.
	latestCleaned.SetStatus(lorig)

So we either should account for adopted resources during the metadata patch and try to make sure Status is not nil. Or fix all the update functions to account for nil status.

Workaround

The workaround we found was to add ACK-specific tags on the AWS side so that there was no diff in tags at the moment of adoption.

[andrewjeffree]

interesting I didn't think of adding the ack specific tags to it that's kinda horrible though at the same time.

[lonelyelk]

interesting I didn't think of adding the ack specific tags to it that's kinda horrible though at the same time.

Oh, yes. I couldn't remove the tags from the resource, so I decided to try whatever it takes to make an empty diff in tags. So I looked at the diff and added the missing tags on the other side (terraform). Namely: services.k8s.aws/controller-version and services.k8s.aws/namespace. It's not a working setup going forward, but it does narrow down the source of the issue quite a bit.

[knottnt]

@andrewjeffree A fix for the panic during adoption of an addon has been shipped in v1.9.1 of the eks-controller.

[andrewjeffree]

Thanks I'll check it out next week.