[S3] invalid memory address or nil pointer dereference as of s3-controller version `1.0.27`

[calebhansard]

Describe the bug

As of version 1.0.27 of the s3 controller, I'm receiving errors in the logs when attempting to reconcile buckets. Version 1.0.26 and older are not affected.

Steps to reproduce

Upgrade s3-controller to 1.0.27 or higher.

Expected outcome

The s3-controller should continue to reconcile buckets after upgrading to 1.0.27 or higher.

Environment

• Kubernetes version: v1.31.7-eks-bcf3d70
• AWS service targeted: S3
• AWS Partition: GovCloud

Bucket manifest and logs

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  name: my-bucket
  namespace: ack-system
spec:
  encryption:
    rules:
    - applyServerSideEncryptionByDefault:
        sseAlgorithm: AES256
  name: my-bucket
  publicAccessBlock:
    blockPublicACLs: true
    blockPublicPolicy: true
    ignorePublicACLs: true
    restrictPublicBuckets: true
  tagging:
    tagSet:
    - key: ApplicationName
      value: my-bucket

{"level":"error","ts":"2025-06-03T14:08:31.885Z","msg":"Observed a panic","controller":"bucket","controllerGroup":"s3.services.k8s.aws","controllerKind":"Bucket","Bucket":{"name":"my-bucket","namespace":"ack-system"},"namespace":"ack-system","name":"my-bucket","reconcileID":"d3907857-c0bf-46e3-b803-f06a3bb63659","panic":"runtime error: invalid memory address or nil pointer dereference","panicGoValue":"\"invalid memory address or nil pointer dereference\"","stacktrace":"goroutine 242 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x296ebc8, 0xc000dfa690}, {0x217eb00, 0x3c50de0})\n\t/go/pkg/mod/k8s.io/apimachinery@v0.31.0/pkg/util/runtime/runtime.go:107 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:105 +0x112\npanic({0x217eb00?, 0x3c50de0?})\n\t/usr/local/go/src/runtime/panic.go:787 +0x132\ngithub.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceManager).addPutFieldsToSpec(0xc0001d1508, {0x296ebc8, 0xc000dfac60}, 0xc0006b3008, 0xc000748a08)\n\t/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/hook.go:326 +0x1b1\ngithub.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceManager).sdkFind(0xc0001d1508, {0x296ebc8, 0xc000dfac60}, 0xc0006b3008)\n\t/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/sdk.go:112 +0x4eb\ngithub.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceManager).ReadOne(0xc0001d1508, {0x296ebc8?, 0xc000dfac60?}, {0x297ca20?, 0xc0006b3008})\n\t/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/manager.go:104 +0x58\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync(0xc00013c5a0, {0x296ebc8, 0xc000dfac60}, {0x297c9b0, 0xc0001d1508}, {0x297ca20, 0xc0006b3008})\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.43.0/pkg/runtime/reconciler.go:511 +0xa58\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile(0xc00013c5a0, {0x296ebc8, 0xc000dfac60}, {0x297c9b0, 0xc0001d1508}, {0x297ca20, 0xc0006b3008})\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.43.0/pkg/runtime/reconciler.go:389 +0x265\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile(0xc00013c5a0, {0x296ebc8, 0xc000dfa690}, {{{0xc000b521b6?, 0x24ec226?}, {0xc00061d848?, 0x100?}}})\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.43.0/pkg/runtime/reconciler.go:288 +0xa25\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc000dfa600?, {0x296ebc8?, 0xc000dfa690?}, {{{0xc000b521b6?, 0x0?}, {0xc00061d848?, 0x0?}}})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x2987060, {0x296ec00, 0xc000c22460}, {{{0xc000b521b6, 0xa}, {0xc00061d848, 0x11}}})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x2987060, {0x296ec00, 0xc000c22460})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:263 +0x20d\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 120\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:220 +0x48d\n","stacktrace":"runtime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:925\ngithub.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceManager).addPutFieldsToSpec\n\t/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/hook.go:326\ngithub.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceManager).sdkFind\n\t/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/sdk.go:112\ngithub.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceManager).ReadOne\n\t/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/manager.go:104\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.43.0/pkg/runtime/reconciler.go:511\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.43.0/pkg/runtime/reconciler.go:389\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.43.0/pkg/runtime/reconciler.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:224"}
{"level":"error","ts":"2025-06-03T14:08:31.885Z","msg":"Reconciler error","controller":"bucket","controllerGroup":"s3.services.k8s.aws","controllerKind":"Bucket","Bucket":{"name":"my-bucket","namespace":"ack-system"},"namespace":"ack-system","name":"my-bucket","reconcileID":"d3907857-c0bf-46e3-b803-f06a3bb63659","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.0/pkg/internal/controller/controller.go:224"}

[github-actions]

Hello @calebhansard 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[michaelhtm]

Hello @calebhansard
This issue should be resolved in v1.0.32 Thanks to @simonoff

[calebhansard]

Thank you! It looks like v1.0.32 resolved the initial issue, but now I am receiving the following error:

{
  "level": "error",
  "ts": "2025-06-03T17:26:04.484Z",
  "msg": "Reconciler error",
  "controller": "bucket",
  "controllerGroup": "s3.services.k8s.aws",
  "controllerKind": "Bucket",
  "Bucket": { "name": "my-bucket", "namespace": "ack-system" },
  "namespace": "ack-system",
  "name": "my-bucket",
  "reconcileID": "5fde86dc-2620-4ce0-aaa1-3c2ca793ed29",
  "error": "Error syncing property 'Accelerate': operation error S3: PutBucketAccelerateConfiguration, https response error StatusCode: 400, RequestID: MYREQID, HostID: MYHOSTID, api error UnsupportedArgument: The request contained an unsupported argument.",
  "errorVerbose": "operation error S3: PutBucketAccelerateConfiguration, https response error StatusCode: 400, RequestID: MYREQID, HostID: MYHOSTID, api error UnsupportedArgument: The request contained an unsupported argument.\nError syncing property 'Accelerate'\ngithub.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceManager).customUpdateBucket\n\t/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/hook.go:195\ngithub.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceManager).sdkUpdate\n\t/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/sdk.go:246\ngithub.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceManager).Update\n\t/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/manager.go:157\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).updateResource\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.47.0/pkg/runtime/reconciler.go:757\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.47.0/pkg/runtime/reconciler.go:500\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.47.0/pkg/runtime/reconciler.go:381\ngithub.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile\n\t/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.47.0/pkg/runtime/reconciler.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"
}

[michaelhtm]

Thanks for the early response!
I am unable to recreate this issue locally.
Can you share the Delta difference or the kubectl describe output?

[calebhansard]

After further investigation, it looks as though the Error syncing property 'Accelerate' error is related to adopted bucket manifests that were automatically updated to include spec.accelerate. After manually removing that line from the affected manifests, the error went away. Buckets that are adopted with v1.0.32 do not include spec.accelerate and do not present any errors.

Note: I also had to manually remove the spec.website line as it was causing similar errors.

[michaelhtm]

Hello @calebhansard
Are you still encountering this issue when adopting this resource? There's a similar issue here and i still can't replicate them

[calebhansard]

No, I was able to resolve this issue, but I had to manually remove spec.accelerate and spec.website from my S3 bucket manifests. It looks as though those fields were automatically inserted by an older version of the S3 controller during resource adoption, and because they were not compatible with GovCloud, they were causing the Error syncing property issue.