s3-controller runtime panic after updating from v0.0.2 to v0.0.3

[dervos]

Describe the bug
When I upgraded from v0.0.2 to v0.0.3 the controller panics when reconciling a bucket.s3.services.k8s.aws/v1alpha1 object created from version v0.0.2.
Observed a panic: "invalid memory address or nil pointer dereference"
When I re-created the Bucket crd opject the controller no longer crashed.

Steps to reproduce

1. Install s3-controller v0.0.2
2. Create bucket.s3.services.k8s.aws/v1alpha1 in v0.0.2
3. Upgrade s3-controller to v0.0.3

Expected outcome

• The controller shouldn't crash
• It should print out a readable error message

Environment

• Kubernetes version: 1.21
• Using EKS (yes/no), if so version? EKS 1.21
• AWS service targeted: S3

I0825 16:01:29.217619       1 request.go:621] Throttling request took 1.002874757s, request: GET:https://10.100.0.1:443/apis/scheduling.k8s.io/v1beta1?timeout=32s
2021-08-25T16:01:30.721Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "0.0.0.0:8080"}
2021-08-25T16:01:30.721Z	INFO	setup	initializing service controller	{"aws.service": "s3"}
2021-08-25T16:01:30.723Z	INFO	setup	starting manager	{"aws.service": "s3"}
2021-08-25T16:01:30.725Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-08-25T16:01:30.725Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "bucket", "source": "kind source: /, Kind="}
2021-08-25T16:01:30.725Z	INFO	controller-runtime.controller	Starting EventSource	{"controller": "adoptedresource", "source": "kind source: /, Kind="}
2021-08-25T16:01:30.825Z	INFO	controller-runtime.controller	Starting Controller	{"controller": "bucket"}
2021-08-25T16:01:30.825Z	INFO	controller-runtime.controller	Starting workers	{"controller": "bucket", "worker count": 1}
2021-08-25T16:01:30.826Z	INFO	controller-runtime.controller	Starting Controller	{"controller": "adoptedresource"}
2021-08-25T16:01:30.826Z	INFO	controller-runtime.controller	Starting workers	{"controller": "adoptedresource", "worker count": 1}
E0825 16:01:31.650621       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 364 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1680ce0, 0x27cdf70)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:48 +0x82
panic(0x1680ce0, 0x27cdf70)
	/usr/local/go/src/runtime/panic.go:967 +0x166
github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.customPreCompare(0xc00000e6c8, 0xc00000e410)
	/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/hook.go:303 +0x95d
github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.newResourceDelta(0xc00000e6c8, 0xc00000e410, 0x0)
	/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/delta.go:41 +0x75
github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceDescriptor).Delta(0x2816810, 0x1c3a380, 0xc00000e6c8, 0x1c3a380, 0xc00000e410, 0x0)
	/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/descriptor.go:70 +0x54
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).updateResource(0xc000136c60, 0x1c24b80, 0xc0001fc6f0, 0x1c2e320, 0xc0005daf00, 0x1c3a380, 0xc00000e6c8, 0x1c3a380, 0xc00000e410, 0x0, ...)
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.12.0/pkg/runtime/reconciler.go:323 +0x1d5
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync(0xc000136c60, 0x1c24b80, 0xc0001fc6f0, 0x1c2e320, 0xc0005daf00, 0x1c3a380, 0xc00000e6c8, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.12.0/pkg/runtime/reconciler.go:221 +0x78d
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile(0xc000136c60, 0x1c24b80, 0xc0001fc6f0, 0x1c2e320, 0xc0005daf00, 0x1c3a380, 0xc00000e6c8, 0xc, 0x7ffe42d95b8d, 0x0, ...)
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.12.0/pkg/runtime/reconciler.go:181 +0x130
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile(0xc000136c60, 0xc0005b0630, 0x7, 0xc000274b20, 0x1b, 0x0, 0xc041b836b13a3b8e, 0xc00044c630, 0xc00044c5a8)
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.12.0/pkg/runtime/reconciler.go:165 +0x7e0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000146a80, 0x16e2560, 0xc0000dbc80, 0x7f138f1e7100)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256 +0x161
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000146a80, 0x203000)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232 +0xae
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000146a80)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00057c7a0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00057c7a0, 0x1beb9a0, 0xc0001fc330, 0x1, 0xc0005d41e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00057c7a0, 0x3b9aca00, 0x0, 0x1910b01, 0xc0005d41e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00057c7a0, 0x3b9aca00, 0xc0005d41e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:193 +0x305
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x151501d]

goroutine 364 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:55 +0x105
panic(0x1680ce0, 0x27cdf70)
	/usr/local/go/src/runtime/panic.go:967 +0x166
github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.customPreCompare(0xc00000e6c8, 0xc00000e410)
	/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/hook.go:303 +0x95d
github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.newResourceDelta(0xc00000e6c8, 0xc00000e410, 0x0)
	/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/delta.go:41 +0x75
github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket.(*resourceDescriptor).Delta(0x2816810, 0x1c3a380, 0xc00000e6c8, 0x1c3a380, 0xc00000e410, 0x0)
	/github.com/aws-controllers-k8s/s3-controller/pkg/resource/bucket/descriptor.go:70 +0x54
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).updateResource(0xc000136c60, 0x1c24b80, 0xc0001fc6f0, 0x1c2e320, 0xc0005daf00, 0x1c3a380, 0xc00000e6c8, 0x1c3a380, 0xc00000e410, 0x0, ...)
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.12.0/pkg/runtime/reconciler.go:323 +0x1d5
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync(0xc000136c60, 0x1c24b80, 0xc0001fc6f0, 0x1c2e320, 0xc0005daf00, 0x1c3a380, 0xc00000e6c8, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.12.0/pkg/runtime/reconciler.go:221 +0x78d
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile(0xc000136c60, 0x1c24b80, 0xc0001fc6f0, 0x1c2e320, 0xc0005daf00, 0x1c3a380, 0xc00000e6c8, 0xc, 0x7ffe42d95b8d, 0x0, ...)
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.12.0/pkg/runtime/reconciler.go:181 +0x130
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile(0xc000136c60, 0xc0005b0630, 0x7, 0xc000274b20, 0x1b, 0x0, 0xc041b836b13a3b8e, 0xc00044c630, 0xc00044c5a8)
	/go/pkg/mod/github.com/aws-controllers-k8s/runtime@v0.12.0/pkg/runtime/reconciler.go:165 +0x7e0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000146a80, 0x16e2560, 0xc0000dbc80, 0x7f138f1e7100)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256 +0x161
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000146a80, 0x203000)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232 +0xae
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000146a80)
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00057c7a0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00057c7a0, 0x1beb9a0, 0xc0001fc330, 0x1, 0xc0005d41e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00057c7a0, 0x3b9aca00, 0x0, 0x1910b01, 0xc0005d41e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00057c7a0, 0x3b9aca00, 0xc0005d41e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:193 +0x305

[RedbackThomson]

Thanks for the report! I see hook.go:303 from the stack trace, which would be here:

https://github.com/aws-controllers-k8s/s3-controller/blob/a2654b987176602d7e517748c94903efd4a12726/pkg/resource/bucket/hook.go#L303

This appears to be referencing the Accelerate.Status object from one of the comparison objects, which was not nil checked.

S3 v0.0.3 introduced a lot of custom code, to support the strange style of the S3 API requests. I'm sure I missed some other places, but I'll double check all of my pointer dereferences in this file and put out a new version. Thanks!