Source controller panic: runtime error: invalid memory address or nil pointer dereference

[mrgreyves]

Describe the bug

Hello!
Installed flux versions of
flux: v0.29.2
helm-controller: v0.20.1
kustomize-controller: v0.24.2
notification-controller: v0.23.4
source-controller: v0.24.0

Periodically flux stops working with the following errors

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1c67451]

goroutine 346 [running]:
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).notify(0xc000a2a540, 0xc000832680, 0xc000757860, {{0xc000922640, 0x32}, {0x28875e0, 0xc000211200}, {0xc000a264b0, 0x3, 0x3}, ...}, ...)
	github.com/fluxcd/source-controller/controllers/helmrepository_controller.go:258 +0x1b1
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).reconcile(0x2935a68, {0x28cab98, 0xc000aeb590}, 0xc000757860, {0xc000611bf0, 0x3, 0xa})
	github.com/fluxcd/source-controller/controllers/helmrepository_controller.go:243 +0x29b
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).Reconcile(0xc000a2a540, {0x28cab98, 0xc000aeb590}, {{{0xc00044f906, 0x2478d00}, {0xc000a1e588, 0x30}}})
	github.com/fluxcd/source-controller/controllers/helmrepository_controller.go:205 +0x493
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc000012a50, {0x28cab98, 0xc000aeb500}, {{{0xc00044f906, 0x2478d00}, {0xc000a1e588, 0x415454}}})
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114 +0x26f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000012a50, {0x28caaf0, 0xc0009a9ac0}, {0x23138e0, 0xc000141960})
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311 +0x33e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000012a50, {0x28caaf0, 0xc0009a9ac0})
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:223 +0x357

Errors occur in the component source controller

Full log:

{"level":"info","ts":"2022-04-22T08:52:51.610Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2022-04-22T08:52:51.611Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2022-04-22T08:52:51.611Z","msg":"Starting server","kind":"health probe","addr":"[::]:9440"}
{"level":"info","ts":"2022-04-22T08:52:51.611Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
I0422 08:52:51.712658       1 leaderelection.go:248] attempting to acquire leader lease flux-system/source-controller-leader-election...
I0422 08:53:31.452185       1 leaderelection.go:258] successfully acquired lease flux-system/source-controller-leader-election
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"setup","msg":"starting file server"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.gitrepository","msg":"Starting EventSource","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","source":"kind source: *v1beta2.GitRepository"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.gitrepository","msg":"Starting Controller","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.helmrepository","msg":"Starting EventSource","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmRepository","source":"kind source: *v1beta2.HelmRepository"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.helmrepository","msg":"Starting Controller","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmRepository"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.helmchart","msg":"Starting EventSource","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","source":"kind source: *v1beta2.HelmChart"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.helmchart","msg":"Starting EventSource","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","source":"kind source: *v1beta2.HelmRepository"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.helmchart","msg":"Starting EventSource","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","source":"kind source: *v1beta2.GitRepository"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.bucket","msg":"Starting EventSource","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"Bucket","source":"kind source: *v1beta2.Bucket"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.bucket","msg":"Starting Controller","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"Bucket"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.helmchart","msg":"Starting EventSource","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","source":"kind source: *v1beta2.Bucket"}
{"level":"info","ts":"2022-04-22T08:53:31.452Z","logger":"controller.helmchart","msg":"Starting Controller","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart"}
{"level":"info","ts":"2022-04-22T08:53:31.553Z","logger":"controller.gitrepository","msg":"Starting workers","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","worker count":2}
{"level":"info","ts":"2022-04-22T08:53:31.553Z","logger":"controller.bucket","msg":"Starting workers","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"Bucket","worker count":2}
{"level":"info","ts":"2022-04-22T08:53:31.553Z","logger":"controller.helmrepository","msg":"Starting workers","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmRepository","worker count":2}
{"level":"info","ts":"2022-04-22T08:53:31.553Z","logger":"controller.helmchart","msg":"Starting workers","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","worker count":2}
{"level":"info","ts":"2022-04-22T08:53:31.842Z","logger":"controller.helmchart","msg":"artifact up-to-date with remote revision: '34.10.0'","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","name":"monitoring-prometheus-community","namespace":"monitoring"}
{"level":"info","ts":"2022-04-22T08:53:33.066Z","logger":"controller.gitrepository","msg":"artifact up-to-date with remote revision: 'main/7d57a646e7a2ded954d8b8f6e912c43edeaa1f76'","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"flux-system","namespace":"flux-system"}
{"level":"info","ts":"2022-04-22T08:53:34.372Z","logger":"controller.helmchart","msg":"artifact up-to-date with remote revision: '10.3.10'","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","name":"consul-consul","namespace":"consul"}
{"level":"info","ts":"2022-04-22T08:53:34.689Z","logger":"controller.helmchart","msg":"artifact up-to-date with remote revision: '9.1.26'","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmChart","name":"nginx-ingress-controller-nginx-ingress-controller","namespace":"nginx-ingress-controller"}
{"level":"info","ts":"2022-04-22T08:53:34.804Z","logger":"controller.helmrepository","msg":"artifact up-to-date with remote revision: 'b89a751ad2a9cc03772861f9481ea13fd959daf00b426ed59fcf8b1126b12377'","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"HelmRepository","name":"consul","namespace":"consul"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1c67451]

goroutine 435 [running]:
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).notify(0xc0006ac360, 0xc000a86820, 0xc0007376c0, {{0xc0005c1050, 0x22}, {0x28875e0, 0xc0009e09c0}, {0xc0006caae0, 0x3, 0x3}, ...}, ...)
	github.com/fluxcd/source-controller/controllers/helmrepository_controller.go:258 +0x1b1
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).reconcile(0x2935a68, {0x28cab98, 0xc000958660}, 0xc0007376c0, {0xc0008b3bf0, 0x3, 0x2})
	github.com/fluxcd/source-controller/controllers/helmrepository_controller.go:243 +0x29b
github.com/fluxcd/source-controller/controllers.(*HelmRepositoryReconciler).Reconcile(0xc0006ac360, {0x28cab98, 0xc000958660}, {{{0xc000983f06, 0x2478d00}, {0xc000983f00, 0x30}}})
	github.com/fluxcd/source-controller/controllers/helmrepository_controller.go:205 +0x493
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc0005460b0, {0x28cab98, 0xc0009585d0}, {{{0xc000983f06, 0x2478d00}, {0xc000983f00, 0x415454}}})
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114 +0x26f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0005460b0, {0x28caaf0, 0xc000a091c0}, {0x23138e0, 0xc0002fcde0})
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311 +0x33e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005460b0, {0x28caaf0, 0xc000a091c0})
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:223 +0x357

Steps to reproduce

1. Bootstrap flux with gitlab repo
2. Check source controller logs

Expected behavior

Flux works =)

Screenshots and recordings

No response

OS / Distro

Macos 12.2.1

Flux version

v0.29.2

Flux check

► checking prerequisites
✔ Kubernetes 1.21.5 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.20.1
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.24.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.23.4
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.24.0
✔ all checks passed

Git provider

Gitlab

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[hiddeco]

Duplicate of fluxcd/source-controller#680, addressed via fluxcd/source-controller#683, release currently being build via fluxcd/source-controller#684. CLI patch should be available within the next couple of hours.

[hiddeco]

To move forward now (the source-controller image is already published), you should be able to move the image tag to v0.24.1 and see the issue resolved.

[mrgreyves]

Updated the image
Everything seems to be fine
Thank you very much =)

[hiddeco]

Now officially available as https://github.com/fluxcd/flux2/releases/tag/v0.29.3 :-)