Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nil pointer exception with clusterLogging.Spec.Collection empty and Forwarder enabled #2312

Closed
andreaskaris opened this issue Jan 18, 2024 · 3 comments · Fixed by #2313
Closed

Nil pointer exception with clusterLogging.Spec.Collection empty and Forwarder enabled #2312

andreaskaris opened this issue Jan 18, 2024 · 3 comments · Fixed by #2313
Assignees
@andreaskaris

Comments

@andreaskaris
Copy link
Contributor

andreaskaris commented Jan 18, 2024 •
edited

Nil pointer exception with clusterLogging.Spec.Collection empty and Forwarder enabled.

The following combination throws a nil pointer exception:

ClusterLogging:

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance" 
  namespace: openshift-logging
spec:
  managementState: "Managed"

ClusterLogForwarder:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  pipelines:
   - inputRefs:
     - audit
     outputRefs:
     - loki-external
     labels:
       job: openshift-audit
       clusterid: sno10
   - inputRefs:
     - application
     outputRefs:
     - loki-external
     labels:
       job: openshift-application
       clusterid: sno10
   - inputRefs:
     - infrastructure
     outputRefs:
     - loki-external
     labels:
       job: openshift-infrastructure
       clusterid: sno10
  outputs:
  - name: loki-external
    type: loki
    url: http://192.168.18.75:310

[akaris@workstation logging]$ oc apply -f configuring-cluster-logging-minimal-minimal.yaml
clusterlogging.logging.openshift.io/instance created
[akaris@workstation logging]$ oc apply -f configuring-log-forwarder.yaml
clusterlogforwarder.logging.openshift.io/instance created
[akaris@workstation logging]$ 

$ LOG_LEVEL=3 make run
(...)
{"_ts":"2024-01-18T11:18:45.623084752Z","_file:line":"internal/reconcile/rbac.go:37","_level":"3","_component":"cluster-logging-operator","_message":"Roles are the same skipping update"}
{"_ts":"2024-01-18T11:18:45.623126639Z","_file:line":"internal/reconcile/rbac.go:70","_level":"3","_component":"cluster-logging-operator","_message":"RoleBindings are the same skipping update"}
{"_ts":"2024-01-18T11:18:45.623159759Z","_file:line":"internal/k8shandler/clusterloggingrequest.go:107","_level":"3","_component":"cluster-logging-operator","_message":"Getting object","namespacedName":{"Namespace":"openshift-logging","Name":"logcollector-token"},"object":{"metadata":{"creationTimestamp":null}}}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15c31d8]

goroutine 351 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1e5
panic({0x1737be0?, 0x28a4ed0?})
	/usr/lib/golang/src/runtime/panic.go:914 +0x21f
github.com/openshift/cluster-logging-operator/internal/k8shandler.(*ClusterLoggingRequest).generateCollectorConfig(0xc000927058)
	/home/akaris/development/cluster-logging-operator/internal/k8shandler/forwarding.go:48 +0x298
github.com/openshift/cluster-logging-operator/internal/k8shandler.(*ClusterLoggingRequest).CreateOrUpdateCollection(0xc000927058)
	/home/akaris/development/cluster-logging-operator/internal/k8shandler/collection.go:72 +0x2b2
github.com/openshift/cluster-logging-operator/internal/k8shandler.Reconcile(0xc0008d84e0, 0xc000898a80, {0x1c12238, 0xc000290180}, {0x7fb45823e568, 0xc000292770}, {0x1c099b0, 0xc0002d8340}, {0xc0003ca158, 0x6}, ...)
	/home/akaris/development/cluster-logging-operator/internal/k8shandler/reconciler.go:67 +0x81f
github.com/openshift/cluster-logging-operator/controllers/forwarding.(*ReconcileForwarder).Reconcile(0xc000700420, {0x30?, 0xc000588000?}, {{{0xc00063dfb0?, 0x0?}, {0xc000331060?, 0x412205?}}})
	/home/akaris/development/cluster-logging-operator/controllers/forwarding/forwarding_controller.go:115 +0xc35
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1c0ab18?, {0x1c0ab18?, 0xc000b04d50?}, {{{0xc00063dfb0?, 0x1684940?}, {0xc000331060?, 0xc000c037d0?}}})
	/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000298460, {0x1c0ab50, 0xc0002a90e0}, {0x17b7fe0?, 0xc000332da0?})
	/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323 +0x353
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000298460, {0x1c0ab50, 0xc0002a90e0})
	/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 107
	/home/akaris/development/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:231 +0x565
exit status 2
make: *** [Makefile:150: run] Error 1
@andreaskaris
Copy link
Contributor Author

/assign

@andreaskaris andreaskaris mentioned this issue Jan 18, 2024
Merged
@jasaltvik
Copy link

Hi @andreaskaris. We just upgraded to OpenShift Logging 5.8 (from 5.7), and are now experiencing this. We are using our own custom Vector, so the collection config stanza is removed. We had an unused ClusterLogForwarder in the cluster, but this is now removed. The cluster logging operator still fails with the same error. Is this expected? If yes, are there any workarounds for this (before #2313 is merged and released)?

Current error:

{"_ts":"2024-02-07T11:24:08.571053532Z","_level":"0","_component":"cluster-logging-operator","_message":"starting up...","go_arch":"amd64","go_os":"linux","go_version":"go1.20.10","operator_version":"5.8.0"}
I0207 11:24:09.623336       1 request.go:690] Waited for 1.028887056s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/broker.amq.io/v2alpha3?timeout=32s
{"_ts":"2024-02-07T11:24:14.686185523Z","_level":"0","_component":"cluster-logging-operator","_message":"migrating resources provided by the manifest"}
{"_ts":"2024-02-07T11:24:14.689878193Z","_level":"0","_component":"cluster-logging-operator","_message":"Registering Components."}
{"_ts":"2024-02-07T11:24:14.705099482Z","_level":"0","_component":"cluster-logging-operator","_message":"Starting the Cmd."}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16b3a38]
goroutine 241 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1fa
panic({0x183a2a0, 0x2946b30})
	/usr/lib/golang/src/runtime/panic.go:884 +0x213
github.com/openshift/cluster-logging-operator/internal/k8shandler.(*ClusterLoggingRequest).generateCollectorConfig(0xc000a4cdb8)
	/remote-source/cluster-logging-operator/app/internal/k8shandler/forwarding.go:50 +0x298
github.com/openshift/cluster-logging-operator/internal/k8shandler.(*ClusterLoggingRequest).CreateOrUpdateCollection(0xc000a4cdb8)
	/remote-source/cluster-logging-operator/app/internal/k8shandler/collection.go:72 +0x2b2
github.com/openshift/cluster-logging-operator/internal/k8shandler.Reconcile(0xc000698340, 0xc0000f8fc0, {0x1d16c38, 0xc00064c180}, {0x7f17183a6e28, 0xc0000f6230}, {0x1d0e5b0, 0xc0005e2040}, {0xc0005fc1e9, 0x7}, ...)
	/remote-source/cluster-logging-operator/app/internal/k8shandler/reconciler.go:67 +0x845
github.com/openshift/cluster-logging-operator/controllers/clusterlogging.(*ReconcileClusterLogging).Reconcile(0xc0005f62a0, {0x1d0f0e0, 0xc0008904b0}, {{{0xc0003bd050?, 0x11?}, {0xc00061ee58?, 0x8?}}})
	/remote-source/cluster-logging-operator/app/controllers/clusterlogging/clusterlogging_controller.go:120 +0xc87
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1d0f0e0?, {0x1d0f0e0?, 0xc0008904b0?}, {{{0xc0003bd050?, 0x17bd100?}, {0xc00061ee58?, 0x0?}}})
	/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003f41e0, {0x1d0f038, 0xc0002d6ff0}, {0x18c0e60?, 0xc000456b00?})
	/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323 +0x35f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003f41e0, {0x1d0f038, 0xc0002d6ff0})
	/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/remote-source/cluster-logging-operator/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:231 +0x587

@jasaltvik
Copy link

jasaltvik commented Feb 7, 2024 •
edited

@andreaskaris Hmm, it actually seems to work when adding an empty collection config stanza

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection: {}

So it seems all is good now.

andreaskaris added a commit to andreaskaris/cluster-logging-operator that referenced this issue Feb 9, 2024
@andreaskaris
Test: catch nil ptr exceptions, update CLF status when CL becomes valid
fd75b65 
Test openshift#2312
and openshift#2314

Signed-off-by: Andreas Karis <ak.karis@gmail.com>
andreaskaris added a commit to andreaskaris/cluster-logging-operator that referenced this issue Feb 9, 2024
@andreaskaris
Test: catch nil ptr exceptions, update CLF status when CL becomes valid
44408cd 
Test openshift#2312
and openshift#2314

Signed-off-by: Andreas Karis <ak.karis@gmail.com>
andreaskaris added a commit to andreaskaris/cluster-logging-operator that referenced this issue Feb 19, 2024
@andreaskaris
Test: catch nil ptr exceptions, update CLF status when CL becomes valid
7483ea0 
Test openshift#2312
and openshift#2314

Signed-off-by: Andreas Karis <ak.karis@gmail.com>
andreaskaris added a commit to andreaskaris/cluster-logging-operator that referenced this issue Feb 19, 2024
@andreaskaris
Test: catch nil ptr exceptions, update CLF status when CL becomes valid
682eded 
Test openshift#2312
and openshift#2314

Signed-off-by: Andreas Karis <ak.karis@gmail.com>
andreaskaris added a commit to andreaskaris/cluster-logging-operator that referenced this issue Feb 19, 2024
@andreaskaris
Test: catch nil ptr exceptions, update CLF status when CL becomes valid
944f73c 
Test openshift#2312
and openshift#2314

Signed-off-by: Andreas Karis <ak.karis@gmail.com>
andreaskaris added a commit to andreaskaris/cluster-logging-operator that referenced this issue Feb 19, 2024
@andreaskaris
Test: catch nil ptr exceptions, update CLF status when CL becomes valid
9c641ce 
Test openshift#2312
and openshift#2314

Signed-off-by: Andreas Karis <ak.karis@gmail.com>
andreaskaris added a commit to andreaskaris/cluster-logging-operator that referenced this issue Feb 19, 2024
@andreaskaris
Test: catch nil ptr exceptions, update CLF status when CL becomes valid
522d892 
Test openshift#2312
and openshift#2314

Signed-off-by: Andreas Karis <ak.karis@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andreaskaris andreaskaris

Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@andreaskaris @jasaltvik