panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference #3793

MolisXYliu · 2023-07-14T01:50:48Z

when i use karmada-operator to create karmada, it only create 3 pods,and i saw panic in operator log
the cr yaml is:

apiVersion: operator.karmada.io/v1alpha1
kind: Karmada
metadata:
  name: karmada-demo
  namespace: test
spec:
  components:
    etcd:
      local:
        imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers/etcd
        imageTag: 3.5.3-0
    karmadaAPIServer:
      imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver
      imageTag: v1.25.4
    karmadaAggregatedAPIServer:
      imageRepository: karmada/karmada-aggregated-apiserver
      imageTag: v1.4.0
    karmadaControllerManager:
      imageRepository: karmada/karmada-controller-manager
      imageTag: v1
    karmadaScheduler:
      imageRepository: karmada/karmada-scheduler
      imageTag: v1.4.0
    karmadaWebhook:
      imageRepository: karmada/karmada-webhook
      imageTag: v1.4.0
    kubeControllerManager:
      imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager
      imageTag: v1.25.4

the pod created are:
NAME READY STATUS RESTARTS AGE
karmada-demo-aggregated-apiserver-5586f4b9cd-cwqk9 1/1 Running 0 6m53s
karmada-demo-apiserver-57f59cf6c-bb7qd 1/1 Running 0 7m12s
karmada-demo-etcd-0 1/1 Running 0 7m15s

the opeartor log is:
I0713 09:45:03.881049 I0713 09:45:03.896229 I0713 09:45:03.896254 I0713 09:45:03.896358 I0713 09:45:03.896376 I0713 09:45:03.896383 I0713 09:45:03.919481 I0713 09:45:03.919508 I0713 09:45:04.287436 I0713 09:45:04.430357 I0713 09:45:05.194173 I0713 09:45:05.194205 I0713 09:45:05.380291 I0713 09:45:05.639565 I0713 09:45:05.639599 I0713 09:45:05.755831 I0713 09:45:06.007362 I0713 09:45:06.335290 I0713 09:45:06.335330 I0713 09:45:06.337617 I0713 09:45:06.345562 I0713 09:45:06.423329 I0713 09:45:06.432840 I0713 09:45:06.432861 I0713 09:45:06.579516 I0713 09:45:09.605369 I0713 09:45:12.647366 I0713 09:45:12.647399 I0713 09:45:12.748508 I0713 09:45:42.775566 I0713 09:45:42.775592 I0713 09:45:42.901829 I0713 09:45:42.901853 I0713 09:45:43.032229 I0713 09:45:47.096793 I0713 09:45:47.096824 I0713 09:45:47.102073 I0713 09:45:47.102110 I0713 09:45:47.167730 I0713 09:45:47.167850 I0713 09:45:47.167897 panic: runtime panic: runtime [signal SIGSEGV: 1 planner.go:85] "Start execute the workflow" workflow=init karmada="test/karmada-demo"
1 crd.go:48] "[prepare-crds] Running prepare-crds task" karmada="test/karmada-demo"
1 crd.go:49] "[prepare-crds] Using crd folder" folder="/var/lib/karmada/1.6.0" karmada="test/karmada-demo"
1 crd.go:69] "[download-crds] Skip download crd yaml files, the crd tar exists on disk" karmada="test/karmada-demo"
1 crd.go:126] "[unpack] These crds yaml files have been decompressed in the path" path="/var/lib/karmada/1.6.0/crds" karmada="test/karmada-demo"
1 crd.go:129] "[unpack] Successfully unpacked crd tar" karmada="test/karmada-demo"
1 cert.go:33] "[certs] Running certs task" karmada="test/karmada-demo"
1 cert.go:88] "[certs] Creating a new certificate authority" certName="ca"
1 cert.go:95] "[certs] Successfully generated ca certificate" certName="ca"
1 cert.go:129] "[certs] Successfully generated certificate" certName="karmada" caName="ca"
1 cert.go:129] "[certs] Successfully generated certificate" certName="apiserver" caName="ca"
1 cert.go:88] "[certs] Creating a new certificate authority" certName="front-proxy-ca"
1 cert.go:95] "[certs] Successfully generated ca certificate" certName="front-proxy-ca"
1 cert.go:129] "[certs] Successfully generated certificate" certName="front-proxy-client" caName="front-proxy-ca"
1 cert.go:88] "[certs] Creating a new certificate authority" certName="etcd-ca"
1 cert.go:95] "[certs] Successfully generated ca certificate" certName="etcd-ca"
1 cert.go:129] "[certs] Successfully generated certificate" certName="etcd-server" caName="etcd-ca"
1 cert.go:129] "[certs] Successfully generated certificate" certName="etcd-client" caName="etcd-ca"
1 namespace.go:28] "[namespace] Running namespace task" karmada="test/karmada-demo"
1 upload.go:168] "[upload-certs] Running upload-certs task" karmada="test/karmada-demo"
1 upload.go:201] "[upload-KarmadaCert] Successfully uploaded karmada certs to secret" karmada="test/karmada-demo"
1 upload.go:235] "[upload-etcdCert] Successfully uploaded etcd certs to secret" karmada="test/karmada-demo"
1 upload.go:262] "[upload-webhookCert] Successfully uploaded webhook certs to secret" karmada="test/karmada-demo"
1 etcd.go:40] "[etcd] Running etcd task" karmada="test/karmada-demo"
1 etcd.go:65] "[deploy-etcd] Successfully installed etcd component" karmada="test/karmada-demo"
1 reflector.go:559] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Watch close - *v1alpha1.Karmada total 9 items received
1 etcd.go:83] "[wait-etcd] the etcd pods is ready" karmada="test/karmada-demo"
1 apiserver.go:70] "[apiserver] Running apiserver task" karmada="test/karmada-demo"
1 apiserver.go:97] "[KarmadaApiserver] Successfully installed karmada-apiserver component" karmada="test/karmada-demo"
1 apiserver.go:114] "[wait-KarmadaAPIServer] the karmada-apiserver is ready" karmada="test/karmada-demo"
1 upload.go:42] "[upload-config] Running task" karmada="test/karmada-demo"
1 upload.go:96] "[UploadAdminKubeconfig] Successfully created secret of karmada apiserver kubeconfig" karmada="test/karmada-demo"
1 apiserver.go:60] "[aggregated-apiserver] Running aggregated apiserver task" karmada="test/karmada-demo"
1 apiserver.go:140] "[KarmadaAggregatedApiserver] Successfully installed karmada-aggregated-apiserver component" karmada="test/karmada-demo"
1 apiserver.go:157] "[wait-KarmadaAggregatedAPIServer] the karmada-aggregated-apiserver is ready" karmada="test/karmada-demo"
1 wait.go:39] "[check-apiserver-health] Running task" karmada="test/karmada-demo"
1 wait.go:47] "[check-apiserver-health] the etcd and karmada-apiserver is healthy" karmada="test/karmada-demo"
1 karmadaresource.go:60] "[karmadaResources] Running karmadaResources task" karmada="test/karmada-demo"
1 karmadaresource.go:79] "[systemName] Successfully created karmada system namespace" namespace="test" karmada="test/karmada-demo"
1 controller.go:51] "Finished syncing karmada" karmada="test/karmada-demo" duration="43.335576209s"
1 controller.go:118] "Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" controller="karmada" controllerGroup="operator.karmada.io" controllerKind="Karmada" Karmada="test/karmada-demo" namespace="test" name="karmada-demo" reconcileID=3a25f05b-c6db-44b5-bc7c-c5deb5b0d533
error: invalid memory address or nil pointer dereference [recovered]
error: invalid memory address or nil pointer dereference
segmentation violation code=0x1 addr=0x18 pc=0x16c44cb]

goroutine 249 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/home/runner/work/karmada/karmada/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0x1fa
panic({0x19204a0, 0x2ac3530})
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/panic.go:884 +0x213
github.com/karmada-io/karmada/operator/pkg/util.ListFiles.func1({0xc000903b30?, 0x21?}, {0x0, 0x0}, {0x0?, 0x0?})
/home/runner/work/karmada/karmada/operator/pkg/util/util.go:137 +0x2b
path/filepath.Walk({0xc000903b30, 0x21}, 0xc00030d630)
/opt/hostedtoolcache/go/1.20.4/x64/src/path/filepath/path.go:569 +0x50
github.com/karmada-io/karmada/operator/pkg/util.ListFiles({0xc000903b30?, 0xc0005b3700?})
/home/runner/work/karmada/karmada/operator/pkg/util/util.go:136 +0x52
github.com/karmada-io/karmada/operator/pkg/tasks/init.createCrds(0xc00030d858?, {0xc000903b30, 0x21})
/home/runner/work/karmada/karmada/operator/pkg/tasks/init/karmadaresource.go:119 +0x49
github.com/karmada-io/karmada/operator/pkg/tasks/init.runCrds({0x1ad1660?, 0xc0002ee1a0?})
/home/runner/work/karmada/karmada/operator/pkg/tasks/init/karmadaresource.go:100 +0x1f1
github.com/karmada-io/karmada/operator/pkg/workflow.run({{0x1b5d547, 0x4}, 0x1c5ef28, 0x0, {0x0, 0x0, 0x0}, 0x0}, {0x1ad1660, 0xc0002ee1a0})
/home/runner/work/karmada/karmada/operator/pkg/workflow/job.go:82 +0xb3
github.com/karmada-io/karmada/operator/pkg/workflow.run({{0x1b68045, 0x10}, 0x1c5ef50, 0x0, {0xc0001d3200, 0x4, 0x4}, 0x1}, {0x1ad1660, 0xc0002ee1a0})
/home/runner/work/karmada/karmada/operator/pkg/workflow/job.go:87 +0x1ce
github.com/karmada-io/karmada/operator/pkg/workflow.(*Job).Run(0xc000696c60)
/home/runner/work/karmada/karmada/operator/pkg/workflow/job.go:62 +0x118
github.com/karmada-io/karmada/operator/pkg/controller/karmada.(*Planner).Execute(0xc000696c90)
/home/runner/work/karmada/karmada/operator/pkg/controller/karmada/planner.go:90 +0x1ac
github.com/karmada-io/karmada/operator/pkg/controller/karmada.(*Controller).syncKarmada(0xc00003f7d0, 0xc000712900)
/home/runner/work/karmada/karmada/operator/pkg/controller/karmada/controller.go:90 +0x133
github.com/karmada-io/karmada/operator/pkg/controller/karmada.(*Controller).Reconcile(0xc00003f7d0, {0x1ddf6f0, 0xc000564030}, {{{0xc000897c88, 0x4}, {0xc000897c90, 0xc}}})
/home/runner/work/karmada/karmada/operator/pkg/controller/karmada/controller.go:80 +0x5c5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1ddf6f0?, {0x1ddf6f0?, 0xc000564030?}, {{{0xc000897c88?, 0x1852540?}, {0xc000897c90?, 0x0?}}})
/home/runner/work/karmada/karmada/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003dab40, {0x1ddf648, 0xc0002f34f0}, {0x1998580?, 0xc0003aea40?})
/home/runner/work/karmada/karmada/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323 +0x35f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003dab40, {0x1ddf648, 0xc0002f34f0})
/home/runner/work/karmada/karmada/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/home/runner/work/karmada/karmada/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
/home/runner/work/karmada/karmada/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:231 +0x587

RainbowMango · 2023-07-14T02:32:59Z

Hi @MolisXYliu Thanks for spotting. Could you please tell us which version of karmada-operator are you using?

MolisXYliu · 2023-07-14T02:37:15Z

Hi @MolisXYliu Thanks for spotting. Could you please tell us which version of karmada-operator are you using?

I use the latest karmada/karmada-operator

Poor12 · 2023-07-14T03:11:07Z

Thanks for spotting. I will take a look of it.

calvin0327 · 2023-07-17T03:02:49Z

https://github.com/karmada-io/karmada/blob/master/operator/pkg/util/util.go#L136

@MolisXYliu It seems to be an error running out here, I will track it.

calvin0327 · 2023-07-17T03:16:38Z

/assign

Poor12 · 2023-07-17T03:19:53Z

I have tried the same yaml in my env. It seems that everything is ok. I feel like some other trigger condition might be missing.
/cc @calvin0327 @MolisXYliu

MolisXYliu · 2023-07-17T03:22:13Z

I have tried the same yaml in my env. It seems that everything is ok. I feel like some other trigger condition might be missing. /cc @calvin0327 @MolisXYliu

my env is offline,and the crds is copy in the pod. dose it make trouble?

calvin0327 · 2023-07-17T03:23:39Z

@MolisXYliu Is this an offline environment and what operating system is the host?
Could you support these message? Thanks.

MolisXYliu · 2023-07-17T03:27:53Z

@MolisXYliu Is this an offline environment and what operating system is the host? Could you support these message? Thanks.

It an offline environment and my operating system is CentOS Linux release 7.6.1810 (Core)

calvin0327 · 2023-07-17T04:48:51Z

#3782 (comment)

Do you copy the crds package to the path /var/lib/karmada/1.6.0/crds.tar.gz?

MolisXYliu · 2023-07-17T05:38:46Z

#3782 (comment)

Do you copy the crds package to the path /var/lib/karmada/1.6.0/crds.tar.gz?

yes! I Copyed！ before i copyed no pod create。 after copy，opeator create 3 pod and print this fault

zishen · 2023-07-17T09:30:15Z

/var/lib/karmada/1.6.0/

You can try karmada v1.6.0， for the path /var/lib/karmada/1.6.0 may chang in v1.7.0.
And func give an nill path.

MolisXYliu · 2023-07-18T01:35:47Z

/var/lib/karmada/1.6.0/

You can try karmada v1.6.0， for the path /var/lib/karmada/1.6.0 may chang in v1.7.0. And func give an nill path.

3ks it works. by the way It is not very easy to deploy karmada cr from operator in a offline env.In the future version should we have a simple way to deploy karmada on offline env. such as point the crds path in yaml?

calvin0327 · 2023-07-18T01:56:29Z

Thanks for your feedback.
Yes, we be planning on building offline karmada operator for this scenario.

MolisXYliu · 2023-07-18T02:07:28Z

Good Job thank you very much!

MolisXYliu · 2023-08-14T09:38:12Z

this scenario

now can i ues it in ofline without copy crds?

MolisXYliu added the kind/bug Categorizes issue or PR as related to a bug. label Jul 14, 2023

karmada-bot assigned calvin0327 Jul 17, 2023

MolisXYliu closed this as completed Jul 18, 2023

zishen mentioned this issue Aug 21, 2023

REQUEST: New membership for zishen karmada-io/community#51

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference #3793

panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference #3793

MolisXYliu commented Jul 14, 2023 •

edited

Loading

RainbowMango commented Jul 14, 2023

MolisXYliu commented Jul 14, 2023

Poor12 commented Jul 14, 2023

calvin0327 commented Jul 17, 2023

calvin0327 commented Jul 17, 2023

Poor12 commented Jul 17, 2023

MolisXYliu commented Jul 17, 2023

calvin0327 commented Jul 17, 2023

MolisXYliu commented Jul 17, 2023

calvin0327 commented Jul 17, 2023

MolisXYliu commented Jul 17, 2023

zishen commented Jul 17, 2023

MolisXYliu commented Jul 18, 2023

calvin0327 commented Jul 18, 2023

MolisXYliu commented Jul 18, 2023

MolisXYliu commented Aug 14, 2023

panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference #3793

panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference #3793

Comments

MolisXYliu commented Jul 14, 2023 • edited Loading

RainbowMango commented Jul 14, 2023

MolisXYliu commented Jul 14, 2023

Poor12 commented Jul 14, 2023

calvin0327 commented Jul 17, 2023

calvin0327 commented Jul 17, 2023

Poor12 commented Jul 17, 2023

MolisXYliu commented Jul 17, 2023

calvin0327 commented Jul 17, 2023

MolisXYliu commented Jul 17, 2023

calvin0327 commented Jul 17, 2023

MolisXYliu commented Jul 17, 2023

zishen commented Jul 17, 2023

MolisXYliu commented Jul 18, 2023

calvin0327 commented Jul 18, 2023

MolisXYliu commented Jul 18, 2023

MolisXYliu commented Aug 14, 2023

MolisXYliu commented Jul 14, 2023 •

edited

Loading