CrashLoop with "unexpected fault address 0x7f15db1b1000"

[agniveshadhikari]

Somehow the cluster went into a state where one server pod was in Evicted state, and another was in crashloopbackoff. I deleted both, so that the new pod that would be created would run smoothly, but this one also crash-loops with the following error message:

level=info ts=2020-12-16T19:26:28.048Z caller=main.go:343 msg="Starting Prometheus" version="(version=2.20.1, branch=HEAD, revision=983ebb4a513302315a8117932ab832815f85e3d2)"
level=info ts=2020-12-16T19:26:28.048Z caller=main.go:344 build_context="(go=go1.14.6, user=root@7cbd4d1c15e0, date=20200805-17:26:58)"
level=info ts=2020-12-16T19:26:28.048Z caller=main.go:345 host_details="(Linux 5.4.0-1026-azure #26~18.04.1-Ubuntu SMP Thu Sep 10 16:19:25 UTC 2020 x86_64 core-charts-prometheus-server-66fb9b87fb-b5cc4 (none))"
level=info ts=2020-12-16T19:26:28.048Z caller=main.go:346 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2020-12-16T19:26:28.048Z caller=main.go:347 vm_limits="(soft=unlimited, hard=unlimited)"
unexpected fault address 0x7f15db1b1000
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7f15db1b1000 pc=0x468938]

goroutine 1 [running]:
runtime.throw(0x286a02d, 0x5)
        /usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc000a93300 sp=0xc000a932d0 pc=0x434b92
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:692 +0x443 fp=0xc000a93330 sp=0xc000a93300 pc=0x44b733
runtime.memmove(0x7f15db1b1000, 0x2867456, 0x1)
        /usr/local/go/src/runtime/memmove_amd64.s:148 +0x108 fp=0xc000a93338 sp=0xc000a93330 pc=0x468938
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffdea9779a0, 0x5, 0x14, 0x30898a0, 0xc000846450, 0x30898a0)
        /app/promql/query_logger.go:120 +0x3a3 fp=0xc000a93490 sp=0xc000a93338 pc=0x1ee2433
main.main()
        /app/cmd/prometheus/main.go:374 +0x4f08 fp=0xc000a93f88 sp=0xc000a93490 pc=0x2099d08
runtime.main()
        /usr/local/go/src/runtime/proc.go:203 +0x1fa fp=0xc000a93fe0 sp=0xc000a93f88 pc=0x4371fa
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc000a93fe8 sp=0xc000a93fe0 pc=0x467671

goroutine 15 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0005207d0)
        /app/vendor/go.opencensus.io/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
        /app/vendor/go.opencensus.io/stats/view/worker.go:32 +0x57

goroutine 16 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x44d9e80)
        /app/vendor/k8s.io/klog/v2/klog.go:1107 +0x8b
created by k8s.io/klog/v2.init.0
        /app/vendor/k8s.io/klog/v2/klog.go:416 +0xd6

goroutine 115 [select]:
github.com/prometheus/prometheus/pkg/logging.(*Deduper).run(0xc0002dec00)
        /app/pkg/logging/dedupe.go:75 +0x1db
created by github.com/prometheus/prometheus/pkg/logging.Dedupe
        /app/pkg/logging/dedupe.go:61 +0xcf

goroutine 121 [chan receive]:
github.com/prometheus/prometheus/storage/remote.(*WriteStorage).run(0xc000981f80)
        /app/storage/remote/write.go:84 +0xb2
created by github.com/prometheus/prometheus/storage/remote.NewWriteStorage
        /app/storage/remote/write.go:77 +0x19a

I tried deleting this pod, and the new one that spawned also crash-loops with the fault address being 0x7f2b94933000

What could have caused this, and how to resolve this?

[brian-brazil]

What is the length and content of your queries.active file? What logs are there from the previous run(s) of Prometheus?

[marcosantar93]

I was having a similar issue and the problem was that the PVC used by prometheus was at full capacity.
You can check with kubectl df-pv plugin:

kubectl df-pv

This was the output for mine:

 PV NAME                                   PVC NAME                                                                                                        NAMESPACE   POD NAME                                                         VOLUME MOUNT NAME                                        SIZE       USED    AVAILABLE  %USED  IUSED  IFREE   %IUSED 
 pvc-53fbb371-b707-43e9-a39c-206004c0842a  prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0          metrics     prometheus-prometheus-kube-prometheus-prometheus-0               prometheus-prometheus-kube-prometheus-prometheus-db      7Gi        7Gi     0          99.80  101    524187  0.02   

I resized the PVC to 16Gi and the issue disappeared. You should also change your retentionSize. Hope this helps!

[rapkalya]

i got this error resolved by clearing space in root directory

[DeveloperSasmita]

I had the same issue quite many times. Try working around retention.size limit or retention.time .
Details can be found here : https://www.robustperception.io/configuring-prometheus-storage-retention

[shan100github]

faced the same issue did the following.

1. used kubectl df-pv as mentioned by @marcosantar93 & located the worker node dir to be cleared off.
   fdisk -l & df -h were handy
2. resized PVC to higher number
3. set the retentionSize & PV size in helm values and did helm upgrade.
4. made sure the volume identified in step1 is resized.