source=main.go:768 msg="Leaving GOMAXPROCS=12: CPU quota undefined" component=automaxprocs

[baryluk]

What did you do?

Updated from v3.1.0 to v3.3.0 on one of the machines.

docker container, using images from quay.io

Host system centos 9 amd64. Some in AWS (x86), some VMs on proxmox.

I have very similar setup (all with same configs and flags, but in terms of host system, 2 are on proxmox, others are on some AWS EC2 instances, but all inside docker containers), and update was fine on 6 different instances (all same flags and env variables if any), but last one failed (one on proxmox, with possibly a bit older Linux kernel and kernel flags).

[foo@bar ~]$ docker logs -f prometheus 
time=2025-04-22T17:12:47.644Z level=INFO source=main.go:228 msg="Experimental memory snapshot on shutdown enabled"
time=2025-04-22T17:12:47.658Z level=INFO source=main.go:687 msg="Starting Prometheus Server" mode=server version="(version=3.3.0, branch=HEAD, revision=3c49e69330ceee9c587a1d994c78535695edff16)"
time=2025-04-22T17:12:47.658Z level=INFO source=main.go:692 msg="operational information" build_context="(go=go1.24.1, platform=linux/amd64, user=root@b362765c931d, date=20250415-13:32:00, tags=netgo,builtinassets,stringlabels)" host_details="(Linux 5.14.0-533.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 22 15:05:06 UTC 2024 x86_64 enrico-prometheus (none))" fd_limits="(soft=1073741816, hard=1073741816)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2025-04-22T17:12:47.668Z level=INFO source=main.go:768 msg="Leaving GOMAXPROCS=12: CPU quota undefined" component=automaxprocs

then prometheus process just exits without continuing, or any more error messages.

[foo@bar ~]$ lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          52 bits physical, 57 bits virtual
  Byte Order:             Little Endian
CPU(s):                   12
  On-line CPU(s) list:    0-11
Vendor ID:                AuthenticAMD
  Model name:             AMD EPYC 9454P 48-Core Processor

If I set environment variable GOMAXPROCS=12 it starts:

[foo@bar ~]$ docker logs -f prometheus 
time=2025-04-22T17:15:58.836Z level=INFO source=main.go:228 msg="Experimental memory snapshot on shutdown enabled"
time=2025-04-22T17:15:58.841Z level=INFO source=main.go:687 msg="Starting Prometheus Server" mode=server version="(version=3.3.0, branch=HEAD, revision=3c49e69330ceee9c587a1d994c78535695edff16)"
time=2025-04-22T17:15:58.841Z level=INFO source=main.go:692 msg="operational information" build_context="(go=go1.24.1, platform=linux/amd64, user=root@b362765c931d, date=20250415-13:32:00, tags=netgo,builtinassets,stringlabels)" host_details="(Linux 5.14.0-533.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 22 15:05:06 UTC 2024 x86_64 enrico-prometheus (none))" fd_limits="(soft=1073741816, hard=1073741816)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2025-04-22T17:15:58.842Z level=INFO source=main.go:768 msg="Honoring GOMAXPROCS=\"12\" as set in environment" component=automaxprocs
time=2025-04-22T17:15:58.846Z level=INFO source=web.go:654 msg="Start listening for connections" component=web address=0.0.0.0:9090
time=2025-04-22T17:15:58.849Z level=INFO source=main.go:1228 msg="Starting TSDB ..."
time=2025-04-22T17:15:58.856Z level=INFO source=tls_config.go:347 msg="Listening on" component=web address=[::]:9090
...

Maybe because of this:

[foo@bar ~]$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos2)/vmlinuz-5.14.0-533.el9.x86_64 root=/dev/mapper/vg_root-root ro audit=1 resume=/dev/mapper/vg_root-swap rd.lvm.lv=vg_root/root rd.lvm.lv=vg_root/swap LANG=en_US.UTF-8 transparent_hugepage=never systemd.unified_cgroup_hierarchy=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
[foo@bar ~]$ 

???

This disabled cgroup v2, and uses legacy cgroup v1. I will for switch to v2, but I do not want to reboot this host atm.

Still prometheus should not crash?

[root@bar ~]# docker version
Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:42:48 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:09 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[root@bar ~]# 

[machine424]

you can try to run it with --no-auto-gomaxprocs to skip that code and see if it has anything to do with the failure.
learning about how the process exits (does it receive a signal or sth?) would also help, docker may be able to tell you about that.

[baryluk]

I am sure that passing --no-auto-gomaxprocs would work, as a workaround, but that does not addresses the underlying issue. (Setting GOMAXPROC manually is a workaround that also works fine)

Outside of docker it works, despite Leaving GOMAXPROCS=12: CPU quota undefined message:

[foo@bar prometheus-3.3.0.linux-amd64]$ ./prometheus --web.listen-address=0.0.0.0:9011
time=2025-04-23T16:55:47.206Z level=INFO source=main.go:640 msg="No time or size retention was set so using the default time retention" duration=15d
time=2025-04-23T16:55:47.206Z level=INFO source=main.go:687 msg="Starting Prometheus Server" mode=server version="(version=3.3.0, branch=HEAD, revision=3c49e69330ceee9c587a1d994c78535695edff16)"
time=2025-04-23T16:55:47.206Z level=INFO source=main.go:692 msg="operational information" build_context="(go=go1.24.1, platform=linux/amd64, user=root@b362765c931d, date=20250415-13:32:00, tags=netgo,builtinassets,stringlabels)" host_details="(Linux 5.14.0-533.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 22 15:05:06 UTC 2024 x86_64 enrico.tensor-tech.io (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2025-04-23T16:55:47.208Z level=INFO source=main.go:768 msg="Leaving GOMAXPROCS=12: CPU quota undefined" component=automaxprocs
time=2025-04-23T16:55:47.216Z level=INFO source=web.go:654 msg="Start listening for connections" component=web address=0.0.0.0:9011
time=2025-04-23T16:55:47.218Z level=INFO source=main.go:1228 msg="Starting TSDB ..."
time=2025-04-23T16:55:47.221Z level=INFO source=tls_config.go:347 msg="Listening on" component=web address=[::]:9011
time=2025-04-23T16:55:47.221Z level=INFO source=tls_config.go:350 msg="TLS is disabled." component=web http2=false address=[::]:9011
time=2025-04-23T16:55:47.223Z level=INFO source=head.go:638 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
time=2025-04-23T16:55:47.223Z level=INFO source=head.go:725 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=10.53µs
time=2025-04-23T16:55:47.223Z level=INFO source=head.go:733 msg="Replaying WAL, this may take a while" component=tsdb
time=2025-04-23T16:55:47.224Z level=INFO source=head.go:805 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0
time=2025-04-23T16:55:47.224Z level=INFO source=head.go:842 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=31.689µs wal_replay_duration=610.477µs wbl_replay_duration=160ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=10.53µs total_replay_duration=706.715µs
time=2025-04-23T16:55:47.226Z level=INFO source=main.go:1249 msg="filesystem information" fs_type=XFS_SUPER_MAGIC
time=2025-04-23T16:55:47.226Z level=INFO source=main.go:1252 msg="TSDB started"
...

If I start a fresh container, it also starts, despite the message:

[foo@bar ~]$ docker run --rm -it quay.io/prometheus/prometheus:v3.3.0 
time=2025-04-23T16:56:57.854Z level=INFO source=main.go:640 msg="No time or size retention was set so using the default time retention" duration=15d
time=2025-04-23T16:56:57.855Z level=INFO source=main.go:687 msg="Starting Prometheus Server" mode=server version="(version=3.3.0, branch=HEAD, revision=3c49e69330ceee9c587a1d994c78535695edff16)"
time=2025-04-23T16:56:57.855Z level=INFO source=main.go:692 msg="operational information" build_context="(go=go1.24.1, platform=linux/amd64, user=root@b362765c931d, date=20250415-13:32:00, tags=netgo,builtinassets,stringlabels)" host_details="(Linux 5.14.0-533.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 22 15:05:06 UTC 2024 x86_64 db4dcb330e07 (none))" fd_limits="(soft=1073741816, hard=1073741816)" vm_limits="(soft=unlimited, hard=unlimited)"
time=2025-04-23T16:56:57.857Z level=INFO source=main.go:768 msg="Leaving GOMAXPROCS=12: CPU quota undefined" component=automaxprocs
time=2025-04-23T16:56:57.862Z level=INFO source=web.go:654 msg="Start listening for connections" component=web address=0.0.0.0:9090
time=2025-04-23T16:56:57.864Z level=INFO source=main.go:1228 msg="Starting TSDB ..."
...

So, I am not sure why the existing container using same image, was showing Leaving GOMAXPROCS=12: CPU quota undefined as a last message, then exiting.

(I also tried setting --memory and --memory-swap, just like my production prometheus on that host, and -e GODEBUG=madvdontneed=1, to make it closer to the one that was failing, but still no difference).

I will try restarting the prod prometheus again later, to recheck logs, maybe strace and/or capture exit code. But I do not want to do that right now, as it takes about 15 minutes to fully do WAL reply.

[machine424]

I was not suggesting --no-auto-gomaxprocs as a workaround but rather to help identify the "hanging" code.

[krajorama]

Hello from the bug,

would you be able to dump the go routines when the process is hanging ?
Using for example:

curl http://localhost:9090/debug/pprof/goroutine?debug=1