No prometheus stats on v5.5

[kaizensparc]

What happened?

When upgrading from v5.4.0 to v5.5.0, the prometheus stats endpoint (/api/v5/prometheus/stats) reported INTERNAL_ERROR

What did you expect to happen?

To have prometheus stats endpoint report the prometheus metrics

How can we reproduce it (as minimally and precisely as possible)?

curl <endpoint>:<port>/api/v5/prometheus/stats on a v5.5.0 emqx node

Anything else we need to know?

Stacktrace is:

{"time":1709051828261886,"level":"warning","mfa":"minirest_handler:apply_callback/3(142)","stacktrace":["{emqx_prometheus,cert_expiry_at_from_path,1,[{file,\"emqx_prometheus.erl\"},{line,886}]}","{emqx_prometheus,gen_point,3,[{file,\"emqx_prometheus.erl\"},{line,881}]}","{emqx_prometheus,'-do_points_of_listeners/2-fun-0-',5,[{file,\"emqx_prometheus.erl\"},{line,869}]}","{lists,foldl,3,[{file,\"lists.erl\"},{line,1350}]}","{emqx_prometheus,'-do_points_of_listeners/2-fun-1-',4,[{file,\"emqx_prometheus.erl\"},{line,861}]}","{lists,foldl,3,[{file,\"lists.erl\"},{line,1350}]}","{emqx_prometheus,'-cert_data/1-fun-0-',3,[{file,\"emqx_prometheus.erl\"},{line,834}]}","{lists,foldl_1,3,[{file,\"lists.erl\"},{line,1355}]}","{emqx_prometheus,cert_data,1,[{file,\"emqx_prometheus.erl\"},{line,831}]}","{emqx_prometheus,fetch_cluster_consistented_data,0,[{file,\"emqx_prometheus.erl\"},{line,273}]}","{emqx_prometheus_cluster,raw_data,2,[{file,\"emqx_prometheus_cluster.erl\"},{line,59}]}","{emqx_prometheus,collect_mf,2,[{file,\"emqx_prometheus.erl\"},{line,193}]}","{prometheus_collector,collect_mf,3,[{file,\"prometheus_collector.erl\"},{line,173}]}","{prometheus_registry,'-collect/2-lc$^0/1-0-',3,[{file,\"prometheus_registry.erl\"},{line,86}]}","{prometheus_registry,collect,2,[{file,\"prometheus_registry.erl\"},{line,87}]}","{prometheus_text_format,format,1,[{file,\"prometheus_text_format.erl\"},{line,74}]}","{emqx_prometheus_api,collect,2,[{file,\"emqx_prometheus_api.erl\"},{line,207}]}","{minirest_handler,apply_callback,3,[{file,\"minirest_handler.erl\"},{line,137}]}","{minirest_handler,handle,2,[{file,\"minirest_handler.erl\"},{line,56}]}","{minirest_handler,init,2,[{file,\"minirest_handler.erl\"},{line,27}]}","{cowboy_handler,execute,2,[{file,\"cowboy_handler.erl\"},{line,41}]}","{cowboy_stream_h,execute,3,[{file,\"cowboy_stream_h.erl\"},{line,318}]}","{cowboy_stream_h,request_process,3,[{file,\"cowboy_stream_h.erl\"},{line,302}]}"],"reason":"{badmatch,{error,enoent}}","pid":"<0.3214.0>","path":"/prometheus/stats","exception":"error"}

EMQX version

$ ./bin/emqx_ctl broker
sysdescr  : EMQX
version   : 5.5.0
datetime  : 2024-02-27T16:58:15.966782284+00:00
uptime    : 39 seconds

OS version

Docker container

Log files

/usr/local/bin/node_dump: 1: eval: netstat: not found

[zmstone]

@kaizensparc thank you for reporting this.
we'll fix it in 5.5.1

[JimMoen]

Thanks for your reporting.

The direct cause of this problem is that the path to a certain certificate file cannot be read (the file does not exist, enoent)
Do you have any detailed information about the certificate paths for ssl,wss or quic listeners?
This can help with fixing faster.
Or: Is there a Disabled listener with an unreachable cert path?

And In the fix: #12606
The prom data of emqx_cert_expiry_at will be 0 for non-existed cert.

[kaizensparc]

Hi!
Thanks for the fix. The wss and quic listeners are disabled, so the cert paths are indeed unreachable. The ssl listener is working so it can access the certificates.

[JimMoen]

The disabled listeners will be skipped now.
And prom data of emqx_cert_expiry_at will be 0 for non-existed cert file (if the listener enabled).

PR merged and plz wait for release v5.5.1.