Queries failing after upgrade to v2.52.0rc0

[swiffer]

What did you do?

Upgrade to v2.52.0rc0

What did you expect to see?

Everything continues working just fine

What did you see instead? Under which circumstances?

Several queries failing (e.g. on Node Exporter Full Dashboard):

Status: 200. Message: ReadArray: expect [ or , or ] or n, but found {, error found in #10 byte of ...|80116"]]}{"metric":{|..., bigger context ...|923262980116"],[1714483080,"47.15923262980116"]]}{"metric":{"device":"/dev/nvme1n1p1","fstype":"vfat|...

Query:

100 - ((node_filesystem_avail_bytes{instance="localhost:9100",job="node_exporter",device!'rootfs'} * 100) / node_filesystem_size_bytes{instance="localhost:9100",job="node_exporter",device!'rootfs'})

System information

Ubuntu 24.04 LTS (GNU/Linux 6.8.0-31-generic x86_64)

Prometheus version

prometheus, version 2.52.0-rc.0 (branch: HEAD, revision: f170a014f31cc97d2aba47cc769e846c68f8545d)
  build user:       root@969a4b73703c
  build date:       20240430-12:21:35
  go version:       go1.22.2
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[ArthurSens]

Thanks for trying the release candidate so quickly!

Could you please any additional information that could help us debug the problem, e.g. logs from the Prometheus instance that is failing

[swiffer]

Sure, this is what I can see currently in the prometheus log file

ts=2024-04-30T14:09:19.769Z caller=api.go:1769 level=error component=web msg="error writing response" bytesWritten=0 err="write tcp 127.0.0.1:9090->127.0.0.1:43910: write: broken pipe"
ts=2024-04-30T14:09:19.778Z caller=api.go:1769 level=error component=web msg="error writing response" bytesWritten=0 err="write tcp 127.0.0.1:9090->127.0.0.1:43912: write: broken pipe"
ts=2024-04-30T14:09:40.963Z caller=api.go:1769 level=error component=web msg="error writing response" bytesWritten=0 err="write tcp 127.0.0.1:9090->127.0.0.1:37742: write: broken pipe"
ts=2024-04-30T14:09:40.972Z caller=api.go:1769 level=error component=web msg="error writing response" bytesWritten=0 err="write tcp 127.0.0.1:9090->127.0.0.1:37752: write: broken pipe"
ts=2024-04-30T14:09:49.962Z caller=api.go:1769 level=error component=web msg="error writing response" bytesWritten=0 err="write tcp 127.0.0.1:9090->127.0.0.1:43922: write: connection reset by peer"
ts=2024-04-30T14:09:49.980Z caller=api.go:1769 level=error component=web msg="error writing response" bytesWritten=0 err="write tcp 127.0.0.1:9090->127.0.0.1:37768: write: connection reset by peer"

(querying prometheus via grafana)

[GiedriusS]

Is it the same if you send the same query through the Prometheus UI/API? If not, could you paste the error from the HTTP response?

[baryluk]

Looks like same bug as this:

#14040

Prometheus is returning malformed JSON

[swiffer]

fixed in https://github.com/prometheus/prometheus/releases/tag/v2.52.0-rc.1