-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CI: ci-external-workload: Cilium API handler panicked #19425
Comments
Marking as a release blocker because the |
I'm not sure if this is a flake, it's breaking #19159 CI for me on every run. It's causing so much noise that I can't see actual failures anymore. This took a while for me to narrow this down to I think 6c5e2d6 is the culprit. If I dump var
As suggested in golang/go#25721 (comment), this might require implementing a custom codec for |
I don't believe 6c5e2d6 is the cause, but the recent bump to prometheus/client_golang is. See: d8e3f28. This bug has been reported upstream, and it seems they reverted adding the new metrics: |
I suggest reverting the update to |
Fix proposed at: #19496. |
This reverts commit d8e3f28. --- See cilium#19425. Extra Go metrics were added to the Prometheus client, some having NaN values, breaking json marshaling on the server side of `cilium metrics list`: msg="Cilium API handler panicked" panic_message="json: unsupported value: NaN" url=/v1/metrics/ ... Signed-off-by: Timo Beckers <timo@isovalent.com>
This reverts commit d8e3f28. --- See #19425. Extra Go metrics were added to the Prometheus client, some having NaN values, breaking json marshaling on the server side of `cilium metrics list`: msg="Cilium API handler panicked" panic_message="json: unsupported value: NaN" url=/v1/metrics/ ... Signed-off-by: Timo Beckers <timo@isovalent.com>
This issue was reintroduced by f321a6a, which upgraded client_golang to 1.12.1 again. 1.12.2 is out, which supposedly fixes this. |
CI failure
Seen in one of the agent logs attached to the zip in here:
https://github.com/cilium/cilium/actions/runs/2158036465
cilium-sysdump-out.zip (23).zip
The text was updated successfully, but these errors were encountered: