Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shutdown hanging, somehow prometheus related #936

Closed
shufps opened this issue Apr 25, 2024 · 0 comments · Fixed by #947 or #946
Closed

shutdown hanging, somehow prometheus related #936

shufps opened this issue Apr 25, 2024 · 0 comments · Fixed by #947 or #946
Labels
team-node Issues for Node Team
Milestone

Comments

@shufps
Copy link
Contributor

shufps commented Apr 25, 2024

(happened with iota-core alpha.18)

We monitor go metrics like heap and noticed that on some nodes prometheus is not accessible anymore, trying to curl the metrics leads to a timeout.

As expected, in this cases the nodes cannot be shutdown gracefully.

Node was healthy + network-healthy and other things like the info endpoint were working.

Here is a profile of iota-core while it is waiting for Close p2p peer database, P2P, Protocol.
pprof.validator-11_20240425-125923_all.zip

@alexsporn alexsporn added the team-consensus Issues for Consensus Team label Apr 25, 2024
@muXxer muXxer added this to the v1.0.0-beta milestone Apr 25, 2024
@alexsporn alexsporn added team-node Issues for Node Team and removed team-consensus Issues for Consensus Team labels Apr 29, 2024
This was linked to pull requests Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team-node Issues for Node Team
Projects
Status: Done
3 participants