Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] - Cardano-node fails to open prometheus ports, no errors #1100

Closed
Oqulent opened this issue May 29, 2020 · 6 comments
Closed

[BUG] - Cardano-node fails to open prometheus ports, no errors #1100

Oqulent opened this issue May 29, 2020 · 6 comments
Assignees
Labels
bug Something isn't working Pioneer Issues related to the Pioneer F&F Testnet priority low Issues/RPs that are low priority issues/PRs in relation to a minimum Shelley testnet and Shelley mai

Comments

@Oqulent
Copy link

Oqulent commented May 29, 2020

Summary
A clear and specific description of what the bug is.
Two cardano-node running on the same machine with different configuration files except EKG ports (which were left the same by mistake).

The first node reads the config files and opens EKG and prometheus ports. The second node reads the config file, sees a port conflict in EKG ports? and does not open a PROMETHEUS port. I could not find traces of any error in logs.

Steps to reproduce
Steps to reproduce the behavior:

  1. Run two+ instances of cardano-node
  2. Specify different prometheus ports for each instance
  3. Leave the same port for EKG on both nodes
  4. Specify two different jobs in prometheus scraping from both nodes.

Expected behavior
A clear and concise description of what you expected to happen.

  1. note the port conflict in EKG ports in logs.
  2. follow through with opening prometheus ports which had no conflict.

System info (please complete the following information):

  • OS: Ubuntu
  • Version 20.04
  • Node version 1.12.1

Screenshots and attachments

  • If applicable, add screenshots, config files and/or logs to help explain the problem.

Additional context
Add any other context about the problem here.

@Oqulent Oqulent added Pioneer Issues related to the Pioneer F&F Testnet bug Something isn't working labels May 29, 2020
@kevinhammond kevinhammond added priority high issues/PRs that MUST be addressed. The release can't happen without this; logging and monitoring labels Jun 1, 2020
@CodiePP
Copy link
Contributor

CodiePP commented Jun 2, 2020

you need to make a copy of the configuration file and change the ports of EKG and Prometheus in one of them, then run each of the nodes with their respective copy of the configuration.
it's possible to run two nodes on the same machine, they just have to use distinct ports and socket files, etc.

@Oqulent
Copy link
Author

Oqulent commented Jun 2, 2020

you need to make a copy of the configuration file and change the ports of EKG and Prometheus in one of them, then run each of the nodes with their respective copy of the configuration.
it's possible to run two nodes on the same machine, they just have to use distinct ports and socket files, etc.

I understand that ports should be different for the different nodes. the issue here is that the mistake in EKG ports (which were left the same) prevented the Prometheus ports from opening, with no traceable errors...

@dcoutts dcoutts added priority low Issues/RPs that are low priority issues/PRs in relation to a minimum Shelley testnet and Shelley mai and removed priority high issues/PRs that MUST be addressed. The release can't happen without this; labels Jun 3, 2020
@dcoutts
Copy link
Contributor

dcoutts commented Jun 3, 2020

Setting priority low since there is a clear workaround.

@vix-io
Copy link
Contributor

vix-io commented Jun 8, 2020

Hi Oqulent, this is Viktor at IOHK Tech Support. Thank you for reporting this bug. We have added it to our backlog and we`ll allocate it to one of our future development sprints. We appreciate your participation and support for the Cardano project!

@CodiePP
Copy link
Contributor

CodiePP commented Jul 2, 2020

tracked in CAD-1148

@denisshevchenko
Copy link
Contributor

If I start two nodes with the same EKG port in the configuration - I see this error message in the log:

[cardano.#ekgview.realizeFrom:Error:5] [2020-07-13 20:19:22.14 UTC] EKGView backend disabled due to initialisation error: EKGServerStartupError "Network.Socket.bind: resource busy (Address already in use)"

or (in JSON-log) this one:

{{{"at":"2020-07-13T20:19:22.14Z","env":"1.14.0:aef5a","ns":"cardano.#ekgview.realizeFrom","data":

{"kind":"LogError","message":"EKGView backend disabled due to initialisation error: EKGServerStartupError \"Network.Socket.bind: resource busy (Address already in use)\""}
,"app":[],"msg":"","pid":"2398","loc":null,"host":"","sev":"Error","thread":"5"}}}

@CodiePP CodiePP closed this as completed Jul 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Pioneer Issues related to the Pioneer F&F Testnet priority low Issues/RPs that are low priority issues/PRs in relation to a minimum Shelley testnet and Shelley mai
Projects
None yet
Development

No branches or pull requests

6 participants