[BUG] - cardano_node_BlockFetchDecision_peers_connectedPeers_int not reported if seveity not Info

[ThaiTheo]

Internal/External
External

Summary
When in the mainnet-config.json the notication level is set higher than Info, the counter cardano_node_BlockFetchDecision_peers_connectedPeers_int is no longer reported.

Steps to reproduce
Steps to reproduce the behavior:

1. Monitor cardano_node_BlockFetchDecision_peers_connectedPeers_int through Prometheus or Grafana
2. Set FetchBlockPeerDecisions in mainnet-config.json to true and start cardano-node in basic setup
3. cardano_node_BlockFetchDecision_peers_connectedPeers_int will show values
4. Stop cardano-node and change minSeverity in mainnet-config.json from Info to Warning
5. Start cardano-node
6. cardano_node_BlockFetchDecision_peers_connectedPeers_int will not show values

Expected behavior
When in mainnet-config.json FetchBlockPeerDecisions is set to true, cardano_node_BlockFetchDecision_peers_connectedPeers_int will always show values.

System info (please complete the following information):

• OS: CentOS
• Version: 8
• Node version: 1.18.0

Screenshots and attachments

• Screenshot shows behavior of 1relay1 after changing severity level from Info to Warning
  

Additional context
The counter cardano_node_BlockFetchDecision_peers_connectedPeers_int is important to determine if a node is isolated from the block-chain. A block-producing node should have 1 peer at least (relay). A relay needs 2 peers at least (block-producing node and a relay).
The severity level should determine what gets logged, not which counters are reported or not.

[stanfieldr]

Same problem from a different angle. I'm using the FileSK scribe and peers do not show up in LiveView

[kevinhammond]

There is ongoing work to improve the logging and monitoring system. This may resolve several of the issues that have cropped up.

[dcoutts]

I think the request here is that this tracer be assigned a "notice" level of severity rather than "info". If so, that should be a trivial one-line change.

[Jimbo4350]

@dcoutts I've reproduced this, however Notice is less severe than Warning so it still won't show up in Prometheus as we filter by the minimum Severity ( @ThaiTheo set the minimum Severity to Warning) . If we want to log things regardless of what we set the minimum Severity at, we need to add another Severity level that trumps all others and perhaps allow the users to configure specific overrides via the config yaml file.
cc: @CodiePP

[denisshevchenko]

FYI, currently RTView doesn't use connectedPeers metric because it has a delay (I don't know why), when the peer is already connected but this value is still 0. Instead, RTView receives peersFromNodeKernel metric with the list of information about peers. And the length of this list is used as a number of connected peers. This value changes immediately.

[Jimbo4350]

Closing this. If this is still relevant please re-open.