Monitoring instance logs not showing any connectivity errors if instance cannot connect to broker

[ramonsmits]

Describe the bug

Description

Monitoring instance logs not showing any connectivity errors if instance cannot connect to broker

For example, by default ASBS connects using AMQP ports but some environments have these blocked.

Expected behavior

Monitoring instance to fail to start when broker is not available or at mimimum report ERROR or FATAL log events indicating broker is not available

Actual behavior

Monitoring instance remains running without indicating failure but in the Azure Portal it shows that the instance is not ingesting messages from the monitoring queue.

Versions

6.6.3 with ASB (other transports might also be affected)

Steps to reproduce

1. Setup monitoring instance with ASB
2. Block ports:

• sudo iptables -A OUTPUT -p tcp -d 57.153.239.59 --dport 5671 -j REJECT
  sudo iptables -A OUTPUT -p tcp -d 57.153.239.59 --dport 5672 -j REJECT
  

3. Launch instance

Observe it is running without reporting failures.

Relevant log output

Additional Information

No response

[danielmarbach]

We've reviewed this bug and added it to our backlog. It is not currently prioritized, but it may be fixed along with other future work in the repository. We will update this issue when work begins.

[nodranael]

Just had the case when a monitoring instance (v6.7.2) just stopped processing messages without logging any issue. The instance keeps running and I had to restart the deployment manually for it to resume processing.

I had the same behaviour with version 4 and we ended up setting an alert to monitor the "particular.monitoring" queue size and restart the instance manually when messages were piling up. It's a bit annoying having to manually intervene in such cases.

It would be nice if the service would resume processing by itself in case of a transient issue, or crash "properly" if it's not able to do so, so that's it restarted by the infrastructure.

Running in AKS
Transport : Azure Service Bus