Reduce the log level of the message sent when disconnected from sentinel discovery service

[zumo64]

In case of scheduled maintenance of Redis Enterprise and Sentinel Discovery Service is temporary down, Jedis Client reports ERROR level messages which triggers internal alerts and causes too much noise in some Production deployments
Could we downgrade this particular message to a lower lever ?

[ggivo]

@zumo64
Could you elaborate a bit?
e.g, Jedis version, example Client configuration? Example messages that are reported?
Steps to reproduce it.

[uglide]

@ggivo see my PR

[ggivo]

@uglide
WARN makes sense since it is an expected and recoverable error. Updated also the UnifiedJedis logs.

[zumo64]

@ggivo Hi

below is the full stack trace:
I am trying get more details from the client and will share
Thanks

Caused by: fr.ca.cat.sdj.common.storage.exception.StorageTechnicalErrorException: redis.clients.jedis.exceptions.JedisConnectionException: Failed to connect to any host resolved for DNS name."," fr.ca.cat.sdj.common.storage.module.legacy.DataStore.store(DataStore.java:118)","time=2024-03-25T12:47:33.485+0100|level=ERROR|env=u1|event_cod=|event_typ=|app_id=ress-PartageContexte|app_version=v1.4.0|component_id=redis.clients.jedis.JedisSentinelPool|thread_id=|req_id=|uom_cod=|corr_id=|usr_id=|message=Lost connection to Sentinel at 10.77.213.66:8001. Sleeping 5000ms and retrying.|exception= redis.clients.jedis.exceptions.JedisConnectionException: Failed to create socket.
 at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:110)
 at redis.clients.jedis.Connection.connect(Connection.java:226)
 at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:144)
 at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:314)
 at redis.clients.jedis.BinaryJedis.initializeFromClientConfig(BinaryJedis.java:92)
 at redis.clients.jedis.BinaryJedis.(http://binaryjedis.java:87/)
 at redis.clients.jedis.Jedis.(http://jedis.java:56/)
 at redis.clients.jedis.JedisSentinelPool$MasterListener.run(JedisSentinelPool.java:391)
Caused by: java.net.ConnectException: Connection refused
 at java.base/sun.nio.ch.Net.pollConnect(Native Method)
 at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
 at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source)
 at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
 at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
 at java.base/java.net.Socket.connect(Unknown Source)
 at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:80)
 ... 7 more

[ggivo]

@zumo64
Thanks for reporting it.
Downgrading the message to WARN makes sense to me, but I’m not sure how that would address the root cause of the alerts.

If I understand correctly, the alerts are triggered based on specific log messages. Even with a lower log level, the alerting system still needs to distinguish between an actual Sentinel issue (e.g., a node is down) and a temporary condition (maintenance or a brief network glitch).

It might make more sense to handle this logic in the alerting system — for example, triggering an alert only if the message persists for a certain period of time.