Skip to content

JedisConnection.close() fails to release connection to pool if subscription cleanup fails #2356

@yu-shiba

Description

@yu-shiba

Hi,
After upgrading to 2.7.1, the application associated with RedisMessageListenerContainer became unresponsive.
The problem was caused by a misconfiguration in our environment where Istio disconnects the connection after 1 hour. When investigating this, we discovered that Jedis leaks when an exception occurs during JedisConnection close.

The following is a stack trace of the problem.
The first exception is a disconnection and JedisConnection#close() is executed, but the second exception occurs at subscription.close() and jedis.close() is not executed.

1st exception

org.springframework.data.redis.RedisConnectionFailureException: Unexpected end of stream.; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:65)
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:42)
	at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
	at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:187)
	at org.springframework.data.redis.connection.jedis.JedisConnection.doWithJedis(JedisConnection.java:820)
	at org.springframework.data.redis.connection.jedis.JedisConnection.pSubscribe(JedisConnection.java:725)
	at org.springframework.data.redis.listener.RedisMessageListenerContainer$Subscriber.doSubscribe(RedisMessageListenerContainer.java:1231)
	at org.springframework.data.redis.listener.RedisMessageListenerContainer$BlockingSubscriber.lambda$eventuallyPerformSubscription$2(RedisMessageListenerContainer.java:1433)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.
	at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:202)
	at redis.clients.jedis.util.RedisInputStream.readByte(RedisInputStream.java:43)
	at redis.clients.jedis.Protocol.process(Protocol.java:165)
	at redis.clients.jedis.Protocol.read(Protocol.java:230)
	at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:352)
	at redis.clients.jedis.Connection.getUnflushedObjectMultiBulkReply(Connection.java:314)
	at redis.clients.jedis.BinaryJedisPubSub.process(BinaryJedisPubSub.java:101)
	at redis.clients.jedis.BinaryJedisPubSub.proceedWithPatterns(BinaryJedisPubSub.java:89)
	at redis.clients.jedis.BinaryJedis.psubscribe(BinaryJedis.java:3900)
	at org.springframework.data.redis.connection.jedis.JedisConnection.lambda$pSubscribe$7(JedisConnection.java:730)
	at org.springframework.data.redis.connection.jedis.JedisConnection.doWithJedis(JedisConnection.java:818)
	... 4 common frames omitted

2nd exception

redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe
	at redis.clients.jedis.Connection.flush(Connection.java:342)
	at redis.clients.jedis.BinaryJedisPubSub.punsubscribe(BinaryJedisPubSub.java:63)
	at org.springframework.data.redis.connection.jedis.JedisSubscription.doClose(JedisSubscription.java:49)
	at org.springframework.data.redis.connection.util.AbstractSubscription.close(AbstractSubscription.java:106)
	at org.springframework.data.redis.connection.jedis.JedisConnection.close(JedisConnection.java:336)
	at org.springframework.data.redis.listener.RedisMessageListenerContainer$Subscriber.closeConnection(RedisMessageListenerContainer.java:1323)
	at org.springframework.data.redis.listener.RedisMessageListenerContainer.handleSubscriptionException(RedisMessageListenerContainer.java:893)
	at org.springframework.data.redis.listener.RedisMessageListenerContainer$BlockingSubscriber.lambda$eventuallyPerformSubscription$2(RedisMessageListenerContainer.java:1437)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Broken pipe
	at java.base/sun.nio.ch.NioSocketImpl.implWrite(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl.write(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl$2.write(Unknown Source)
	at java.base/java.net.Socket$SocketOutputStream.write(Unknown Source)
	at redis.clients.jedis.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52)
	at redis.clients.jedis.util.RedisOutputStream.flush(RedisOutputStream.java:133)
	at redis.clients.jedis.Connection.flush(Connection.java:339)
	... 8 common frames omitted

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions