Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentinel lookup connection leaks if no master address reported #1558

Closed
wwwjinlong opened this issue Dec 13, 2020 · 7 comments
Closed

Sentinel lookup connection leaks if no master address reported #1558

wwwjinlong opened this issue Dec 13, 2020 · 7 comments
Assignees
@mp911de
Labels
type: bug A general bug
Milestone
5.3.7

Comments

@wwwjinlong
Copy link

wwwjinlong commented Dec 13, 2020 •
edited by mp911de

Bug Report

Current Behavior

The collection to sentinel can't be closed when the master Id is error.

2020-12-13 19:34:47,090 [main] DEBUG io.lettuce.core.EpollProvider.<clinit>(70) -Starting without optional epoll library
2020-12-13 19:34:47,092 [main] DEBUG io.lettuce.core.KqueueProvider.<clinit>(72) -Starting without optional kqueue library
2020-12-13 19:34:47,094 [main] DEBUG io.lettuce.core.resource.DefaultEventLoopGroupProvider.allocate(85) -Allocating executor io.netty.channel.nio.NioEventLoopGroup
2020-12-13 19:34:47,094 [main] DEBUG io.lettuce.core.resource.DefaultEventLoopGroupProvider.createEventLoopGroup(196) -Creating executor io.netty.channel.nio.NioEventLoopGroup
2020-12-13 19:34:47,104 [main] DEBUG io.lettuce.core.resource.DefaultEventLoopGroupProvider.addReference(98) -Adding reference to io.netty.channel.nio.NioEventLoopGroup@32d8e58d, existing ref count 0
2020-12-13 19:34:47,120 [main] DEBUG io.lettuce.core.RedisClient.lambda$getSocketAddress$18(735) -Connecting to Redis using Sentinels [RedisURI [host='sentinel-26370-3.sinnet.hubiidc.com', port=26370]], MasterId sentinel-redis-test-6
2020-12-13 19:34:47,126 [main] DEBUG io.lettuce.core.RedisClient.connectSentinelAsync(535) -Trying to get a Redis Sentinel connection for one of: [RedisURI [host='sentinel-26370-3.sinnet.hubiidc.com', port=26370]]
2020-12-13 19:34:47,141 [main] DEBUG io.lettuce.core.RedisClient.doConnectSentinelAsync(601) -Connecting to Redis Sentinel, address: RedisURI [host='sentinel-26370-3.sinnet.hubiidc.com', port=26370]
2020-12-13 19:34:47,143 [main] DEBUG io.lettuce.core.RedisClient.lambda$getSocketAddressSupplier$19(775) -Resolved SocketAddress sentinel-26370-3.sinnet.hubiidc.com:26370 using RedisURI [host='sentinel-26370-3.sinnet.hubiidc.com', port=26370]
2020-12-13 19:34:47,143 [main] DEBUG io.lettuce.core.AbstractRedisClient.initializeChannelAsync0(312) -Connecting to Redis at sentinel-26370-3.sinnet.hubiidc.com:26370
2020-12-13 19:34:47,365 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO  org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest(552) -[Consumer clientId=consumer-hubi-woodpecker-consumer_CLEAN-1, groupId=hubi-woodpecker-consumer_CLEAN] (Re-)joining group
2020-12-13 19:34:47,540 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler.channelRegistered(181) -[channel=0x6757bd3d, [id: 0xf034cb72] (inactive), chid=0x1] channelRegistered()
2020-12-13 19:34:47,559 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler.channelActive(294) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, chid=0x1] channelActive()
2020-12-13 19:34:47,560 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.lambda$notifyChannelActive$0(421) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, epid=0x2] activateEndpointAndExecuteBufferedCommands 0 command(s) buffered
2020-12-13 19:34:47,560 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.lambda$notifyChannelActive$0(426) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, epid=0x2] activating endpoint
2020-12-13 19:34:47,560 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.flushCommands(508) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, epid=0x2] flushCommands()
2020-12-13 19:34:47,561 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.flushCommands(523) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, epid=0x2] flushCommands() Flushing 0 commands
2020-12-13 19:34:47,561 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.ConnectionWatchdog.channelActive(180) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, last known addr=sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370] channelActive()
2020-12-13 19:34:47,561 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler.channelActive(308) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, chid=0x1] channelActive() done
2020-12-13 19:34:47,562 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.AbstractRedisClient.lambda$null$4(359) -Connecting to Redis at sentinel-26370-3.sinnet.hubiidc.com:26370: Success
2020-12-13 19:34:55,801 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO  org.apache.kafka.clients.consumer.internals.AbstractCoordinator$2.onSuccess(503) -[Consumer clientId=consumer-hubi-woodpecker-consumer_CLEAN-1, groupId=hubi-woodpecker-consumer_CLEAN] Successfully joined group with generation 462
2020-12-13 19:34:55,814 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO  org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.invokePartitionsAssigned(273) -[Consumer clientId=consumer-hubi-woodpecker-consumer_CLEAN-1, groupId=hubi-woodpecker-consumer_CLEAN] Adding newly assigned partitions: WOODPECKER_ALARM_TOPIC-3
2020-12-13 19:34:55,840 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.RedisChannelHandler.dispatch(179) -dispatching command SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2020-12-13 19:34:55,844 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.channelWriteAndFlush(389) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, epid=0x2] write() writeAndFlush command SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2020-12-13 19:34:55,845 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler.write(374) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, chid=0x1] write(ctx, SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], promise)
2020-12-13 19:34:55,851 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandEncoder.encode(101) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370] writing command SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2020-12-13 19:34:55,856 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.write(173) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, epid=0x2] write() done
2020-12-13 19:34:55,857 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.ConnectionWatchdog.userEventTriggered(148) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, last known addr=sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370] userEventTriggered(ctx, io.lettuce.core.ConnectionEvents$Activated@3a6492da)
2020-12-13 19:34:55,923 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler.channelRead(570) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, chid=0x1] Received: 5 bytes, 1 commands in the stack
2020-12-13 19:34:55,923 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler.decode(616) -[channel=0x6757bd3d, /10.234.224.10:62140 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, chid=0x1] Stack contains: 1 commands
2020-12-13 19:34:55,923 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.RedisStateMachine.decode(131) -Decode SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2020-12-13 19:34:55,929 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.RedisStateMachine.decode(145) -Decoded SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], empty stack: true
2020-12-13 19:34:55,930 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.RedisChannelHandler.close(131) -close()
2020-12-13 19:34:55,930 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.RedisChannelHandler.closeAsync(145) -closeAsync()
2020-12-13 19:34:55,930 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.closeAsync(549) -[unknown, epid=0x1] closeAsync()

org.springframework.data.redis.connection.PoolException: Could not get a resource from the pool; nested exception is io.lettuce.core.RedisConnectionException: Cannot provide redisAddress using sentinel for masterId sentinel-redis-test-6
	at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:109)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1197)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1178)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedConnection(LettuceConnectionFactory.java:942)
	at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:353)
	at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:134)
	at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:97)
	at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:84)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:215)
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:188)
	at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:96)
	at org.springframework.data.redis.core.DefaultSetOperations.isMember(DefaultSetOperations.java:203)
	at com.hubi.woodpecker.consumer.app.config.RedisUtils.sHasKey(RedisUtils.java:53)
	at com.hubi.woodpecker.consumer.app.RedisTest.reids(RedisTest.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
	at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84)
	at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
	at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
	at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by: io.lettuce.core.RedisConnectionException: Cannot provide redisAddress using sentinel for masterId sentinel-redis-test-6
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:72)
	at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56)
	at io.lettuce.core.AbstractRedisClient.getConnection(AbstractRedisClient.java:242)
	at io.lettuce.core.RedisClient.connect(RedisClient.java:206)
	at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.lambda$getConnection$1(StandaloneConnectionProvider.java:115)
	at java.util.Optional.orElseGet(Optional.java:267)
	at org.springframework.data.redis.connection.lettuce.StandaloneConnectionProvider.getConnection(StandaloneConnectionProvider.java:115)
	at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.lambda$null$0(LettucePoolingConnectionProvider.java:97)
	at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:211)
	at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:201)
	at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:58)
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:899)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:429)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:354)
	at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:122)
	at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:117)
	at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:103)
	... 44 common frames omitted

Input Code

Input Code 
// your code here;

Expected behavior/code

The collection to sentinel with an exception should be closed.

Environment

  • Lettuce version(s): 5.3.5.RELEASE
  • Redis version: 5.0.8

Possible Solution

Additional context
@wwwjinlong wwwjinlong added the type: bug A general bug label Dec 13, 2020
@mp911de
Copy link
Collaborator

mp911de commented Dec 13, 2020

I'm not sure what you're asking for. Can you explain what you expect that should happen?

@wwwjinlong
Copy link
Author

wwwjinlong commented Dec 13, 2020 •
edited 

 private Mono<SocketAddress> lookupRedis(RedisURI sentinelUri) {

        Mono<StatefulRedisSentinelConnection<String, String>> connection = Mono
                .fromCompletionStage(() -> connectSentinelAsync(newStringStringCodec(), sentinelUri, timeout));

        return connection.flatMap(c -> {

            String sentinelMasterId = sentinelUri.getSentinelMasterId();
            return c.reactive().getMasterAddrByName(sentinelMasterId).map(it -> {

                if (it instanceof InetSocketAddress) {

                    InetSocketAddress isa = (InetSocketAddress) it;
                    SocketAddress resolved = clientResources.socketAddressResolver()
                            .resolve(RedisURI.create(isa.getHostString(), isa.getPort()));

                    logger.debug("Resolved Master {} SocketAddress {}:{} to {}", sentinelMasterId, isa.getHostString(),
                            isa.getPort(), resolved);

                    return resolved;
                }

                return it;
            }).timeout(this.timeout) //
                    .onErrorResume(e -> {

                        RedisCommandTimeoutException ex = ExceptionFactory
                                .createTimeoutException("Cannot obtain master using SENTINEL MASTER", timeout);
                        ex.addSuppressed(e);

                        return Mono.fromCompletionStage(c::closeAsync).then(Mono.error(ex));
                    }).flatMap(it -> Mono.fromCompletionStage(c::closeAsync) //
                            .thenReturn(it));
        });
    }

if “c.reactive().getMasterAddrByName(sentinelMasterId)” is empty，”c::closeAsync “ should be called。

@wwwjinlong
Copy link
Author

I'm not sure what you're asking for. Can you explain what you expect that should happen?

if “c.reactive().getMasterAddrByName(sentinelMasterId)” is empty，”c::closeAsync “ should be called。

@mp911de
Copy link
Collaborator

mp911de commented Dec 13, 2020

Technically, getMasterAddrByName(…) should never be empty. Either Redis reports the IP address or an error (master unknown, no master available).

I don't know why this happens, however, you're right, the connection remains open and causes a resource leak.

Can you capture a log on trace level to see the actual Redis response? Independent from that, we need to fix the issue.

@mp911de mp911de changed the title io.lettuce.core.protocol.DefaultEndpoint.closeAsync(549) -[unknown, epid=0x1] closeAsync() Sentinel lookup connection leaks if no master address reported Dec 13, 2020
@mp911de mp911de added this to the 5.3.6 milestone Dec 13, 2020
@wwwjinlong
Copy link
Author

wwwjinlong commented Dec 14, 2020 •
edited by mp911de

Technically, getMasterAddrByName(…) should never be empty. Either Redis reports the IP address or an error (master unknown, no master available).

I don't know why this happens, however, you're right, the connection remains open and causes a resource leak.

Can you capture a log on trace level to see the actual Redis response? Independent from that, we need to fix the issue.

The trace log is below.

2020-12-14 14:52:22,309 [lettuce-nioEventLoop-4-1] TRACE io.lettuce.core.RedisPublisher.subscribe(114) -subscribe: reactor.core.publisher.MonoNext$NextSubscriber@162981579
2020-12-14 14:52:22,320 [lettuce-nioEventLoop-4-1] TRACE io.lettuce.core.RedisPublisher$RedisSubscription.subscribe(227) -UNSUBSCRIBED subscribe: reactor.core.publisher.MonoNext$NextSubscriber@162981579
2020-12-14 14:52:22,321 [lettuce-nioEventLoop-4-1] TRACE io.lettuce.core.RedisPublisher$RedisSubscription.request(244) -NO_DEMAND request: 9223372036854775807
2020-12-14 14:52:22,323 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.RedisChannelHandler.dispatch(179) -dispatching command SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2020-12-14 14:52:22,325 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.channelWriteAndFlush(389) -[channel=0x344beefe, /10.234.224.201:54349 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, epid=0x2] write() writeAndFlush command SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2020-12-14 14:52:22,326 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler.write(374) -[channel=0x344beefe, /10.234.224.201:54349 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, chid=0x1] write(ctx, SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], promise)
2020-12-14 14:52:22,329 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO  org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.refreshCommittedOffsetsIfNeeded(799) -[Consumer clientId=consumer-huobi-woodpecker-consumer_CLEAN-1, groupId=huobi-woodpecker-consumer_CLEAN] Setting offset for partition WOODPECKER_ALARM_TOPIC-3 to the committed offset FetchPosition{offset=5380129, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[kafka2-test2.sinnet.hubiidc.com:9092 (id: 4186 rack: null)], epoch=absent}}
2020-12-14 14:52:22,330 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO  org.springframework.core.log.LogAccessor.info(279) -huobi-woodpecker-consumer_CLEAN: partitions assigned: [WOODPECKER_ALARM_TOPIC-3]
2020-12-14 14:52:22,333 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandEncoder.encode(101) -[channel=0x344beefe, /10.234.224.201:54349 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370] writing command SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2020-12-14 14:52:22,333 [lettuce-nioEventLoop-4-1] TRACE io.lettuce.core.protocol.CommandEncoder.encode(103) -[channel=0x344beefe, /10.234.224.201:54349 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370] Sent: *3
$8
SENTINEL
$23
get-master-addr-by-name
$20
sentinel-redis-test6
2020-12-14 14:52:22,337 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.write(173) -[channel=0x344beefe, /10.234.224.201:54349 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, epid=0x2] write() done
2020-12-14 14:52:22,338 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.ConnectionWatchdog.userEventTriggered(148) -[channel=0x344beefe, /10.234.224.201:54349 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, last known addr=sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370] userEventTriggered(ctx, io.lettuce.core.ConnectionEvents$Activated@22aadd84)
2020-12-14 14:52:22,345 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler.channelRead(570) -[channel=0x344beefe, /10.234.224.201:54349 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, chid=0x1] Received: 5 bytes, 1 commands in the stack
2020-12-14 14:52:22,345 [lettuce-nioEventLoop-4-1] TRACE io.lettuce.core.protocol.CommandHandler.channelRead(585) -[channel=0x344beefe, /10.234.224.201:54349 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, chid=0x1] Buffer: *-1
2020-12-14 14:52:22,345 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler.decode(616) -[channel=0x344beefe, /10.234.224.201:54349 -> sentinel-26370-3.sinnet.hubiidc.com/172.18.4.105:26370, chid=0x1] Stack contains: 1 commands
2020-12-14 14:52:22,345 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.RedisStateMachine.decode(131) -Decode SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2020-12-14 14:52:22,353 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.RedisStateMachine.decode(145) -Decoded SubscriptionCommand [type=SENTINEL, output=SocketAddressOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], empty stack: true
2020-12-14 14:52:22,353 [lettuce-nioEventLoop-4-1] TRACE io.lettuce.core.RedisPublisher$RedisSubscription.onAllDataRead(336) -DEMAND onAllDataRead()
2020-12-14 14:52:22,353 [lettuce-nioEventLoop-4-1] TRACE io.lettuce.core.RedisPublisher$RedisSubscription.onDataAvailable(322) -DEMAND onDataAvailable()
2020-12-14 14:52:22,354 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.RedisChannelHandler.close(131) -close()
2020-12-14 14:52:24,112 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.RedisChannelHandler.closeAsync(145) -closeAsync()
2020-12-14 14:52:24,112 [lettuce-nioEventLoop-4-1] DEBUG io.lettuce.core.protocol.DefaultEndpoint.closeAsync(549) -[unknown, epid=0x1] closeAsync()

@mp911de mp911de modified the milestones: 5.3.6, 5.3.7 Jan 12, 2021
@mp911de
Copy link
Collaborator

mp911de commented Jan 14, 2021

Is there a way how to test that? I wasn't able to reproduce an empty response. However, going to fix the issue.

@mp911de mp911de self-assigned this Jan 14, 2021
mp911de added a commit that referenced this issue Jan 14, 2021
@mp911de
Use Mono.usingWhen(…) to avoid connection leaks on master lookup thro…
c3337f5 
…ugh Sentinel #1558

Previously, we used our own hooks to open and close the lookup connection that could leave connections open when no master node was found.
mp911de added a commit that referenced this issue Jan 14, 2021
@mp911de
Use Mono.usingWhen(…) to avoid connection leaks on master lookup thro…
767831e 
…ugh Sentinel #1558

Previously, we used our own hooks to open and close the lookup connection that could leave connections open when no master node was found.
mp911de added a commit that referenced this issue Jan 14, 2021
@mp911de
Use Mono.usingWhen(…) to avoid connection leaks on master lookup thro…
232a232 
…ugh Sentinel #1558

Previously, we used our own hooks to open and close the lookup connection that could leave connections open when no master node was found.
@mp911de
Copy link
Collaborator

mp911de commented Jan 14, 2021

That's fixed now.

@mp911de mp911de closed this as completed Jan 14, 2021
This was referenced Mar 16, 2021
Closed
Closed
Merged
Merged
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mp911de mp911de

Labels
type: bug A general bug
Projects
None yet
Milestone
5.3.7
Development

No branches or pull requests
2 participants
@mp911de @wwwjinlong