Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lettuce suppresses initial exception in Redis Cluster #427

Closed
maxd opened this issue Dec 16, 2016 · 6 comments
Closed

Lettuce suppresses initial exception in Redis Cluster #427

maxd opened this issue Dec 16, 2016 · 6 comments
Labels
type: bug A general bug
Milestone
Lettuce 4.4.0

Comments

@maxd
Copy link

maxd commented Dec 16, 2016

For example I set invalid password and want to connect to Redis Cluster. In this case Lettuce suppress initial exception:

com.lambdaworks.redis.RedisCommandExecutionException: ERR Client sent AUTH, but no password is set
  at com.lambdaworks.redis.LettuceFutures.await(LettuceFutures.java:122)
  at com.lambdaworks.redis.LettuceFutures.awaitOrCancel(LettuceFutures.java:92)
  at com.lambdaworks.redis.AbstractRedisAsyncCommands.auth(AbstractRedisAsyncCommands.java:78)
  at com.lambdaworks.redis.cluster.RedisClusterClient.connectStateful(RedisClusterClient.java:640)
  at com.lambdaworks.redis.cluster.RedisClusterClient.connectToNode(RedisClusterClient.java:485)
  at com.lambdaworks.redis.cluster.RedisClusterClient$NodeConnectionFactoryImpl.connectToNode(RedisClusterClient.java:993)
  at com.lambdaworks.redis.cluster.topology.ClusterTopologyRefresh.getConnections(ClusterTopologyRefresh.java:204)
  at com.lambdaworks.redis.cluster.topology.ClusterTopologyRefresh.loadViews(ClusterTopologyRefresh.java:67)
  at com.lambdaworks.redis.cluster.RedisClusterClient.loadPartitions(RedisClusterClient.java:757)
  at com.lambdaworks.redis.cluster.RedisClusterClient.initializePartitions(RedisClusterClient.java:733)
  at com.lambdaworks.redis.cluster.RedisClusterClient.connectClusterImpl(RedisClusterClient.java:507)
  at com.lambdaworks.redis.cluster.RedisClusterClient.connect(RedisClusterClient.java:350)

and raise new exception here without any information about initial exception.

How do you think is it possible to pass initial exception in getCause() method of new exception?

Lettuce 4.3.0.Final
@mp911de mp911de added the type: bug A general bug label Dec 16, 2016
@mp911de
Copy link
Collaborator

mp911de commented Dec 16, 2016

The only way to get a cause is by initially throwing an exception from ClusterTopologyRefresh. There are various error cases, beginning with connection errors up to timeouts. The mechanism is resilient so lettuce can work with at least one node, even if the other nodes fail.

The main question is: When to throw an exception?

  1. Throwing an exception if at least one node fails kills resiliency and lettuce is no longer fail-safe
  2. Throwing an exception if all nodes fail seems the most appealing approach right now although it's not consistent if multiple nodes fail

What do you think?

The exception needs to be handled in the background refresh job (ClusterTopologyRefreshTask.run).

@maxd
Copy link
Author

maxd commented Dec 16, 2016 •
edited

In my case I want to show human readable message about problem if my application couldn't establish initial connection to Redis Cluster at all (after call of RedisClusterClient#connect method). If it can connect to one seed node from N it is ok because all works as expected (connect method establish connection to Redis Cluster). Maybe in the future my application loose connection to that one node too. In this case the resilient mechanism should start working as it works now.

So, as you can see I don't want to change current behavior of lettuce. I just want to see initial exception from all seed nodes or from last seed node only if the lettuce couldn't establish initial connection to Redis Cluster.

@mp911de mp911de added this to the Lettuce 4.4.0 milestone Dec 16, 2016
mp911de added a commit that referenced this issue Dec 18, 2016
@mp911de
Propagate cause if cluster topology refresh can't connect to any node #…
bc05815 
…427

RedisClusterClient.loadPartitions now propagates the cause if lettuce can't connect to any of the topology refresh sources (initial seed nodes). Cause propagation covers I/O errors and authentication issues. Command execution issues are just logged.
mp911de added a commit that referenced this issue Dec 18, 2016
@mp911de
Propagate cause if cluster topology refresh can't connect to any node #…
642b403 
…427

RedisClusterClient.loadPartitions now propagates the cause if lettuce can't connect to any of the topology refresh sources (initial seed nodes). Cause propagation covers I/O errors and authentication issues. Command execution issues are just logged.
@mp911de
Copy link
Collaborator

mp911de commented Dec 18, 2016

That's implemented now. A possible stack trace can look like:

com.lambdaworks.redis.RedisException: Cannot retrieve initial cluster partitions from initial URIs [RedisURI [host='localhost', port=7579], RedisURI [host='localhost', port=7580], RedisURI [host='localhost', port=7581]]
	at com.lambdaworks.redis.cluster.RedisClusterClient.loadPartitions(RedisClusterClient.java:841)
	at com.lambdaworks.redis.cluster.RedisClusterClient.initializePartitions(RedisClusterClient.java:794)
	at com.lambdaworks.redis.cluster.RedisClusterClient.connectClusterImpl(RedisClusterClient.java:555)
	at com.lambdaworks.redis.cluster.RedisClusterClient.connect(RedisClusterClient.java:353)
	at com.lambdaworks.redis.cluster.RedisClusterClient.connect(RedisClusterClient.java:330)
	at com.lambdaworks.redis.cluster.ClusterPartiallyDownTest.seedNodesAreOffline(ClusterPartiallyDownTest.java:105)
Caused by: com.lambdaworks.redis.RedisConnectionException: Unable to establish a connection to Redis Cluster
	at com.lambdaworks.redis.cluster.topology.AsyncConnections.get(AsyncConnections.java:84)
	at com.lambdaworks.redis.cluster.topology.ClusterTopologyRefresh.loadViews(ClusterTopologyRefresh.java:72)
	at com.lambdaworks.redis.cluster.RedisClusterClient.loadPartitions(RedisClusterClient.java:820)
	... 30 more
	Suppressed: com.lambdaworks.redis.RedisConnectionException: Unable to connect to localhost/127.0.0.1:7580
		at com.lambdaworks.redis.cluster.topology.ClusterTopologyRefresh.lambda$getConnections$355(ClusterTopologyRefresh.java:230)
		at com.lambdaworks.redis.cluster.topology.ClusterTopologyRefresh$$Lambda$8/1705277839.accept(Unknown Source)
		at java.util.concurrent.CompletableFuture$WhenCompleteCompletion.run(CompletableFuture.java:1298)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
		at java.util.concurrent.CompletableFuture$WhenCompleteCompletion.run(CompletableFuture.java:1303)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
		at java.util.concurrent.CompletableFuture$ThenApply.run(CompletableFuture.java:723)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
		at java.util.concurrent.CompletableFuture$ThenApply.run(CompletableFuture.java:723)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2361)
		at com.lambdaworks.redis.AbstractRedisClient.lambda$initializeChannelAsync$339(AbstractRedisClient.java:300)
		at com.lambdaworks.redis.AbstractRedisClient$$Lambda$4/767511741.operationComplete(Unknown Source)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
		at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:574)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:488)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
		at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
		at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
		at java.lang.Thread.run(Thread.java:745)
	Caused by: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:7580
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:205)
		... 19 more
	Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:7580
		at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
		at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
		at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:346)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
		... 7 more
	Suppressed: com.lambdaworks.redis.RedisConnectionException: Unable to connect to localhost/127.0.0.1:7579
		at com.lambdaworks.redis.cluster.topology.ClusterTopologyRefresh.lambda$getConnections$355(ClusterTopologyRefresh.java:230)
		at com.lambdaworks.redis.cluster.topology.ClusterTopologyRefresh$$Lambda$8/1705277839.accept(Unknown Source)
		at java.util.concurrent.CompletableFuture$WhenCompleteCompletion.run(CompletableFuture.java:1298)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
		at java.util.concurrent.CompletableFuture$WhenCompleteCompletion.run(CompletableFuture.java:1303)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
		at java.util.concurrent.CompletableFuture$ThenApply.run(CompletableFuture.java:723)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
		at java.util.concurrent.CompletableFuture$ThenApply.run(CompletableFuture.java:723)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2361)
		at com.lambdaworks.redis.AbstractRedisClient.lambda$initializeChannelAsync$339(AbstractRedisClient.java:300)
		at com.lambdaworks.redis.AbstractRedisClient$$Lambda$4/767511741.operationComplete(Unknown Source)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
		at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:574)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:488)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
		at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
		at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
		at java.lang.Thread.run(Thread.java:745)
	Caused by: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:7579
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:205)
		... 19 more
	Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:7579
		at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
		at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
		at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:346)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
		... 7 more
	Suppressed: com.lambdaworks.redis.RedisConnectionException: Unable to connect to localhost/127.0.0.1:7581
		at com.lambdaworks.redis.cluster.topology.ClusterTopologyRefresh.lambda$getConnections$355(ClusterTopologyRefresh.java:230)
		at com.lambdaworks.redis.cluster.topology.ClusterTopologyRefresh$$Lambda$8/1705277839.accept(Unknown Source)
		at java.util.concurrent.CompletableFuture$WhenCompleteCompletion.run(CompletableFuture.java:1298)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
		at java.util.concurrent.CompletableFuture$WhenCompleteCompletion.run(CompletableFuture.java:1303)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
		at java.util.concurrent.CompletableFuture$ThenApply.run(CompletableFuture.java:723)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:210)
		at java.util.concurrent.CompletableFuture$ThenApply.run(CompletableFuture.java:723)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:193)
		at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2361)
		at com.lambdaworks.redis.AbstractRedisClient.lambda$initializeChannelAsync$339(AbstractRedisClient.java:300)
		at com.lambdaworks.redis.AbstractRedisClient$$Lambda$4/767511741.operationComplete(Unknown Source)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
		at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:129)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:639)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:574)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:488)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
		at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
		at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
		at java.lang.Thread.run(Thread.java:745)
	Caused by: java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:7581
		at java.util.concurrent.CompletableFuture.internalComplete(CompletableFuture.java:205)
		... 19 more
	Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:7581
		at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
		at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
		at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:346)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
		... 7 more

@mp911de mp911de closed this as completed Dec 18, 2016
@maxd
Copy link
Author

maxd commented Dec 19, 2016

Excellent! :) I see all suppressed exceptions too. Thank you!

@mp911de mp911de changed the title Lettuce suppress initial exception in Redis Cluster Lettuce suppresses initial exception in Redis Cluster May 30, 2017
@hackpros
Copy link

spring.redis.timeout: 1000

@mp911de
Copy link
Collaborator

mp911de commented Apr 27, 2018

Not exactly sure what you're trying to express @hackpros. Care to elaborate, possibly in a new ticket?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: bug A general bug
Projects
None yet
Milestone
Lettuce 4.4.0
Development

No branches or pull requests
3 participants
@maxd @mp911de @hackpros