Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reactive stream spec violation when using command timeout #1576

Closed
martin-tarjanyi opened this issue Jan 11, 2021 · 3 comments
Closed

Reactive stream spec violation when using command timeout #1576

martin-tarjanyi opened this issue Jan 11, 2021 · 3 comments
Assignees
Labels
type: blocker An issue that is blocking us from releasing type: bug A general bug
Milestone

Comments

@martin-tarjanyi
Copy link

Bug Report

Related issue: #1274

Current Behavior

When a RedisCommandTimeoutException happens it is possible (depending on CPU task scheduling) that the timeout error will be triggered after the underlying command is completed thus violating reactive spec. Futhermore, this error causes the DefaultEventExecutor to terminate which leaves most subsequent Redis commands in a stuck state when publishOnScheduler(true).

Stack trace
Thread=[lettuce-eventExecutorLoop-5-7] Logger=io.netty.util.concurrent.SingleThreadEventExecutor Type=WARN Unexpected exception from an event executor: 
reactor.core.Exceptions$BubblingException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 100 millisecond(s)
	at reactor.core.Exceptions.bubble(Exceptions.java:173) ~[reactor-core-3.3.9.RELEASE.jar!/:3.3.9.RELEASE]
	at reactor.core.publisher.Operators.onErrorDropped(Operators.java:635) ~[reactor-core-3.3.9.RELEASE.jar!/:3.3.9.RELEASE]
	at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:83) ~[reactor-core-3.3.9.RELEASE.jar!/:3.3.9.RELEASE]
	at io.lettuce.core.RedisPublisher$OnComplete.run(RedisPublisher.java:1094) ~[lettuce-core-5.3.3.RELEASE.jar!/:5.3.3.RELEASE]
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-common-4.1.51.Final.jar!/:4.1.51.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.51.Final.jar!/:4.1.51.Final]

Input Code

https://github.com/martin-tarjanyi/lettuce-executor-reject-repro

Expected behavior/code

No violation of reactive stream spec, no terminated executor, no stuck Redis commands

Environment

  • Lettuce version: 5.3.5.RELEASE
  • Redis version: any

Possible Solution

As pointed out by @mp911de SubscriptionCommand and CommandWrapper do not ensure single completion. Fixing that will fix this issue.

@mp911de mp911de changed the title Possible reactive stream spec violation when using command timeout Reactive stream spec violation when using command timeout Jan 11, 2021
@mp911de mp911de added type: bug A general bug type: blocker An issue that is blocking us from releasing labels Jan 11, 2021
@mp911de mp911de self-assigned this Jan 11, 2021
@mp911de mp911de modified the milestones: 6.0.2, 5.3.6 Jan 12, 2021
mp911de added a commit that referenced this issue Jan 12, 2021
We now ensure that CommandWrapper and its subclass SubscriptionCommand trigger only a single completion signal. Previously, a race could happen inside the completion notification which could cause a completion and error signal or error + completion because of improper guarding.
mp911de added a commit that referenced this issue Jan 12, 2021
We now ensure that CommandWrapper and its subclass SubscriptionCommand trigger only a single completion signal. Previously, a race could happen inside the completion notification which could cause a completion and error signal or error + completion because of improper guarding.
mp911de added a commit that referenced this issue Jan 12, 2021
We now ensure that CommandWrapper and its subclass SubscriptionCommand trigger only a single completion signal. Previously, a race could happen inside the completion notification which could cause a completion and error signal or error + completion because of improper guarding.
@mp911de
Copy link
Collaborator

mp911de commented Jan 12, 2021

That's fixed now.

@mp911de mp911de closed this as completed Jan 12, 2021
@martin-tarjanyi
Copy link
Author

We appreciate the really quick turnaround. Thank you @mp911de!

@mp911de
Copy link
Collaborator

mp911de commented Jan 12, 2021

The releases 6.0.2 respective 5.3.6 are now available from Maven Central

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: blocker An issue that is blocking us from releasing type: bug A general bug
Projects
None yet
Development

No branches or pull requests

2 participants