Handle concurrent WebSockets messages #6210

emlautarom1 · 2023-10-20T19:01:03Z

Fixes #6169

Changes

Use Task instead of Action when dealing with subscription messages.

Types of changes

What types of changes does your code introduce?

Bugfix (a non-breaking change that fixes an issue)
New feature (a non-breaking change that adds functionality)
Breaking change (a change that causes existing functionality not to work as expected)
Optimization
Refactoring
Documentation update
Build-related changes
Other: Description

Testing

Requires testing

Yes
No

If yes, did you write tests?

Yes
No

Notes on testing

An integration test was added to reproduce the reported issue. The test is annotated with [Explicit] since it requires an external WebSockets server running. We could make the test self contained by running a local server but this has produced issues in the past on CI (#6102).

Documentation

Requires documentation update

Yes
No

Requires explanation in Release Notes

Yes
No

- Use async everywhere

emlautarom1 · 2023-10-20T19:09:03Z

On #6078 we introduced streaming for socket messages to avoid OOM errors on big payloads. Unfortunately, this change introduced a new type of problem on subscriptions: it's possible for Nethermind to raise multiple events in a very short span of time, and when sending those messages we need to be careful of not interleaving them.

This PR addresses this problem by using Task over Action and ensuring that messages are sent sequentially, even in the presence of streaming.

LukaszRozmej

Also this would require release note information as it could break existing subscriptions in plug-ins.

LukaszRozmej · 2023-10-21T00:36:47Z

src/Nethermind/Nethermind.JsonRpc/Modules/Subscribe/Subscription.cs

@@ -22,7 +22,7 @@ protected Subscription(IJsonRpcDuplexClient jsonRpcDuplexClient)
        public string Id { get; }
        public abstract string Type { get; }
        public IJsonRpcDuplexClient JsonRpcDuplexClient { get; }
-        private Channel<Action> SendChannel { get; } = Channel.CreateUnbounded<Action>(new UnboundedChannelOptions() { SingleReader = true });


Would making the channel static - shared between subscriptions - fix the problem?
I think regardless we could/should make it static, there is no point in each subscription having own channel and it may cause problems?

Would making the channel static - shared between subscriptions - fix the problem?

Unfortunately, this also does not work. We still have the same original concurrency issues (web socket gets in a bad state).

Why making the channel static didn't fix the problem? Then it is shared by all subscriptions and should have make all messages sequential?

Maybe channel should be on web socket itself and not on subscription if that is the case.

Or if we want to keep the lock we don't need channels at all?

Why making the channel static didn't fix the problem?

I think the reason is that each subscription instance starts its own ProcessMessage loop

nethermind/src/Nethermind/Nethermind.JsonRpc/Modules/Subscribe/Subscription.cs

Lines 15 to 20 in fe2b14e

protected Subscription(IJsonRpcDuplexClient jsonRpcDuplexClient)

{

Id = string.Concat("0x", Guid.NewGuid().ToString("N"));

JsonRpcDuplexClient = jsonRpcDuplexClient;

ProcessMessages();

}

This means that we ensure no overlapping processing per subscription, but there are no guarantees of synchronization across them.

A possible solution would be to put the lock in the Subscription as static, ensuring that only one subscription at a time is doing work. I don't like this approach because a subscription might not need to be synchronized (because it does not use any shared resources like WebSockets).

Or if we want to keep the lock we don't need channels at all?

Channels were introduced in #3458 to ensure ordering of messages. I believe that we could remove them and it should not affect the behavior due to locking, but I didn't want to introduce such change in this PR.

Need to double check, but I don't think that when two thread is waiting for a lock, there is a guarentee that they are released in order.

Also you can have communication on websocket that isn't subscriptions but normal request-response. I think we could downscale to having 1 channel per all subscriptions though, which would make it cheaper.

LukaszRozmej · 2023-10-21T00:37:24Z

src/Nethermind/Nethermind.JsonRpc.Test/Modules/SubscribeModuleTests.cs

+        [TestCase(5)]
+        [TestCase(10)]
+        [Explicit("Requires a WS server running")]
+        public async Task NewPendingTransactionSubscription_multiple_fast_messages(int messages)


Can you make test with competing subscriptions (same/different channel)

Added test and it fails: since each subscription handles it's own queue of tasks we end up with the same problem.

- Locks :(

emlautarom1 · 2023-10-23T18:00:46Z

src/Nethermind/Nethermind.JsonRpc/WebSockets/JsonRpcSocketsClient.cs

-            await using CounterStream resultData = new CounterStream(buffered);
-
-            if (result.IsCollection)
+            await _sendSemaphore.WaitAsync();


The final solution requires the use of locks: through a semaphore, we ensure that only one message at a time is being sent using the underlying WebSocket.

We preserve the streaming approach, meaning that we can send large objects without failing due to OOM.

asdacap

If you are up for it refactor the whole thing by converting/adapt the underling event into IAsyncEnumerable<something>, and eventually to a json result. Potentially easier to understand.

* Test for #6169 * Use async on Subscription message processing * Fix `ScheduleAction` usages - Use async everywhere * Parametrize test * Reduce number of messages * Add test for multiple concurrent subscriptions * Add locks to avoid concurrency issues - Locks :( * Dispose semaphore * Targeted new * Reduce delays

emlautarom1 added 5 commits October 20, 2023 15:36

Test for #6169

a05a74a

Use async on Subscription message processing

8b44f59

Fix ScheduleAction usages

99a3200

- Use async everywhere

Parametrize test

0da4e8b

Reduce number of messages

9c3ec2d

emlautarom1 requested review from benaadams, asdacap and marcindsobczak October 20, 2023 19:01

emlautarom1 self-assigned this Oct 20, 2023

emlautarom1 mentioned this pull request Oct 20, 2023

Websockets failing after upgrade to 1.21.0 #6169

Closed

LukaszRozmej reviewed Oct 21, 2023

View reviewed changes

emlautarom1 added 4 commits October 23, 2023 11:34

Add test for multiple concurrent subscriptions

fe571ac

Add locks to avoid concurrency issues

4a085cd

- Locks :(

Dispose semaphore

9478eae

Targeted new

a9f7700

emlautarom1 commented Oct 23, 2023

View reviewed changes

emlautarom1 force-pushed the fix/interleaved_websockets branch from 5ed6e09 to a9f7700 Compare October 23, 2023 18:12

Reduce delays

bc619e9

emlautarom1 requested a review from LukaszRozmej October 23, 2023 18:17

asdacap approved these changes Oct 24, 2023

View reviewed changes

emlautarom1 merged commit 646ab7a into master Oct 25, 2023
61 checks passed

emlautarom1 deleted the fix/interleaved_websockets branch October 25, 2023 13:49

fselmo mentioned this pull request Jan 24, 2024

IPC subscriptions intermittently fail with double messages #6614

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle concurrent WebSockets messages #6210

Handle concurrent WebSockets messages #6210

emlautarom1 commented Oct 20, 2023

emlautarom1 commented Oct 20, 2023 •

edited

LukaszRozmej left a comment •

edited

LukaszRozmej Oct 21, 2023

emlautarom1 Oct 23, 2023

LukaszRozmej Oct 24, 2023

LukaszRozmej Oct 24, 2023

LukaszRozmej Oct 24, 2023

emlautarom1 Oct 24, 2023

emlautarom1 Oct 24, 2023

asdacap Oct 24, 2023

LukaszRozmej Oct 24, 2023

LukaszRozmej Oct 21, 2023

emlautarom1 Oct 23, 2023

emlautarom1 Oct 23, 2023

asdacap left a comment

	protected Subscription(IJsonRpcDuplexClient jsonRpcDuplexClient)
	{
	Id = string.Concat("0x", Guid.NewGuid().ToString("N"));
	JsonRpcDuplexClient = jsonRpcDuplexClient;
	ProcessMessages();
	}

Handle concurrent WebSockets messages #6210

Handle concurrent WebSockets messages #6210

Conversation

emlautarom1 commented Oct 20, 2023

Changes

Types of changes

What types of changes does your code introduce?

Testing

Requires testing

If yes, did you write tests?

Notes on testing

Documentation

Requires documentation update

Requires explanation in Release Notes

emlautarom1 commented Oct 20, 2023 • edited

LukaszRozmej left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asdacap left a comment

Choose a reason for hiding this comment

emlautarom1 commented Oct 20, 2023 •

edited

LukaszRozmej left a comment •

edited