Duplicates messages received when using 2 machines #1973

Closed
Xiaohongt opened this Issue Apr 30, 2013 · 3 comments

Comments

Projects
None yet
2 participants
Contributor

Xiaohongt commented Apr 30, 2013

Functional impact:
client could receive hundreds duplicate messages

Repro:
1). use the asp.net Message Loops sample to repro
2). use ServiceBus scale-out, use 2 topics, e.g.
. in the asp.net sample ConfigureSignalR, add:

string SBConnectionString = "......";
GlobalHost.DependencyResolver.UseServiceBus(new ServiceBusScaleoutConfiguration(SBConnectionString, "test") {  TopicCount = 2 });

3). publish the app to Azure cloud services app with 2 instances
4). request first client using longPolling for the message loops page, set 1 milliseconds for the sleep input box, click "Start Message Loops" button
5). request second client using longPolling for the message loops page,
set 1 milliseconds for the sleep input box, click "Start Message Loops" button
6) wait about 5 mins, we can see hundreds dup messages received on one of clients

Expected result:
no dup message received on client

Actual result:
hundreds dup messages received on one of clients

davidfowl was assigned Apr 30, 2013

@davidfowl davidfowl added a commit that referenced this issue May 2, 2013

@davidfowl davidfowl Fixed duplicate messages from being received in scaleout.
- Use backplane generated timestamp to determine what to do when
  the cursor is bigger than the max cursor on that particular node.
- Extended the cursor id format to support timestamps as ticks in hex.
- Renamed CreationTime to ServerCreationTime and made CreationTime the
  backplane creation time
- Added stress test to verify the duplicate behavior when flip flopping between nodes.

#1973
71b9e89

@davidfowl davidfowl added a commit that referenced this issue May 4, 2013

@davidfowl davidfowl Fixed duplicate messages from being received in scaleout.
- Use backplane generated timestamp to determine what to do when
  the cursor is bigger than the max cursor on that particular node.
- Extended the cursor id format to support timestamps as ticks in hex.
- Renamed CreationTime to ServerCreationTime and made CreationTime the
  backplane creation time
- Added stress test to verify the duplicate behavior when flip flopping between nodes.

#1973
fff9098

Xiaohongt was assigned May 4, 2013

Contributor

Xiaohongt commented May 6, 2013

We repro that TrySearch failed for longPolling client and returned GetAllMessages, we need change the design “if you fall between the range but we can't find a cursor, then we've been reset” because the slower/faster issue between the instances

@davidfowl davidfowl added a commit that referenced this issue May 7, 2013

@davidfowl davidfowl Use the closest mapping in the store when there's gaps.
- Pick the first element in the fragment bigger than
  requested mapping id when failing to find the element.
- Added unit tests.

#1973
fb0e8e7

@davidfowl davidfowl added a commit that referenced this issue May 7, 2013

@davidfowl davidfowl Use the closest mapping in the store when there's gaps.
- Pick the first element in the fragment bigger than
  requested mapping id when failing to find the element.
- Added unit tests.

#1973
32b738e
Contributor

Xiaohongt commented May 8, 2013

We repro another issue causes dup messages

@davidfowl davidfowl added a commit that referenced this issue May 8, 2013

@davidfowl davidfowl Fixed duplicate messages when topic store overrun.
- Added unit tests

#1973
8a84af8

@davidfowl davidfowl added a commit that referenced this issue May 11, 2013

@davidfowl davidfowl Resolve issues with cursor ids crossing streams.
- Take the stream index the into account when checking for
  overrun buffers.
- Only process the next cursor if it's bigger than what we've
  already seen.
- Only update cursors relevant to the stream being processed.

#1973
d0a2830

@davidfowl davidfowl added a commit that referenced this issue May 11, 2013

@davidfowl davidfowl Resolve issues with cursor ids crossing streams.
- Take the stream index the into account when checking for
  overrun buffers.
- Only process the next cursor if it's bigger than what we've
  already seen.
- Only update cursors relevant to the stream being processed.

#1973
b9561bc
Contributor

Xiaohongt commented May 13, 2013

verified

Xiaohongt closed this May 13, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment