Issue-147: Add feature to pause/resume SQSMessagePoller #179

jkwpappin · 2025-02-26T16:21:46Z

Issue #147 :

Description of changes:
Use case Blue/Green deployments, where the inactive service can change and we want to stop/start the message consumption dependent upon the active service.

I have followed the contribution guideline and have aimed to follow the existing repositories coding style.

I added a PollingControlToken with Start()/Stop() methods that is setup initially by the MessageBusBuilder and stored in the MessageConfiguration, but is also registered as a singleton in the ServiceCollection.

Consumers can retrieve the PollingControlToken via DI and can then call the Start()/Stop() methods against it when they need to control the message receipt.

Testing:

Unit tests added.
Integration tests added but I have not been able to run these locally.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

jkwpappin · 2025-02-26T16:27:13Z

I exposed a configuration item, PollingControlToken, that can be used to start/stop message receipt, so that the BackgroundService (MessagePumpService) can remain unmodified.

philasmar · 2025-03-05T14:16:16Z

@jkwpappin Thanks for the PR!
I ran your branch through our internal pipeline and got 2 errors:


[xUnit.net 00:00:00.73]     AWS.Messaging.UnitTests.SQSMessagePollerTests.SQSMessagePoller_PollingControlRestarted_PollsSQS [FAIL]
--
119 | Failed AWS.Messaging.UnitTests.SQSMessagePollerTests.SQSMessagePoller_PollingControlRestarted_PollsSQS [170 ms]
120 | Error Message:
121 | Moq.MockException :
122 | Expected invocation on the mock at least once, but was never performed: x => x.ReceiveMessageAsync(It.IsAny<ReceiveMessageRequest>(), It.IsAny<CancellationToken>())
123 |  
124 | Performed invocations:
125 |  
126 | Mock<IAmazonSQS:4> (x):
127 |  
128 | IAmazonSQS.ReceiveMessageAsync(ReceiveMessageRequest, CancellationToken)
129 |  
130 | Stack Trace:
131 | at Moq.Mock.Verify(Mock mock, LambdaExpression expression, Times times, String failMessage) in C:\projects\moq4\src\Moq\Mock.cs:line 330
132 | at Moq.Mock`1.Verify[TResult](Expression`1 expression, Times times) in C:\projects\moq4\src\Moq\Mock`1.cs:line 825
133 | at AWS.Messaging.UnitTests.SQSMessagePollerTests.SQSMessagePoller_PollingControlRestarted_PollsSQS() in /codebuild/output/src1469879486/src/github.com/awslabs/aws-dotnet-messaging/test/AWS.Messaging.UnitTests/SQSMessagePollerTests.cs:line 98
134 | --- End of stack trace from previous location ---
135 | Results File: /codebuild/output/src1469879486/src/github.com/awslabs/aws-dotnet-messaging/testresults/_a9e0afdbb673_2025-03-05_14_09_07.trx
136 |  
137 | Failed!  - Failed:     1, Passed:   238, Skipped:     0, Total:   239, Duration: 40 s - AWS.Messaging.UnitTests.dll (net6.0)
138 | [xUnit.net 00:04:47.73]     AWS.Messaging.IntegrationTests.SubscriberTests.ReceiveMultipleMessagesOnlyWhenPollingControlTokenStarted [FAIL]
139 | Failed AWS.Messaging.IntegrationTests.SubscriberTests.ReceiveMultipleMessagesOnlyWhenPollingControlTokenStarted [3 s]
140 | Error Message:
141 | Assert.Equal() Failure: Values differ
142 | Expected: 5
143 | Actual:   1
144 | Stack Trace:
145 | at AWS.Messaging.IntegrationTests.SubscriberTests.ReceiveMultipleMessagesOnlyWhenPollingControlTokenStarted() in /codebuild/output/src1469879486/src/github.com/awslabs/aws-dotnet-messaging/test/AWS.Messaging.IntegrationTests/SubscriberTests.cs:line 208
146 | --- End of stack trace from previous location ---
147 | Results File: /codebuild/output/src1469879486/src/github.com/awslabs/aws-dotnet-messaging/testresults/_a9e0afdbb673_2025-03-05_14_09_14.trx
148 |  
149 | Failed!  - Failed:     1, Passed:    27, Skipped:     0, Total:    28, Duration: 5 m 42 s - AWS.Messaging.IntegrationTests.dll (net6.0)

Could you please take a look at that?

johnp-trainline · 2025-03-05T14:39:00Z

Thanks @philasmar I will take a look at the failing build

src/AWS.Messaging/Configuration/MessageBusBuilder.cs

jkwpappin · 2025-03-13T16:34:33Z

@philasmar I've managed to reproduce this behavior with the failing test.
When I run these tests in Visual Studio, or via the command dotnet test --verbosity normal --filter ".UnitTests" they will pass.

But doing the restore, build and test separately as per the github actions steps, they fail:
dotnet restore dotnet build --no-restore dotnet test --no-build --verbosity normal --filter ".UnitTests"

Going to keep investigating. Working theory is there might be a concurrency issue here that is manifesting itself for some reason when run with the -no-build flag 😬

jkwpappin · 2025-03-14T10:54:22Z

@philasmar this seems to be a concurrency issue in the tests, where sometimes the background worker won't have completed it's work when the verification is called on the Moq. I can't reproduce the failure locally anymore... when running the tests they always seem to pass now, I did make some modifications to the tests for how they block. Are you able to approve the waiting build workflows to see if they are still failing with the latest changes to the tests? Can you also reproduce these test failures when you run them? Thanks.

(cherry picked from commit 4a442fb)

(cherry picked from commit fba5d5b)

(cherry picked from commit 1946aec)

(cherry picked from commit b86353c)

… during the async execution (cherry picked from commit 6c3f0cc)

(cherry picked from commit 1b4ac7a)

philasmar · 2025-03-18T14:17:10Z

I ran the tests again and got the same result:


Starting test execution, please wait...Starting test execution, please wait...
--
109 |  
110 | A total of 1 test files matched the specified pattern.A total of 1 test files matched the specified pattern.
111 |  
112 | [xUnit.net 00:00:00.72]     AWS.Messaging.UnitTests.SQSMessagePollerTests.SQSMessagePoller_PollingControlRestarted_PollsSQS [FAIL]
113 | Failed AWS.Messaging.UnitTests.SQSMessagePollerTests.SQSMessagePoller_PollingControlRestarted_PollsSQS [341 ms]
114 | Error Message:
115 | Moq.MockException :
116 | Expected invocation on the mock at least once, but was never performed: x => x.ReceiveMessageAsync(It.IsAny<ReceiveMessageRequest>(), It.IsAny<CancellationToken>())
117 |  
118 | Performed invocations:
119 |  
120 | Mock<IAmazonSQS:2> (x):
121 | No invocations performed.
122 |  
123 | Stack Trace:
124 | at Moq.Mock.Verify(Mock mock, LambdaExpression expression, Times times, String failMessage) in C:\projects\moq4\src\Moq\Mock.cs:line 330
125 | at Moq.Mock`1.Verify[TResult](Expression`1 expression, Times times) in C:\projects\moq4\src\Moq\Mock`1.cs:line 825
126 | at AWS.Messaging.UnitTests.SQSMessagePollerTests.SQSMessagePoller_PollingControlRestarted_PollsSQS() in /codebuild/output/src2523043111/src/github.com/awslabs/aws-dotnet-messaging/test/AWS.Messaging.UnitTests/SQSMessagePollerTests.cs:line 99
127 | --- End of stack trace from previous location ---
128 | Results File: /codebuild/output/src2523043111/src/github.com/awslabs/aws-dotnet-messaging/testresults/_ca53910f0699_2025-03-18_14_07_06.trx
129 |  
130 | Failed!  - Failed:     1, Passed:   239, Skipped:     0, Total:   240, Duration: 40 s - AWS.Messaging.UnitTests.dll (net6.0)
131 | [xUnit.net 00:04:48.02]     AWS.Messaging.IntegrationTests.SubscriberTests.ReceiveMultipleMessagesOnlyWhenPollingControlTokenStarted [FAIL]
132 | Failed AWS.Messaging.IntegrationTests.SubscriberTests.ReceiveMultipleMessagesOnlyWhenPollingControlTokenStarted [3 s]
133 | Error Message:
134 | Assert.Equal() Failure: Values differ
135 | Expected: 5
136 | Actual:   1
137 | Stack Trace:
138 | at AWS.Messaging.IntegrationTests.SubscriberTests.ReceiveMultipleMessagesOnlyWhenPollingControlTokenStarted() in /codebuild/output/src2523043111/src/github.com/awslabs/aws-dotnet-messaging/test/AWS.Messaging.IntegrationTests/SubscriberTests.cs:line 209
139 | --- End of stack trace from previous location ---
140 | Results File: /codebuild/output/src2523043111/src/github.com/awslabs/aws-dotnet-messaging/testresults/_ca53910f0699_2025-03-18_14_07_13.trx
141 |  
142 | Failed!  - Failed:     1, Passed:    27, Skipped:     0, Total:    28, Duration: 5 m 42 s - AWS.Messaging.IntegrationTests.dll (net6.0)

philasmar · 2025-03-18T14:18:20Z

I ran them locally on my machine. The unit test passed but the integ test failed:

[xUnit.net 00:06:07.24]   Finished:    AWS.Messaging.IntegrationTests
Results File: /CodeBase/aws-dotnet-messaging/testresults/_80a99733e51c_2025-03-18_10_11_29.trx
  AWS.Messaging.IntegrationTests test failed with 1 error(s) (368.5s)
    /CodeBase/aws-dotnet-messaging/test/AWS.Messaging.IntegrationTests/SubscriberTests.cs(209): error TESTERROR: 
      AWS.Messaging.IntegrationTests.SubscriberTests.ReceiveMultipleMessagesOnlyWhenPollingControlTokenStarted (3s 371ms):
       Error Message: Assert.Equal() Failure: Values differ
      Expected: 5
      Actual:   2
      Stack Trace:
         at AWS.Messaging.IntegrationTests.SubscriberTests.ReceiveMultipleMessagesOnlyWhenPollingControlTokenStarted() in 
      /CodeBase/aws-dotnet-messaging/test/AWS.Messaging.IntegrationTests/SubscriberTests.cs:line 209
      --- End of stack trace from previous location ---

Test summary: total: 268, failed: 1, succeeded: 267, skipped: 0, duration: 368.5s
Build failed with 1 error(s) in 369.2s

philasmar · 2025-03-18T14:07:59Z

test/AWS.Messaging.UnitTests/MessagePublisherTests.cs

-        Assert.Equal("ErrorMessage", publishResponse.Result.InnerException.Message);
-        Assert.Equal("ErrorCode", ((EventBridgePutEventsException)publishResponse.Result.InnerException).ErrorCode);
+        Assert.Equal("Message failed to publish.", publishResponse.Message);
+        Assert.Equal("ErrorMessage", publishResponse.InnerException.Message);


Could you please fix the warnings?

Dereference of a possibly null reference.

jkwpappin · 2025-03-24T16:35:28Z

Hey @philasmar shall I close this PR as you've now raised this one to replace it?

philasmar · 2025-03-24T16:37:57Z

Yes please. I was waiting for all the tests to pass on the new one before communicating it on this PR.

philasmar · 2025-04-02T13:02:26Z

This change has been released in https://www.nuget.org/packages/AWS.Messaging/0.20.0-preview
@jkwpappin Thank you for your contribution!

jkwpappin mentioned this pull request Feb 26, 2025

Feature Request: Start and stop poller? #147

Open

2 tasks

boblodgett requested review from GarrettBeatty and philasmar February 26, 2025 18:26

GarrettBeatty reviewed Mar 6, 2025

View reviewed changes

src/AWS.Messaging/Configuration/MessageBusBuilder.cs Outdated Show resolved Hide resolved

jkwpappin added 6 commits March 17, 2025 10:57

Allow pausing and resumption of message consumption

9e05912

(cherry picked from commit 4a442fb)

Add changelog

f6608c8

(cherry picked from commit fba5d5b)

Remove redundant parenthesis

3c2d631

(cherry picked from commit 1946aec)

Fix whitespace

1293200

(cherry picked from commit b86353c)

Add a ConfigureAwait as we're trying to do something and verify a Moq…

f4ebf6e

… during the async execution (cherry picked from commit 6c3f0cc)

Use SpinWait to hold the thread

3c6a6e2

(cherry picked from commit 1b4ac7a)

johnp-trainline force-pushed the jpappin/issue-147 branch from 1b4ac7a to 3c6a6e2 Compare March 17, 2025 11:09

philasmar reviewed Mar 24, 2025

View reviewed changes

philasmar mentioned this pull request Mar 24, 2025

Issue-147: Add feature to pause/resume SQSMessagePoller #197

Merged

philasmar closed this Apr 2, 2025

Issue-147: Add feature to pause/resume SQSMessagePoller #179

Issue-147: Add feature to pause/resume SQSMessagePoller #179

Uh oh!

Conversation

jkwpappin commented Feb 26, 2025

Uh oh!

jkwpappin commented Feb 26, 2025

Uh oh!

philasmar commented Mar 5, 2025

Uh oh!

johnp-trainline commented Mar 5, 2025

Uh oh!

Uh oh!

jkwpappin commented Mar 13, 2025

Uh oh!

jkwpappin commented Mar 14, 2025

Uh oh!

philasmar commented Mar 18, 2025

Uh oh!

philasmar commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philasmar Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

philasmar Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

jkwpappin commented Mar 24, 2025

Uh oh!

philasmar commented Mar 24, 2025

Uh oh!

philasmar commented Apr 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

philasmar commented Mar 18, 2025 •

edited

Loading