Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Akka.Delivery: fix ProducerControllerImpl<T> state bug #7034

Merged
merged 5 commits into from Jan 4, 2024
Merged

Akka.Delivery: fix ProducerControllerImpl<T> state bug #7034

merged 5 commits into from Jan 4, 2024

Conversation

Aaronontheweb
Copy link
Member

@Aaronontheweb Aaronontheweb commented Jan 3, 2024 •
edited

Changes

Fixes #7033

Working on troubleshooting the issues with ReliableDeliveryRandomSpecs

Checklist

For significant changes, please ensure that the following have been completed (delete if not relevant):

@Aaronontheweb
Copy link
Member Author 

Akka.Tests.Delivery.ReliableDeliveryRandomSpecs.ReliableDelivery_with_random_failures_must_work_with_flaky_network_without_resending

Xunit.Sdk.FailException
Failed: Timeout 00:02:00 while waiting for a message of type Akka.Tests.Delivery.TestConsumer+Collected 
   at Akka.TestKit.Xunit2.XunitAssertions.Fail(String format, Object[] args) in C:\Repositories\olympus\akka.net\src\contrib\testkits\Akka.TestKit.Xunit2\XunitAssertions.cs:line 26
   at Akka.TestKit.TestKitBase.InternalExpectMsgEnvelopeAsync[T](Nullable`1 timeout, Action`2 assert, String hint, CancellationToken cancellationToken, Boolean shouldLog) in C:\Repositories\olympus\akka.net\src\core\Akka.TestKit\TestKitBase_Expect.cs:line 438
   at Akka.TestKit.TestKitBase.InternalExpectMsgEnvelopeAsync[T](Nullable`1 timeout, Action`1 msgAssert, Action`1 senderAssert, String hint, CancellationToken cancellationToken) in C:\Repositories\olympus\akka.net\src\core\Akka.TestKit\TestKitBase_Expect.cs:line 413
   at Akka.TestKit.TestKitBase.InternalExpectMsgAsync[T](Nullable`1 timeout, Action`1 msgAssert, Action`1 senderAssert, String hint, CancellationToken cancellationToken) in C:\Repositories\olympus\akka.net\src\core\Akka.TestKit\TestKitBase_Expect.cs:line 381
   at Akka.TestKit.TestKitBase.ExpectMsgAsync[T](Nullable`1 duration, String hint, CancellationToken cancellationToken) in C:\Repositories\olympus\akka.net\src\core\Akka.TestKit\TestKitBase_Expect.cs:line 53
   at Akka.Tests.Delivery.ReliableDeliveryRandomSpecs.Test(Int32 numberOfMessages, Double producerDropProbability, Double consumerDropProbability, Option`1 durableFailProbability, Boolean resendLost) in C:\Repositories\olympus\akka.net\src\core\Akka.Tests\Delivery\ReliableDeliveryRandomSpecs.cs:line 87
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 276
--- End of stack trace from previous location ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90


[INFO][01/03/2024 21:22:53.110Z][Thread 0005][ActorSystem(test)] consumerDropProbability [0.31277439939453006], producerDropProbability [0.3323438677156083], consumerDelay [00:00:00.0170000], producerDelay [00:00:00], durableFailProbability [None], durableDelay [00:00:00]
[DEBUG][01/03/2024 21:22:53.134Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [1]
[DEBUG][01/03/2024 21:22:53.136Z][Thread 0032][akka://test/user/consumer-controller-1] Received Start, unstash [0] messages
[DEBUG][01/03/2024 21:22:53.137Z][Thread 0032][ActorSystem(test)] Cancel all timers
[DEBUG][01/03/2024 21:22:53.137Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [2]
[DEBUG][01/03/2024 21:22:53.140Z][Thread 0032][akka://test/user/consumer-controller-1] Registering to new ProducerController [[akka://test/user/producer-controller-1#1815389119]], previous was [[akka://test/deadLetters]].
[DEBUG][01/03/2024 21:22:53.140Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [2]
[DEBUG][01/03/2024 21:22:53.140Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [3]
[INFO][01/03/2024 21:22:53.143Z][Thread 0033][akka://test/user/producer-1] Sent [0]
[DEBUG][01/03/2024 21:22:53.149Z][Thread 0034][akka://test/user/producer-controller-1] Sending [Message: Job(msg-0)] with seqNo [1] to consumer
[DEBUG][01/03/2024 21:22:53.149Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.ProducerController+ResendFirst] with generation [1]
[DEBUG][01/03/2024 21:22:53.151Z][Thread 0034][akka://test/user/consumer-controller-1] Received first SequencedMessage from seqNr [1], delivering to consumer.
[DEBUG][01/03/2024 21:22:53.152Z][Thread 0034][akka://test/user/consumer-controller-1] Associated with new ProducerController [[akka://test/user/producer-controller-1#1815389119]], seqNr [1].
[DEBUG][01/03/2024 21:22:53.152Z][Thread 0034][akka://test/user/consumer-controller-1] Sending Request with requestUpToSeqNr [20] after first SequencedMessage.
[DEBUG][01/03/2024 21:22:53.153Z][Thread 0012][akka://test/user/producer-controller-1] Received request, confirmed [0], requested [20], current [2]
[DEBUG][01/03/2024 21:22:53.153Z][Thread 0032][akka://test/user/consumer-1] SeqNr 1 was delivered to consumer.
[DEBUG][01/03/2024 21:22:53.153Z][Thread 0032][ActorSystem(test)] Start timer [JobDelivery { Msg = Job(msg-0), ConfirmTo = [akka://test/user/consumer-controller-1#476727834], ProducerId = p-1, SeqNr = 1 }] with generation [1]
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [1]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-1)] with seqNo [2] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [2]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-2)] with seqNo [3] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [3]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-3)] with seqNo [4] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [4]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-4)] with seqNo [5] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [5]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-5)] with seqNo [6] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [6]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-6)] with seqNo [7] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [7]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-7)] with seqNo [8] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [8]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-8)] with seqNo [9] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [9]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-9)] with seqNo [10] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [10]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-10)] with seqNo [11] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [11]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-11)] with seqNo [12] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [12]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-12)] with seqNo [13] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [13]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-13)] with seqNo [14] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [14]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-14)] with seqNo [15] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [15]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-15)] with seqNo [16] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [16]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-16)] with seqNo [17] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [17]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-controller-1] Sending [Message: Job(msg-17)] with seqNo [18] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0032][akka://test/user/producer-1] Sent [18]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0032][akka://test/user/producer-controller-1] Sending [Message: Job(msg-18)] with seqNo [19] to consumer
[INFO][01/03/2024 21:22:53.160Z][Thread 0012][akka://test/user/producer-1] Sent [19]
[DEBUG][01/03/2024 21:22:53.160Z][Thread 0032][akka://test/user/producer-controller-1] Sending [Message: Job(msg-19)] with seqNo [20] to consumer
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [2], stashing while waiting for consumer to confirm [1]. Stashed size [1].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [3], stashing while waiting for consumer to confirm [1]. Stashed size [2].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] [Testing] dropping message [SequencedMessage { ProducerId = p-1, SeqNr = 4, Message = Message: Job(msg-3), First = False, Ack = False }] due to fuzzing factor
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] [Testing] dropping message [SequencedMessage { ProducerId = p-1, SeqNr = 5, Message = Message: Job(msg-4), First = False, Ack = False }] due to fuzzing factor
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [6], stashing while waiting for consumer to confirm [1]. Stashed size [3].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [7], stashing while waiting for consumer to confirm [1]. Stashed size [4].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [8], stashing while waiting for consumer to confirm [1]. Stashed size [5].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [9], stashing while waiting for consumer to confirm [1]. Stashed size [6].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [10], stashing while waiting for consumer to confirm [1]. Stashed size [7].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [11], stashing while waiting for consumer to confirm [1]. Stashed size [8].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [12], stashing while waiting for consumer to confirm [1]. Stashed size [9].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] [Testing] dropping message [SequencedMessage { ProducerId = p-1, SeqNr = 13, Message = Message: Job(msg-12), First = False, Ack = False }] due to fuzzing factor
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [14], stashing while waiting for consumer to confirm [1]. Stashed size [10].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [15], stashing while waiting for consumer to confirm [1]. Stashed size [11].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [16], stashing while waiting for consumer to confirm [1]. Stashed size [12].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [17], stashing while waiting for consumer to confirm [1]. Stashed size [13].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [18], stashing while waiting for consumer to confirm [1]. Stashed size [14].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [19], stashing while waiting for consumer to confirm [1]. Stashed size [15].
[DEBUG][01/03/2024 21:22:53.161Z][Thread 0034][akka://test/user/consumer-controller-1] [Testing] dropping message [SequencedMessage { ProducerId = p-1, SeqNr = 20, Message = Message: Job(msg-19), First = False, Ack = False }] due to fuzzing factor
[INFO][01/03/2024 21:22:53.193Z][Thread 0034][akka://test/user/consumer-1] processed [1] [msg: msg-0] from [p-1]
[DEBUG][01/03/2024 21:22:53.194Z][Thread 0032][akka://test/user/consumer-controller-1] Received Confirmed seqNr [1] from consumer, stashed size [15].
[DEBUG][01/03/2024 21:22:53.195Z][Thread 0032][akka://test/user/consumer-controller-1] Sending Request after first with confirmedSeqNr [1], requestUpToSeqNr [20]
[DEBUG][01/03/2024 21:22:53.196Z][Thread 0032][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [2], delivering to consumer.
[DEBUG][01/03/2024 21:22:53.196Z][Thread 0032][akka://test/user/consumer-1] SeqNr 2 was delivered to consumer.
[DEBUG][01/03/2024 21:22:53.196Z][Thread 0032][ActorSystem(test)] Start timer [JobDelivery { Msg = Job(msg-1), ConfirmTo = [akka://test/user/consumer-controller-1#476727834], ProducerId = p-1, SeqNr = 2 }] with generation [2]
[INFO][01/03/2024 21:22:53.220Z][Thread 0032][akka://test/user/consumer-1] processed [2] [msg: msg-1] from [p-1]
[DEBUG][01/03/2024 21:22:53.220Z][Thread 0035][akka://test/user/consumer-controller-1] Received Confirmed seqNr [2] from consumer, stashed size [14].
[DEBUG][01/03/2024 21:22:53.220Z][Thread 0035][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [3], delivering to consumer.
[DEBUG][01/03/2024 21:22:53.220Z][Thread 0032][akka://test/user/consumer-1] SeqNr 3 was delivered to consumer.
[DEBUG][01/03/2024 21:22:53.220Z][Thread 0032][ActorSystem(test)] Start timer [JobDelivery { Msg = Job(msg-2), ConfirmTo = [akka://test/user/consumer-controller-1#476727834], ProducerId = p-1, SeqNr = 3 }] with generation [3]
[INFO][01/03/2024 21:22:53.235Z][Thread 0032][akka://test/user/consumer-1] processed [3] [msg: msg-2] from [p-1]
[DEBUG][01/03/2024 21:22:53.235Z][Thread 0034][akka://test/user/consumer-controller-1] Received Confirmed seqNr [3] from consumer, stashed size [13].
[DEBUG][01/03/2024 21:22:53.235Z][Thread 0034][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [6], but expected [4], delivering to consumer anyway.
[DEBUG][01/03/2024 21:22:53.236Z][Thread 0032][akka://test/user/consumer-1] SeqNr 6 was delivered to consumer.
[DEBUG][01/03/2024 21:22:53.236Z][Thread 0032][ActorSystem(test)] Start timer [JobDelivery { Msg = Job(msg-5), ConfirmTo = [akka://test/user/consumer-controller-1#476727834], ProducerId = p-1, SeqNr = 6 }] with generation [4]
[INFO][01/03/2024 21:22:53.265Z][Thread 0032][akka://test/user/consumer-1] processed [6] [msg: msg-5] from [p-1]
[DEBUG][01/03/2024 21:22:53.265Z][Thread 0032][akka://test/user/consumer-controller-1] Received Confirmed seqNr [6] from consumer, stashed size [12].
[DEBUG][01/03/2024 21:22:53.265Z][Thread 0032][akka://test/user/consumer-controller-1] Received SequencedMessage seqNr [7], delivering to consumer.
[DEBUG][01/03/2024 21:22:53.265Z][Thread 0032][akka://test/user/consumer-1] SeqNr 7 was delivered to consumer.
[DEBUG][01/03/2024 21:22:53.265Z][Thread 0032][ActorSystem(test)] Start timer [JobDelivery { Msg = Job(msg-6), ConfirmTo = [akka://test/user/consumer-controller-1#476727834], ProducerId = p-1, SeqNr = 7 }] with generation [5]
[INFO][01/03/2024 21:22:53.296Z][Thread 0007][akka://test/user/consumer-1] processed [7] [msg: msg-6] from [p-1]
[DEBUG][01/03/2024 21:22:53.296Z][Thread 0007][akka://test/user/consumer-controller-1] Received Confirmed seqNr [7] from consumer, stashed size [11].
[DEBUG][01/03/2024 21:22:53.296Z][Thread 0007][akka://test/user/consumer-controller-1] [Testing] dropping message [SequencedMessage { ProducerId = p-1, SeqNr = 8, Message = Message: Job(msg-7), First = False, Ack = False }] due to fuzzing factor
[DEBUG][01/03/2024 21:22:53.652Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [3]
[DEBUG][01/03/2024 21:22:53.652Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [4]
[DEBUG][01/03/2024 21:22:53.652Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:22:54.409Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [4]
[DEBUG][01/03/2024 21:22:54.409Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [5]
[DEBUG][01/03/2024 21:22:54.409Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:22:55.166Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [5]
[DEBUG][01/03/2024 21:22:55.166Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [6]
[DEBUG][01/03/2024 21:22:55.166Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:22:55.931Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [6]
[DEBUG][01/03/2024 21:22:55.931Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [7]
[DEBUG][01/03/2024 21:22:55.932Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:22:55.932Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:22:55.932Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:22:56.696Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [7]
[DEBUG][01/03/2024 21:22:56.696Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [8]
[DEBUG][01/03/2024 21:22:56.696Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:22:56.696Z][Thread 0035][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:22:56.696Z][Thread 0035][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:22:57.459Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [8]
[DEBUG][01/03/2024 21:22:57.460Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [9]
[DEBUG][01/03/2024 21:22:57.460Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:22:57.460Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:22:57.460Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:22:58.219Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [9]
[DEBUG][01/03/2024 21:22:58.219Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [10]
[DEBUG][01/03/2024 21:22:58.219Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:22:58.980Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [10]
[DEBUG][01/03/2024 21:22:58.980Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [11]
[DEBUG][01/03/2024 21:22:58.980Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:22:58.980Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:22:58.980Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:22:59.739Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [11]
[DEBUG][01/03/2024 21:22:59.740Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [12]
[DEBUG][01/03/2024 21:22:59.740Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:22:59.740Z][Thread 0035][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:22:59.740Z][Thread 0035][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:00.495Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [12]
[DEBUG][01/03/2024 21:23:00.495Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [13]
[DEBUG][01/03/2024 21:23:00.495Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:00.495Z][Thread 0035][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:00.496Z][Thread 0035][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:01.259Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [13]
[DEBUG][01/03/2024 21:23:01.259Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [14]
[DEBUG][01/03/2024 21:23:01.259Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:01.259Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:01.259Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:02.026Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [14]
[DEBUG][01/03/2024 21:23:02.026Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [15]
[DEBUG][01/03/2024 21:23:02.026Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:02.026Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:02.026Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:02.783Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [15]
[DEBUG][01/03/2024 21:23:02.783Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [16]
[DEBUG][01/03/2024 21:23:02.783Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:02.783Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:02.783Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:03.546Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [16]
[DEBUG][01/03/2024 21:23:03.546Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [17]
[DEBUG][01/03/2024 21:23:03.546Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:04.318Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [17]
[DEBUG][01/03/2024 21:23:04.318Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [18]
[DEBUG][01/03/2024 21:23:04.318Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:04.318Z][Thread 0035][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:04.318Z][Thread 0035][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:05.086Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [18]
[DEBUG][01/03/2024 21:23:05.086Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [19]
[DEBUG][01/03/2024 21:23:05.086Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:05.086Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:05.086Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:05.846Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [19]
[DEBUG][01/03/2024 21:23:05.846Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [20]
[DEBUG][01/03/2024 21:23:05.846Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:05.846Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:05.846Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:06.610Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [20]
[DEBUG][01/03/2024 21:23:06.610Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [21]
[DEBUG][01/03/2024 21:23:06.610Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:06.610Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:06.610Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:07.374Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [21]
[DEBUG][01/03/2024 21:23:07.374Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [22]
[DEBUG][01/03/2024 21:23:07.374Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:07.374Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:07.374Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:08.139Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [22]
[DEBUG][01/03/2024 21:23:08.139Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [23]
[DEBUG][01/03/2024 21:23:08.139Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:08.902Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [23]
[DEBUG][01/03/2024 21:23:08.902Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [24]
[DEBUG][01/03/2024 21:23:08.902Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:09.666Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [24]
[DEBUG][01/03/2024 21:23:09.666Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [25]
[DEBUG][01/03/2024 21:23:09.666Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:10.430Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [25]
[DEBUG][01/03/2024 21:23:10.430Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [26]
[DEBUG][01/03/2024 21:23:10.430Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:11.194Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [26]
[DEBUG][01/03/2024 21:23:11.194Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [27]
[DEBUG][01/03/2024 21:23:11.194Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:11.194Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:11.194Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:11.965Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [27]
[DEBUG][01/03/2024 21:23:11.965Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [28]
[DEBUG][01/03/2024 21:23:11.965Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:11.966Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:11.966Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:12.732Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [28]
[DEBUG][01/03/2024 21:23:12.732Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [29]
[DEBUG][01/03/2024 21:23:12.732Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:12.732Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:12.732Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:13.496Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [29]
[DEBUG][01/03/2024 21:23:13.496Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [30]
[DEBUG][01/03/2024 21:23:13.496Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:13.496Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:13.496Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:14.256Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [30]
[DEBUG][01/03/2024 21:23:14.256Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [31]
[DEBUG][01/03/2024 21:23:14.256Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:14.256Z][Thread 0035][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:14.256Z][Thread 0035][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:15.021Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [31]
[DEBUG][01/03/2024 21:23:15.021Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [32]
[DEBUG][01/03/2024 21:23:15.021Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:15.021Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:15.021Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:15.774Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [32]
[DEBUG][01/03/2024 21:23:15.774Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [33]
[DEBUG][01/03/2024 21:23:15.774Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:15.774Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:15.774Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:16.535Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [33]
[DEBUG][01/03/2024 21:23:16.535Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [34]
[DEBUG][01/03/2024 21:23:16.535Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:17.301Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [34]
[DEBUG][01/03/2024 21:23:17.301Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [35]
[DEBUG][01/03/2024 21:23:17.301Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:17.301Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:17.301Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:18.067Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [35]
[DEBUG][01/03/2024 21:23:18.067Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [36]
[DEBUG][01/03/2024 21:23:18.067Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:18.067Z][Thread 0035][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:18.067Z][Thread 0035][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:18.829Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [36]
[DEBUG][01/03/2024 21:23:18.829Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [37]
[DEBUG][01/03/2024 21:23:18.829Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:18.829Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:18.829Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:19.591Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [37]
[DEBUG][01/03/2024 21:23:19.591Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [38]
[DEBUG][01/03/2024 21:23:19.591Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:19.591Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:19.591Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:20.353Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [38]
[DEBUG][01/03/2024 21:23:20.353Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [39]
[DEBUG][01/03/2024 21:23:20.353Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:20.353Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:20.353Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:21.118Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [39]
[DEBUG][01/03/2024 21:23:21.118Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [40]
[DEBUG][01/03/2024 21:23:21.118Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:21.118Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:21.118Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:21.883Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [40]
[DEBUG][01/03/2024 21:23:21.883Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [41]
[DEBUG][01/03/2024 21:23:21.883Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:21.883Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:21.883Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:22.645Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [41]
[DEBUG][01/03/2024 21:23:22.645Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [42]
[DEBUG][01/03/2024 21:23:22.645Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:23.408Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [42]
[DEBUG][01/03/2024 21:23:23.408Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [43]
[DEBUG][01/03/2024 21:23:23.408Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:24.172Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [43]
[DEBUG][01/03/2024 21:23:24.172Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [44]
[DEBUG][01/03/2024 21:23:24.172Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:24.172Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:24.172Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:24.933Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [44]
[DEBUG][01/03/2024 21:23:24.933Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [45]
[DEBUG][01/03/2024 21:23:24.933Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:25.697Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [45]
[DEBUG][01/03/2024 21:23:25.697Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [46]
[DEBUG][01/03/2024 21:23:25.697Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:25.697Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:25.697Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:26.457Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [46]
[DEBUG][01/03/2024 21:23:26.457Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [47]
[DEBUG][01/03/2024 21:23:26.457Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:27.217Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [47]
[DEBUG][01/03/2024 21:23:27.217Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [48]
[DEBUG][01/03/2024 21:23:27.218Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:27.218Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:27.218Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:27.977Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [48]
[DEBUG][01/03/2024 21:23:27.977Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [49]
[DEBUG][01/03/2024 21:23:27.978Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:28.742Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [49]
[DEBUG][01/03/2024 21:23:28.742Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [50]
[DEBUG][01/03/2024 21:23:28.742Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:28.742Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:28.742Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:29.507Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [50]
[DEBUG][01/03/2024 21:23:29.507Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [51]
[DEBUG][01/03/2024 21:23:29.507Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:29.507Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:29.507Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:30.270Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [51]
[DEBUG][01/03/2024 21:23:30.270Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [52]
[DEBUG][01/03/2024 21:23:30.270Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:30.270Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:30.270Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:31.033Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [52]
[DEBUG][01/03/2024 21:23:31.033Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [53]
[DEBUG][01/03/2024 21:23:31.033Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:31.033Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:31.033Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:31.799Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [53]
[DEBUG][01/03/2024 21:23:31.799Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [54]
[DEBUG][01/03/2024 21:23:31.799Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:31.799Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:31.799Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:32.559Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [54]
[DEBUG][01/03/2024 21:23:32.559Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [55]
[DEBUG][01/03/2024 21:23:32.559Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:33.323Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [55]
[DEBUG][01/03/2024 21:23:33.323Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [56]
[DEBUG][01/03/2024 21:23:33.323Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:33.323Z][Thread 0035][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:33.324Z][Thread 0035][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:34.086Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [56]
[DEBUG][01/03/2024 21:23:34.086Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [57]
[DEBUG][01/03/2024 21:23:34.086Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:34.851Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [57]
[DEBUG][01/03/2024 21:23:34.851Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [58]
[DEBUG][01/03/2024 21:23:34.851Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:34.851Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:34.851Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:35.610Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [58]
[DEBUG][01/03/2024 21:23:35.610Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [59]
[DEBUG][01/03/2024 21:23:35.610Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:35.611Z][Thread 0035][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:35.611Z][Thread 0035][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:36.373Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [59]
[DEBUG][01/03/2024 21:23:36.373Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [60]
[DEBUG][01/03/2024 21:23:36.373Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:37.140Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [60]
[DEBUG][01/03/2024 21:23:37.140Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [61]
[DEBUG][01/03/2024 21:23:37.140Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:37.140Z][Thread 0035][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:37.140Z][Thread 0035][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:37.897Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [61]
[DEBUG][01/03/2024 21:23:37.897Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [62]
[DEBUG][01/03/2024 21:23:37.897Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:37.897Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:37.897Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:38.658Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [62]
[DEBUG][01/03/2024 21:23:38.658Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [63]
[DEBUG][01/03/2024 21:23:38.658Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:38.658Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:38.658Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:39.420Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [63]
[DEBUG][01/03/2024 21:23:39.420Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [64]
[DEBUG][01/03/2024 21:23:39.420Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:40.175Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [64]
[DEBUG][01/03/2024 21:23:40.175Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [65]
[DEBUG][01/03/2024 21:23:40.175Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:40.175Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:40.175Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:40.939Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [65]
[DEBUG][01/03/2024 21:23:40.939Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [66]
[DEBUG][01/03/2024 21:23:40.940Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:41.702Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [66]
[DEBUG][01/03/2024 21:23:41.702Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [67]
[DEBUG][01/03/2024 21:23:41.702Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:41.702Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:41.702Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:42.461Z][Thread 0035][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [67]
[DEBUG][01/03/2024 21:23:42.461Z][Thread 0035][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [68]
[DEBUG][01/03/2024 21:23:42.461Z][Thread 0035][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:42.461Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:42.461Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:43.221Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [68]
[DEBUG][01/03/2024 21:23:43.221Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [69]
[DEBUG][01/03/2024 21:23:43.221Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:43.222Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:43.222Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:43.984Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [69]
[DEBUG][01/03/2024 21:23:43.984Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [70]
[DEBUG][01/03/2024 21:23:43.984Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:44.749Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [70]
[DEBUG][01/03/2024 21:23:44.749Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [71]
[DEBUG][01/03/2024 21:23:44.749Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:44.749Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:44.749Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:45.511Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [71]
[DEBUG][01/03/2024 21:23:45.511Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [72]
[DEBUG][01/03/2024 21:23:45.511Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:45.511Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:45.511Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:46.273Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [72]
[DEBUG][01/03/2024 21:23:46.273Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [73]
[DEBUG][01/03/2024 21:23:46.273Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:46.273Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:46.273Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:47.036Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [73]
[DEBUG][01/03/2024 21:23:47.036Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [74]
[DEBUG][01/03/2024 21:23:47.037Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:47.037Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:47.037Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:47.797Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [74]
[DEBUG][01/03/2024 21:23:47.797Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [75]
[DEBUG][01/03/2024 21:23:47.797Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:47.797Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:47.797Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:48.561Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [75]
[DEBUG][01/03/2024 21:23:48.561Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [76]
[DEBUG][01/03/2024 21:23:48.561Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:49.323Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [76]
[DEBUG][01/03/2024 21:23:49.323Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [77]
[DEBUG][01/03/2024 21:23:49.323Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:49.323Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:49.323Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:50.083Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [77]
[DEBUG][01/03/2024 21:23:50.083Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [78]
[DEBUG][01/03/2024 21:23:50.083Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:50.084Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:50.084Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:50.849Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [78]
[DEBUG][01/03/2024 21:23:50.849Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [79]
[DEBUG][01/03/2024 21:23:50.849Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:50.849Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:50.849Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:51.613Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [79]
[DEBUG][01/03/2024 21:23:51.613Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [80]
[DEBUG][01/03/2024 21:23:51.613Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:51.613Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:51.613Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:52.375Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [80]
[DEBUG][01/03/2024 21:23:52.375Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [81]
[DEBUG][01/03/2024 21:23:52.375Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:53.138Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [81]
[DEBUG][01/03/2024 21:23:53.138Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [82]
[DEBUG][01/03/2024 21:23:53.138Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:53.138Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:53.138Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:53.903Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [82]
[DEBUG][01/03/2024 21:23:53.903Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [83]
[DEBUG][01/03/2024 21:23:53.903Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:54.667Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [83]
[DEBUG][01/03/2024 21:23:54.667Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [84]
[DEBUG][01/03/2024 21:23:54.667Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:54.667Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:54.667Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:55.431Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [84]
[DEBUG][01/03/2024 21:23:55.431Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [85]
[DEBUG][01/03/2024 21:23:55.431Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:56.192Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [85]
[DEBUG][01/03/2024 21:23:56.192Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [86]
[DEBUG][01/03/2024 21:23:56.192Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:56.192Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:56.192Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:56.954Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [86]
[DEBUG][01/03/2024 21:23:56.954Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [87]
[DEBUG][01/03/2024 21:23:56.954Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:57.714Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [87]
[DEBUG][01/03/2024 21:23:57.714Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [88]
[DEBUG][01/03/2024 21:23:57.714Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:57.714Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:23:57.714Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:23:58.476Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [88]
[DEBUG][01/03/2024 21:23:58.476Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [89]
[DEBUG][01/03/2024 21:23:58.476Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:23:59.242Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [89]
[DEBUG][01/03/2024 21:23:59.242Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [90]
[DEBUG][01/03/2024 21:23:59.242Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:00.006Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [90]
[DEBUG][01/03/2024 21:24:00.006Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [91]
[DEBUG][01/03/2024 21:24:00.006Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:00.006Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:00.007Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:00.769Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [91]
[DEBUG][01/03/2024 21:24:00.769Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [92]
[DEBUG][01/03/2024 21:24:00.769Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:01.533Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [92]
[DEBUG][01/03/2024 21:24:01.533Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [93]
[DEBUG][01/03/2024 21:24:01.533Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:01.533Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:01.533Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:02.294Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [93]
[DEBUG][01/03/2024 21:24:02.294Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [94]
[DEBUG][01/03/2024 21:24:02.294Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:02.294Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:02.294Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:03.057Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [94]
[DEBUG][01/03/2024 21:24:03.057Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [95]
[DEBUG][01/03/2024 21:24:03.057Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:03.057Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:03.057Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:03.816Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [95]
[DEBUG][01/03/2024 21:24:03.816Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [96]
[DEBUG][01/03/2024 21:24:03.816Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:03.816Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:03.816Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:04.576Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [96]
[DEBUG][01/03/2024 21:24:04.576Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [97]
[DEBUG][01/03/2024 21:24:04.576Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:05.337Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [97]
[DEBUG][01/03/2024 21:24:05.337Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [98]
[DEBUG][01/03/2024 21:24:05.337Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:06.100Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [98]
[DEBUG][01/03/2024 21:24:06.100Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [99]
[DEBUG][01/03/2024 21:24:06.100Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:06.100Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:06.100Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:06.861Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [99]
[DEBUG][01/03/2024 21:24:06.861Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [100]
[DEBUG][01/03/2024 21:24:06.861Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:06.861Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:06.861Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:07.621Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [100]
[DEBUG][01/03/2024 21:24:07.621Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [101]
[DEBUG][01/03/2024 21:24:07.621Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:07.621Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:07.621Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:08.384Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [101]
[DEBUG][01/03/2024 21:24:08.384Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [102]
[DEBUG][01/03/2024 21:24:08.384Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:08.384Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:08.384Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:09.147Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [102]
[DEBUG][01/03/2024 21:24:09.147Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [103]
[DEBUG][01/03/2024 21:24:09.147Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:09.906Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [103]
[DEBUG][01/03/2024 21:24:09.906Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [104]
[DEBUG][01/03/2024 21:24:09.906Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:09.906Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:09.906Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:10.670Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [104]
[DEBUG][01/03/2024 21:24:10.670Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [105]
[DEBUG][01/03/2024 21:24:10.670Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:10.670Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:10.670Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:11.435Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [105]
[DEBUG][01/03/2024 21:24:11.435Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [106]
[DEBUG][01/03/2024 21:24:11.435Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:11.435Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:11.435Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:12.203Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [106]
[DEBUG][01/03/2024 21:24:12.203Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [107]
[DEBUG][01/03/2024 21:24:12.203Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:12.203Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:12.203Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:12.976Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [107]
[DEBUG][01/03/2024 21:24:12.976Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [108]
[DEBUG][01/03/2024 21:24:12.976Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:12.976Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:12.976Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:13.736Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [108]
[DEBUG][01/03/2024 21:24:13.736Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [109]
[DEBUG][01/03/2024 21:24:13.736Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:13.736Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:13.736Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:14.501Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [109]
[DEBUG][01/03/2024 21:24:14.501Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [110]
[DEBUG][01/03/2024 21:24:14.501Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:14.501Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:14.501Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:15.262Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [110]
[DEBUG][01/03/2024 21:24:15.262Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [111]
[DEBUG][01/03/2024 21:24:15.262Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:15.262Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:15.262Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:16.026Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [111]
[DEBUG][01/03/2024 21:24:16.026Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [112]
[DEBUG][01/03/2024 21:24:16.026Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:16.788Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [112]
[DEBUG][01/03/2024 21:24:16.788Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [113]
[DEBUG][01/03/2024 21:24:16.788Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:17.547Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [113]
[DEBUG][01/03/2024 21:24:17.547Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [114]
[DEBUG][01/03/2024 21:24:17.547Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:17.548Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:17.548Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:18.308Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [114]
[DEBUG][01/03/2024 21:24:18.308Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [115]
[DEBUG][01/03/2024 21:24:18.308Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:19.070Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [115]
[DEBUG][01/03/2024 21:24:19.070Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [116]
[DEBUG][01/03/2024 21:24:19.070Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:19.833Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [116]
[DEBUG][01/03/2024 21:24:19.833Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [117]
[DEBUG][01/03/2024 21:24:19.833Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:19.833Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:19.833Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:20.594Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [117]
[DEBUG][01/03/2024 21:24:20.594Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [118]
[DEBUG][01/03/2024 21:24:20.594Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:21.357Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [118]
[DEBUG][01/03/2024 21:24:21.357Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [119]
[DEBUG][01/03/2024 21:24:21.357Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:21.357Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:21.357Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:22.117Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [119]
[DEBUG][01/03/2024 21:24:22.117Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [120]
[DEBUG][01/03/2024 21:24:22.117Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:22.117Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:22.117Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:22.877Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [120]
[DEBUG][01/03/2024 21:24:22.877Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [121]
[DEBUG][01/03/2024 21:24:22.877Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:22.877Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:22.877Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:23.642Z][Thread 0033][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [121]
[DEBUG][01/03/2024 21:24:23.642Z][Thread 0033][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [122]
[DEBUG][01/03/2024 21:24:23.642Z][Thread 0033][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:23.642Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:23.642Z][Thread 0033][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:24.404Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [122]
[DEBUG][01/03/2024 21:24:24.404Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [123]
[DEBUG][01/03/2024 21:24:24.404Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:25.175Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [123]
[DEBUG][01/03/2024 21:24:25.175Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [124]
[DEBUG][01/03/2024 21:24:25.175Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:25.175Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:25.175Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:25.936Z][Thread 0003][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [124]
[DEBUG][01/03/2024 21:24:25.936Z][Thread 0003][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [125]
[DEBUG][01/03/2024 21:24:25.936Z][Thread 0003][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:25.936Z][Thread 0003][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:25.936Z][Thread 0003][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:26.698Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [125]
[DEBUG][01/03/2024 21:24:26.698Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [126]
[DEBUG][01/03/2024 21:24:26.698Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:27.456Z][Thread 0003][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [126]
[DEBUG][01/03/2024 21:24:27.456Z][Thread 0003][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [127]
[DEBUG][01/03/2024 21:24:27.456Z][Thread 0003][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:27.456Z][Thread 0003][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:27.456Z][Thread 0003][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:28.221Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [127]
[DEBUG][01/03/2024 21:24:28.221Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [128]
[DEBUG][01/03/2024 21:24:28.221Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:28.221Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:28.221Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:28.986Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [128]
[DEBUG][01/03/2024 21:24:28.986Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [129]
[DEBUG][01/03/2024 21:24:28.986Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:28.986Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:28.986Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:29.748Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [129]
[DEBUG][01/03/2024 21:24:29.748Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [130]
[DEBUG][01/03/2024 21:24:29.748Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:29.748Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:29.748Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:30.510Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [130]
[DEBUG][01/03/2024 21:24:30.510Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [131]
[DEBUG][01/03/2024 21:24:30.510Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:30.510Z][Thread 0003][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:30.510Z][Thread 0003][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:31.275Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [131]
[DEBUG][01/03/2024 21:24:31.275Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [132]
[DEBUG][01/03/2024 21:24:31.275Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:31.275Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:31.275Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:32.037Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [132]
[DEBUG][01/03/2024 21:24:32.037Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [133]
[DEBUG][01/03/2024 21:24:32.037Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:32.037Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:32.037Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:32.799Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [133]
[DEBUG][01/03/2024 21:24:32.799Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [134]
[DEBUG][01/03/2024 21:24:32.799Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:32.799Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:32.799Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:33.562Z][Thread 0003][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [134]
[DEBUG][01/03/2024 21:24:33.562Z][Thread 0003][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [135]
[DEBUG][01/03/2024 21:24:33.562Z][Thread 0003][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:33.562Z][Thread 0003][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:33.562Z][Thread 0003][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:34.323Z][Thread 0003][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [135]
[DEBUG][01/03/2024 21:24:34.323Z][Thread 0003][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [136]
[DEBUG][01/03/2024 21:24:34.323Z][Thread 0003][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:35.086Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [136]
[DEBUG][01/03/2024 21:24:35.086Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [137]
[DEBUG][01/03/2024 21:24:35.086Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:35.086Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:35.086Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:35.848Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [137]
[DEBUG][01/03/2024 21:24:35.848Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [138]
[DEBUG][01/03/2024 21:24:35.848Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:35.848Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:35.848Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:36.613Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [138]
[DEBUG][01/03/2024 21:24:36.613Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [139]
[DEBUG][01/03/2024 21:24:36.613Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:36.613Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:36.613Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:37.376Z][Thread 0003][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [139]
[DEBUG][01/03/2024 21:24:37.376Z][Thread 0003][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [140]
[DEBUG][01/03/2024 21:24:37.376Z][Thread 0003][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:37.376Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:37.376Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:38.137Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [140]
[DEBUG][01/03/2024 21:24:38.137Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [141]
[DEBUG][01/03/2024 21:24:38.137Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:38.902Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [141]
[DEBUG][01/03/2024 21:24:38.902Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [142]
[DEBUG][01/03/2024 21:24:38.902Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:38.902Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:38.902Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:39.665Z][Thread 0003][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [142]
[DEBUG][01/03/2024 21:24:39.665Z][Thread 0003][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [143]
[DEBUG][01/03/2024 21:24:39.665Z][Thread 0003][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:39.665Z][Thread 0003][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:39.665Z][Thread 0003][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:40.424Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [143]
[DEBUG][01/03/2024 21:24:40.424Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [144]
[DEBUG][01/03/2024 21:24:40.424Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:40.424Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:40.424Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:41.188Z][Thread 0003][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [144]
[DEBUG][01/03/2024 21:24:41.188Z][Thread 0003][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [145]
[DEBUG][01/03/2024 21:24:41.188Z][Thread 0003][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:41.188Z][Thread 0003][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:41.188Z][Thread 0003][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:41.951Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [145]
[DEBUG][01/03/2024 21:24:41.951Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [146]
[DEBUG][01/03/2024 21:24:41.951Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:42.714Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [146]
[DEBUG][01/03/2024 21:24:42.714Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [147]
[DEBUG][01/03/2024 21:24:42.714Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:42.714Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:42.714Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:43.477Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [147]
[DEBUG][01/03/2024 21:24:43.477Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [148]
[DEBUG][01/03/2024 21:24:43.477Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:43.477Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:43.477Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:44.242Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [148]
[DEBUG][01/03/2024 21:24:44.242Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [149]
[DEBUG][01/03/2024 21:24:44.242Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:44.242Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:44.242Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:45.001Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [149]
[DEBUG][01/03/2024 21:24:45.001Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [150]
[DEBUG][01/03/2024 21:24:45.001Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:45.002Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:45.002Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:45.763Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [150]
[DEBUG][01/03/2024 21:24:45.763Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [151]
[DEBUG][01/03/2024 21:24:45.763Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:46.523Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [151]
[DEBUG][01/03/2024 21:24:46.523Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [152]
[DEBUG][01/03/2024 21:24:46.523Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:46.523Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:46.523Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:47.298Z][Thread 0003][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [152]
[DEBUG][01/03/2024 21:24:47.298Z][Thread 0003][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [153]
[DEBUG][01/03/2024 21:24:47.298Z][Thread 0003][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:48.056Z][Thread 0007][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [153]
[DEBUG][01/03/2024 21:24:48.056Z][Thread 0007][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [154]
[DEBUG][01/03/2024 21:24:48.056Z][Thread 0007][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:48.056Z][Thread 0007][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:48.056Z][Thread 0007][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:48.823Z][Thread 0036][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [154]
[DEBUG][01/03/2024 21:24:48.823Z][Thread 0036][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [155]
[DEBUG][01/03/2024 21:24:48.823Z][Thread 0036][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:48.823Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:48.823Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:49.587Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [155]
[DEBUG][01/03/2024 21:24:49.587Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [156]
[DEBUG][01/03/2024 21:24:49.587Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:49.587Z][Thread 0034][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:49.587Z][Thread 0034][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:50.349Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [156]
[DEBUG][01/03/2024 21:24:50.349Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [157]
[DEBUG][01/03/2024 21:24:50.349Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:51.113Z][Thread 0034][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [157]
[DEBUG][01/03/2024 21:24:51.113Z][Thread 0034][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [158]
[DEBUG][01/03/2024 21:24:51.113Z][Thread 0034][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:51.113Z][Thread 0036][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:51.113Z][Thread 0036][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:51.880Z][Thread 0037][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [158]
[DEBUG][01/03/2024 21:24:51.880Z][Thread 0037][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [159]
[DEBUG][01/03/2024 21:24:51.880Z][Thread 0037][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:51.880Z][Thread 0037][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:51.880Z][Thread 0037][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:52.641Z][Thread 0032][ActorSystem(test)] Cancel timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [159]
[DEBUG][01/03/2024 21:24:52.641Z][Thread 0032][ActorSystem(test)] Start timer [Akka.Delivery.Internal.ConsumerController`1+Retry[Akka.Tests.Delivery.TestConsumer+Job]] with generation [160]
[DEBUG][01/03/2024 21:24:52.641Z][Thread 0032][akka://test/user/consumer-controller-1] Retry sending Request with confirmedSeqNr [7], requestUpToSeqNr [17]
[DEBUG][01/03/2024 21:24:52.641Z][Thread 0032][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]
[DEBUG][01/03/2024 21:24:52.641Z][Thread 0032][akka://test/user/producer-controller-1] Expanded requestedSeqNr from [17] to [31], because current [21] and all were probably lost.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0036][CoordinatedShutdown (akka://test)] Performing phase [before-service-unbind] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [service-unbind] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [service-requests-done] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [service-stop] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [before-cluster-shutdown] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [cluster-sharding-shutdown-region] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [cluster-leave] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [cluster-exiting] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [cluster-exiting-done] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [cluster-shutdown] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.154Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [before-actor-system-terminate] with [0] tasks.
[DEBUG][01/03/2024 21:24:53.158Z][Thread 0037][CoordinatedShutdown (akka://test)] Performing phase [actor-system-terminate] with [1] tasks: [terminate-system]
[DEBUG][01/03/2024 21:24:53.158Z][Thread 0037][ActorSystem(test)] System shutdown initiated
[DEBUG][01/03/2024 21:24:53.161Z][Thread 0003][ActorSystem(test)] Cancel all timers
[DEBUG][01/03/2024 21:24:53.161Z][Thread 0037][ActorSystem(test)] Cancel all timers
[DEBUG][01/03/2024 21:24:53.162Z][Thread 0007][ActorSystem(test)] Cancel all timers
[DEBUG][01/03/2024 21:24:53.163Z][Thread 0034][ActorSystem(test)] Cancel all timers
[DEBUG][01/03/2024 21:24:53.166Z][Thread 0017][EventStream] Shutting down: StandardOutLogger started
[DEBUG][01/03/2024 21:24:53.166Z][Thread 0017][EventStream] All default loggers stopped

@Aaronontheweb
Copy link
Member Author

I think this is the key line:

[DEBUG][01/03/2024 21:22:55.932Z][Thread 0033][akka://test/user/producer-controller-1] Received request, confirmed [7], requested [17], current [21]

In this state, I don't believe the Producer can do a redelivery of any un-ACKed messages, so the states are getting messed up somewhere between the producer and consumer contorllers.

Aaronontheweb
Aaronontheweb commented Jan 4, 2024
View reviewed changes
Copy link
Member Author

@Aaronontheweb Aaronontheweb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Detailed my changes

src/core/Akka/Delivery/Internal/ProducerControllerImpl.cs
newRequestedSeqNr, newRequestedSeqNr2, stateAfterAck.CurrentSeqNr);

if (newRequestedSeqNr > CurrentState.RequestedSeqNr)
if (newRequestedSeqNr2 > CurrentState.RequestedSeqNr)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the bug - we need to evaluate newRequestedSeqNr2 here for when the producer believes that the consumer has lost data (and re-delivery isn't enabled). This is what was causing this test to get stuck and probably some of the others in this suite.

@Aaronontheweb Aaronontheweb changed the title Troubleshoot ReliableDeliveryRandomSpecs Akka.Delivery: fix ProducerControllerImpl<T> state bug Jan 4, 2024
@Aaronontheweb Aaronontheweb added the akka-delivery Akka.Delivery APIs label Jan 4, 2024
@Aaronontheweb Aaronontheweb added this to the 1.5.15 milestone Jan 4, 2024
@Aaronontheweb Aaronontheweb marked this pull request as ready for review January 4, 2024 14:17
Arkatufus
Arkatufus approved these changes Jan 4, 2024
View reviewed changes
Copy link
Contributor

@Arkatufus Arkatufus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Aaronontheweb Aaronontheweb merged commit 1c4f071 into akkadotnet:dev Jan 4, 2024
8 of 12 checks passed
@Aaronontheweb Aaronontheweb deleted the fix-racy-tests branch January 4, 2024 15:42
@Aaronontheweb Aaronontheweb mentioned this pull request Jan 10, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Arkatufus Arkatufus Arkatufus approved these changes

Assignees
No one assigned
Labels
akka-delivery Akka.Delivery APIs
Projects
None yet
Milestone
1.5.15
2 participants
@Aaronontheweb @Arkatufus