Orchestration activities stuck not completing retry attempts, SB showing timer events for both.

[devinlstoen]

Hello, I'm trying to get a better understanding why these two orchestration are not completing their activity retries. The orchestrations are scheduled for 9 total retries of the RetryCreateSupportTicketActivity, but the logs are only showing that this activity has tried 6 times. The 9 retries are supposed to take ~9 hours to complete, but it looks like the last attempt was made on 1/25 19:03:32

Here's what were seeing in Service Bus. There are two timer events corresponding to the orchestrations. They look to be stuck.

[cgillum]

I don't think there's enough information here to provide help. Can you share the snippet of code that shows how you're configuring your retry policy?

[devinlstoen]

Sure, line 284 is where we are defining the number of retries for the RetryCreateSupportTicketActivity. [cid:12af285b-55cc-48f6-a92e-c302d4004c4d]

…

________________________________ From: Chris Gillum ***@***.***> Sent: Wednesday, February 7, 2024 10:56 AM To: Azure/durabletask ***@***.***> Cc: Devin Stoen ***@***.***>; Author ***@***.***> Subject: Re: [Azure/durabletask] Orchestration activities stuck not completing retry attempts, SB showing timer events for both. (Issue #1035) I don't think there's enough information here to provide help. Can you share the snippet of code that shows how you're configuring your retry policy? — Reply to this email directly, view it on GitHub<#1035 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZGG27GMKBWLDDQOTYQRDLDYSPE4VAVCNFSM6AAAAABCTRPGPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSGY3TKOJUG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[devinlstoen]

Then on line 73 we're constructing the retry options for this activity: [cid:22b01918-a9c8-41ed-add7-d0622f342f1c]

…

________________________________ From: Devin Stoen ***@***.***> Sent: Wednesday, February 7, 2024 11:24 AM To: Azure/durabletask ***@***.***>; Azure/durabletask ***@***.***> Cc: Author ***@***.***> Subject: Re: [Azure/durabletask] Orchestration activities stuck not completing retry attempts, SB showing timer events for both. (Issue #1035) Sure, line 284 is where we are defining the number of retries for the RetryCreateSupportTicketActivity. [cid:12af285b-55cc-48f6-a92e-c302d4004c4d]

________________________________ From: Chris Gillum ***@***.***> Sent: Wednesday, February 7, 2024 10:56 AM To: Azure/durabletask ***@***.***> Cc: Devin Stoen ***@***.***>; Author ***@***.***> Subject: Re: [Azure/durabletask] Orchestration activities stuck not completing retry attempts, SB showing timer events for both. (Issue #1035) I don't think there's enough information here to provide help. Can you share the snippet of code that shows how you're configuring your retry policy? — Reply to this email directly, view it on GitHub<#1035 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZGG27GMKBWLDDQOTYQRDLDYSPE4VAVCNFSM6AAAAABCTRPGPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSGY3TKOJUG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jviau]

@cgillum , I chatted with @devinlstoen offline and I think something of importance is they have not provided an IOrchestrationServiceInstanceStore or IOrchestrationServiceBlobStore to ServiceBusOrchestrationService (they supply null). I think this would be problematic for ServiceBus would it not? It looks like it at minimum expects a non-null IOrchestrationServiceBlobStore