-
Notifications
You must be signed in to change notification settings - Fork 36
ServiceBus Session support #16
Comments
Can you detail more on the scenarios you're thinking of for it, and how you'd like it to work? The new extensibility model we're opening up would allow you to write one yourself if you wanted to. However, if this is something that many people would find useful, it would make sense for us to bake into our core ServiceBus assembly. It's not clear to me whether this would need to be a completely new trigger binding, or whether on the trigger/receive side of things it couldn't just be an additional mode on the existing ServiceBusTrigger binding. Behind the scenes, it polls for and accepts sessions, and invokes your job function for session messages. As far as sending session based messages, I think our ServiceBus binding (e.g. used when sending a BrokeredMessage via out param binding) would work as is? You just set your session ID on the message before sending. |
I'm going to use it for handling the messages of one session in a specific sequence and preferably multiple sessions at once. Since it's currently not possible, I'm going to use this article to process multiple sessions per webjob. public static void ProcessQueueMessages([ServiceBusTrigger(queueName:"inputqueue", useSessions: true)] List<string> messages) { ... } Or whatever type of object you expect. Also you would need to specify somewhere in the config the timeout between sessions; after how many (milli)seconds will you assume the session is finished? And you are correct, for sending messages you just need to set something in the SessionId. |
I'd love to see this feature as well. I need my inbound messages to be in a guaranteed delivery order, and it seems using the Service Bus Session is the way to do this. However it is not at all clear how I would implement this in a WebJob. |
Not having this is a blocker for me currently. We're using queues + the webjobs SDK as a very lightweight, scalable reactive work item list - messages come on, and processing those messages leads to 0 .. M new messages going back onto the queue. When we need to have guaranteed sequential processing - almost simulating an actor-style model - service bus sessions is the only option. This is crucial if we have some critical region of data processing and don't want e.g. transactions on SQL database (or can't e.g. Azure blobs). Having an easy way to target messages on a specific session and have them work transparently within webjobs would be a huge benefit and would open a ton of possibilities. So on the input side I would love to see it "just work": -
|
I need this feature as well. We have a WebAPI call that's taking longer and longer as we add more work to each call. When it started to timeout in Azure, we decided to move the process into a WebJob so that clients can initiate the task, then check back to get progress. The clients in this case are browsers making AJAX calls. If we can't correlate the requests with the responses, they can't check on progress or completion of the WebJob. We started looking at Azure Storage Queues, but found that there's no way for the original caller to only receive responses intended for them. We can't create a response queue for each client, because each client is a browser. We saw that Service Bus Queues and Topics allow us to use the SessionId to correlate requests and responses. The client (browser) can filter the response queue to only see responses intended for them. It sounded like a perfect solution. After wrestling with configuration we got the JobHost.RunAndBlock() to almost run, but we get the following error: It is not possible for an entity that requires sessions to create a non-sessionful message receiver. We found this documentation at http://www.cloudcasts.net/devguide/Default.aspx?id=13029 If sessions are required on a queue it is not possible to use the QueueClient Receive method to dequeue messages. Attempting to do so will result in. InvalidOperationException It is not possible for an entity that requires sessions to create a non-sessionful message receiver. We assume the WebJobs SDK is using the QueueClient Receive method. We're going to try to work around this by having the request queue send a Guid as part of the payload and not require Sessions on the request queue. We'll require sessions on the response queue and see if we can use the Guid in the payload as the SessionId for the response. Will let you know if that works. |
One thing I've noticed since my OP is that you can use |
FYI, the work-around mentioned above worked perfectly: |
I would really like to see this implemented as well. I don't use the WebJobs SDK directly that much, but if I understand it correctly this would allow for Azure Functions written in for example JavaScript to handle message sessions (something that's not supported in the Node.js Azure SDK since it uses the REST interface of Azure Service Bus). |
Was there any traction on this? |
+1 |
@amit-kumar2 -- is this the issue that you brought up to me yesterday? If so, would you want to weigh in with your scenario here so we have it captured? |
HI, i am using below code with Servicebus and Topic-Subscription with session enabled public static void ProcessTopicMessage([ServiceBusTrigger("%ServicebusTopic%", "%ServicebusSubscription%")] BrokeredMessage message, TextWriter logger) but the problem i face with continuously running web job is host.RunAndBlock(); throws below error so how to resolve this issue? Azure web job does not support Session enabled topic-subscription? An unhandled exception of type 'System.InvalidOperationException' occurred in mscorlib.dll Additional information: It is not possible for an entity that requires sessions to create a non-sessionful message receiver. TrackingId:074fbc74-b5e9-4fc2-886b-ed2f4e324304_G5_B15, SystemTracker:az-g3ms-servicebus:Topic:az-rsd-dev-topic|az-rsd-dev-subscription, Timestamp:5/11/2018 5:16:14 AM |
Hi @appalaraju were you able to resolve your issue?. In my case I'm not getting any exceptions, but the web job is not being fired either. If I disable sessions in my queue the web job starts working again. |
As far as we have been able to determine, we simply have to accept that the queue will throw exceptions on occasion with no discern-able reason so we just have to handle and ignore it. As far as not working when sessions are enabled, we have not encountered that. Have you tried turning on diagnostics in the portal? |
Related to background errors coming from MessageReceivers - we just made some improvements in that area: Azure/azure-webjobs-sdk#1703 |
Hi @raulbojalil,MikeYeager Have you tried with Topic + Session enabled Subscription? in my case, i have created a web job and used the code(ProcessTopicMessage) which i specified above with Topic and session enabled subscription and ran it locally( in my machine). i am facing the issue which i specified above. any idea on when can Azure web job works properly with Topic + Session enabled Subscription? |
Warning: This horse is dead 🐴! You need to process your messages in FIFO style? Better use some other technology and stop waiting for Microsoft to support their own technology stack! |
If Microsoft cannot prioritize this, would anybody else be interested in attempting to put together a PR for this? I am not sure I can do it alone, but I'm happy to take a stab at it. |
It'd be good to have this feature :( |
So it's been over 2 years since this issue was raised and still nothing on whether/when this will be addressed. Please vote for it at https://feedback.azure.com/forums/355860-azure-functions/suggestions/13882992-support-for-session-enabled-azure-service-bus-queu |
This is a limitation for us, any eta to implement this ? |
@ridhoq If you want I join the community support for this |
On October 18th 2018 a comment was added to https://feedback.azure.com/forums/355860-azure-functions/suggestions/13882992-support-for-session-enabled-azure-service-bus-queu to say the work has been planned... so there is hope! |
Assigning this to sprint 45 for initial design and scoping |
Design is ready for internal review. Will be reviewing and planning in Sprint 47. When finalized, we'll respond back here. |
Design work completed. Moving this to sprint 48 to track the implementation |
Pre-release nuget package is out. Change Using:
Also you can specify new
I'll appreciate for any feedback. |
Moving this to sprint 49 to allow more time for feedback before this work is merged. |
Is this feature available in Azure functions? |
Hi @alrod, I am doing a proof of concept of using ServiceBus session feature in one of our Azure Functions. The Azure Function that I am trying to enable ServiceBus session has AutoComplete set to false and the function manually Complete, DeadLetter or Abandon depending on the outcome. When use IMessageSession Complete, DeadLetter or Abandon I exeperiece intermittent SessionLockLostException under load and evntually succeed. But if I set AutoComplete to true I don't see this error at all. This is the only issue we have faced so far blocking us using this feature. Are you able to shed some light this issue? Thanks |
Hi @alrod, Question: Service Bus Explorer has a queue option "Enforce Message Ordering". What is this option for? Should it be set along with enabling sessions if we want to guarantee processing in the order received? |
@alrod Still aiming for a release in this month? |
Are public static async Task Run([ServiceBusTrigger("name", Connection = "connection", IsSessionsEnabled = true)]Message message, ILogger log)
{
await doThingAsync();
} As soon as the debugger hit the async call in the code, the :EDIT: Nevermind. I was debugging and the lock was expiring, causing the message to be grabbed again. |
I'm testing this session capability and am getting frequent occurrences of the error "The session lock has expired on the MessageSession". Can anyone explain why this might be happening? When I use these options:
then when the error occurs, it stops all processing of my Azure function - the only recourse seems to be to restart the function. I'm assuming that's occurring because I have When I remove those SessionHandlerOptions and just let everything be defaults, I still get the error, but processing is still able to continue. I'm assuming that's because while that function instance is dead in the error state, processing can continue using another instance. If I'm right, then that's only a temporary solution as I will still run out of instances once they all hit that error. So I really need to figure out why the error is occurring in the first place and how to prevent it. Anyone have any ideas? |
Sessions support is released in https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.ServiceBus/3.1.0 |
@mpnow, Default lock session period is 30 secs (check Regarding stopping processing if |
@dinukajayamuni, please see #16 (comment). I am getting
|
Hi @alrod , Thanks for the above info about the session lock timeout. I'm still having troubles, though. No matter what I set I notice there's a |
Hi @alrod, static DeviceClient deviceClient; i am using IOT Hub endpoints and MessageRouting to forward messages from device to required topic here "Message" class belongs to Microsoft.Azure.Devices.Client so i am unable to keep "SessionId" for this "Message" class. even i keep below statement to keep "SessionId" for message, session concept is not working after reaching this message to topic could you please let me know how to apply "SessionId" to above scenario? |
@appalaraju , I am not aware how to sent |
@mpnow, can you please upload the project with repro so I can take a look. |
Thanks @alrod |
@alrod I experience the same behavior. No matter |
@pzbyszynski, |
@alrod Your sample works indeed. Nevertheless, in my production code I have my function set up in the following way:
and I have still tons of Any input on that would be highly appreciated @alrod |
@alrod |
I can confirm this. Any update on this issue? |
Options and descriptions sourced from Azure/azure-functions-servicebus-extension#16. Defaults determined experimentally by reading the Function Runtime logging.
@cgrevil, @buyckkdelaware. I will take a look on the issue during next sprint: |
The feature is already released. |
Guys, I need some suggestions on implementation of concurrent sessions. if patient_2 completes first then it should pick the next message-id and process ..... They all should execute parallel and once they finish executing their respective message then it should pick up the next one and process accordingly. Any comments or suggestions are welcome. |
Hello, I am facing the issue with the Microsoft.Azure.ServiceBus.SessionLockLostException as well. From what I found even though you are increasing the MaxAutoRenewDuration it won't help. The thing you need to increase is the Subscription's LockDuration(by default 30 seconds), but the maximum allowed value is 5 minutes. When I try to increase it over that value I get the following exception: I am using ServiceBusExplorer to manage my topics and subscriptions. What you have to do is manually renew the session lock before it expires, so you don't get the same message from the queue with the new lock while you are still processing the first arrived one. But the IMessageSession.LockedUntilUtc property returns a value that is a bit larger from what I have set to the LockDuration. So for the moment I have to hardcore it and don't depend on the IMessageSession properties. |
Can you provide more detail as to exactly what code you're executing to do the above? It would be much appreciated. |
So in the handler I have registered with SubscriptionClient.RegisterSessionHandler(), I do the following: ` var timeout = 25 * 1000;
I dispose the timer also when I complete or sent the message to the deadletter queue. As the service I am working on needs to scale I have to renew it manually, otherwise if the AutoRenewSessionLock kicks in, another instance may pick up the message or the same instance will start processing the same message again even though such is in progress. |
You are absolutely right. After hours of effort, I saw that the team was still in another realm. |
I've posted this question yesterday on StackOverflow but I guess this is a better place to ask; is it possible to have something like ServiceBusSessionTrigger for handling session through the webjobs sdk?
The text was updated successfully, but these errors were encountered: