New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Singleton support for Functions to ensure only one function running at a time #912

Open
mathewc opened this Issue Nov 11, 2016 · 68 comments

Comments

Projects
None yet
@mathewc
Contributor

mathewc commented Nov 11, 2016

We should discuss whether we want to bring this functionality forward for Functions. However, it will be relatively simple for us to do - we just need to expose new properties via function.json for function level singletons. See here for singleton doc.

In addition to Function level singleton support, we might also consider supporting Listener level Singleton as well.

When we do this, we should make sure it works across multiple languages (not just C#)

@lindydonna lindydonna added the feature label Nov 14, 2016

@lindydonna lindydonna added this to the Next - Triaged milestone Nov 14, 2016

@vladkosarev

This comment has been minimized.

vladkosarev commented Feb 20, 2017

We would love to have that. Let us specify singleton in different scopes just like in web jobs.

@mathewc

This comment has been minimized.

Contributor

mathewc commented Feb 22, 2017

Thanks @vladkosarev. This issue has been on the back burner, since not many people have been asking for it yet. What specifically is your scenario?

@vladkosarev

This comment has been minimized.

vladkosarev commented Feb 22, 2017

Being able to process messages from multiple queues in series as long as they affect the same entity. So if we have two queues and both of them will affect Order/1 (which is encoded in queue messages) we want the two different functions to kick in in series and process messages from those two queues one by one instead of in parallel. The idea here is having serialized access to data storage for a particular entity.

@ajukraine

This comment has been minimized.

ajukraine commented Feb 23, 2017

@vladkosarev sounds like actor model. I would love to see how efficient it's with using scoped singleton and Azure functions. Another option is to use Service Fabric, but it's huge piece of infrastructure to be handled. Prefer to start with something as lightweight as Azure functions

@vladkosarev

This comment has been minimized.

vladkosarev commented Feb 23, 2017

That is exactly what I'm trying to achieve. Actor-like model using Azure functions. Now that they announced that functions will support serverless framework this might not be as important but I'd still like this ability in the 'raw'. Service Fabric is great but it's still not serverless. I want to pay for compute resources not for VMs. Consumption functions + actors on top is my path to nirvana. Obviously you can't have in memory state, etc but it would still be a good start to the path of properly infinitely scalable architecture.

@WonderPanda

This comment has been minimized.

WonderPanda commented Apr 21, 2017

@lindydonna Hey I'm just wondering if there are any updates in regards to when we might expect support for singleton functionality with Azure Functions. Based on the conversation here it seemed like it might be low hanging fruit. It would be extremely helpful to be able to specify singleton behavior in function.json

@tmakin

This comment has been minimized.

tmakin commented Apr 30, 2017

I have use case where this would very helpful. I have long running background processing tasks which are triggered via a storage queue. In order to prevent spikes in the database load, I need to ensure that that there is not more than one queue item being processed at a time.

My current solution is to use a timer trigger with a short interval to manually poll the queue, but a singleton flag for queue triggers would be a much tidier option.

@ericleigh007

This comment has been minimized.

ericleigh007 commented Jun 22, 2017

Yes, I didn't know what this was from the title. Perhaps rename the report to something more descriptive.

My case is just about the same. Would like to guarantee only one queue function running at a time. NOTE with the current time limitations we cannot just wait.

-thanks Donna
-e

@lindydonna lindydonna changed the title from Add Singleton support for Functions to Add Singleton support for Functions to ensure only one function running at a time Jun 22, 2017

@alohaninja

This comment has been minimized.

alohaninja commented Aug 3, 2017

Seems like you could support locking on your own - you just need shared storage backing it - SQL Azure, Azure Blob/Table/Queue, Redis, etc. Would be great just to add a [SingletonAttribute] to our Azure Functions like webjobs has. host.json has configurable options for singletons, but I don't know if it supports Azure functions (AF) versus Web Jobs. Could just add an environment key which has the storage connection string etc, and assign it in our host.json.

@alexjebens

This comment has been minimized.

alexjebens commented Aug 3, 2017

@lindydonna any updates on a timeline for this feature?

I have a couple of Projects where I would like to switch from WebJobs to Azure Functions as well as some new Projects that need serial processing for queues, which require this functionality.

From my understanding of the way this works for WebJobs is a lock blob with a lease in the storage accounts. Azure Functions appear to already use this mechanism for the Timer-Trigger.

@alexjebens

This comment has been minimized.

alexjebens commented Aug 3, 2017

@alohaninja Supporting locking on our own is not trivial. e.g. In a Queue-Trigger Function you could only throw an exception so that the message is put back in the queue, this may however lead to the message being marked as poison and therefore lost if you cannot process it in time. Additionaly the Function will still be invoked leading to extra costs.

According to this issue there is currently no support for the Singleton Attribute in Azure Functions and the host.json options are therefore moot.

Possible Workarounds:

  • Timer-Trigger Functions appear to run as Singletons. They produce a blob in the storage account under "locks/" and create a lease. This requires implementing the input on your own.
  • ServiceBus-Trigger Functions with serviceBus.maxConcurrentCalls to 1 in host.json. This is however a global setting and I would like use this on a per function basis.

@lindydonna is it possible to confirm that Timer-Trigger Functions run as singletons?

@alohaninja

This comment has been minimized.

alohaninja commented Aug 3, 2017

@aboersch - came here because we are using Timer-Trigger functions and they can run simultaneously, I was looking for a way to ensure you cannot have concurrent timer events - seems to occur during app restart (DLLs change in /bin, app restart via portal).

Configured function.json to ignore trigger firing @ restarts via "runOnStartup": false, but we still see concurrent executions if a function was running before the cycle. Seems like the locking doesn't account well for function updates (restart events) - it fires the trigger even though an existing process is already running. To verify this - use kudu process explorer and you'll see multiple processes for the same function.

For now - I just use kudu process explorer to kill any existing processes before making any app updates or restarting the function host - would be great if the portal allowed you to kill the process for a running azure function.

@alexjebens

This comment has been minimized.

alexjebens commented Aug 3, 2017

@alohaninja I am aware of the troubles with the restart. I usually stop the app before updating because of it, however all my functions are designed to be interrupted at any time.

If you look into your storage account you will see a container called azure-webjob-hosts. There will be several folders here {hostname}-{random-number} which contain a host.Functions.{function-name}.listener file for each timer-trigger function. This file is being used to lock with a blob lease.

Every time your app is (re)started a new folder is created ({hostname}-{random-number}). Since the new folder is empty there is no blob and no lease to check for, hence the parallel execution.

This should perhaps be a separate issue though.

@rossdargan

This comment has been minimized.

rossdargan commented Aug 26, 2017

I could really do with this feature to. The issue I have is that I need to call an external api based to get more data any time a message gets added to a queue. I tend to get around 300+ messages over a few minutes every 8 hours. The issue I'm having is azure spins up 16 servers to handle the spike in messages (which is cool...) however this is utterly destroying the server I'm calling.

I've set batch size and singleton in the host.json but that appears to have no impact (setting batch size just results in more servers being started).

@BowserKingKoopa

This comment has been minimized.

BowserKingKoopa commented Sep 1, 2017

I'm in desperate need of this as well. I have a bunch of webjobs I'd like to move to Azure Functions, but I can't because they need to run as singletons. Some are queue based and need to be run in order. Others call external apis that are very sensitive about how rapidly I call them.

@rossdargan

This comment has been minimized.

rossdargan commented Sep 1, 2017

To work around this I made a buffer queue and use a scheduled function to see how many items are in the processing queue and move a few items over depending on the count.

@WonderPanda

This comment has been minimized.

WonderPanda commented Sep 28, 2017

@BowserKingKoopa @rossdargan I haven't had time to experiment with it yet but in the configuration settings for host.json there's an option for WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT. Sounds like it's still being worked on but using this in conjunction with batch size might help you achieve singleton semantics for the meantime.

@bwwilliam

This comment has been minimized.

bwwilliam commented Oct 5, 2017

Any updates on this please? Has anyone found a way to make enforce Functions run as a single instance max.

@KeenFann

This comment has been minimized.

KeenFann commented Oct 11, 2017

We also have a need for this. Both singleton and scale out limitation at function level.

@ajukraine

This comment has been minimized.

ajukraine commented Oct 11, 2017

I guess, if you can't wait for it to be implemented in SDK, you can use it already in Durable Azure Functions (https://docs.microsoft.com/en-us/azure/azure-functions/durable-functions-overview)

See section Stateful singletons

@AntonChernysh

This comment has been minimized.

AntonChernysh commented Oct 12, 2017

+1.
Use case: Simple slack bot event handler, should only send message once.

@WonderPanda

This comment has been minimized.

WonderPanda commented Oct 12, 2017

@AntonChernysh I think you might be confused about Singleton behavior... There is nothing today preventing you from building a slack bot that only responds to messages only once

@AntonChernysh

This comment has been minimized.

AntonChernysh commented Oct 12, 2017

@WonderPanda looks like my function is scaling and running multiple times, therefore getting reply as many times as functions started. I have the same function (python 3.6) running on AWS lambda with no problem.
I'd appreciate If you could advice something to make function run only once.

@WonderPanda

This comment has been minimized.

WonderPanda commented Oct 12, 2017

@AntonChernysh What triggers your function?

@bwwilliam

This comment has been minimized.

bwwilliam commented Oct 12, 2017

Any idea when singleton can be made available please? I'm implementing CQRS pattern with functions. My event publisher needs to be singleton so it can process the events in the right order/sequence. Thanks

@AntonChernysh

This comment has been minimized.

AntonChernysh commented Oct 12, 2017

@WonderPanda post message to function's HTTPs endpoint. Can we continue in skype? anton.chernysh

@SimonLuckenuik

This comment has been minimized.

SimonLuckenuik commented Jun 28, 2018

@mathewc, @jeffhollan are there any plan to support Singleton with Consumption plan at all?

Can you confirm that the only impact of [Singleton] in a consumption plan is around billing (being charged while waiting for the singleton lock to be released)? (Not sure I understand why the Blob lease approach would not work specifically in consumption plan).

I would love paying to wait if it provides me with an easy/ready to use Singleton pattern. Otherwise it means, re-implementing the Blob lease approach that was already implemented by for Webjobs, documented here and executing it from my Function.

This Singleton topic has been floating around for a while, maybe officially testing it and supporting it would remove some confusion and enable new scenarios. Functions is built over Webjobs, so initial assumption by developers is probably that if it works in Webjobs, it will work for Functions.

@jeffhollan

This comment has been minimized.

Member

jeffhollan commented Jun 28, 2018

Longer thread and want to make sure I understand and also share what may be possible today. From basic understanding of thread it sounds like the most common scenario is something like "I want to process queue items 1 by 1 because queue message A needs to be completed before queue message B starts."

We do allow you to control concurrency to a fair degree per instance with the host.json file. Here you could set something like Service Bus max concurrency to 1 with 0 prefetch so each instance was only processing 1 message at a time (obviously with impacts to throughput - as per singleton as a whole).

The one piece that the host.json doesn't control is the concurrency across all possible instances. If you do something like drop 1000 messages in a queue it's very likely we will scale out a few instances as competing consumers to help drain the queue. There is the WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT app setting that does let you limit scale out. It's not perfect, but for small values (like less than 5) it should work well. We also have a work item to make this work for any value.

That said I think if you set the correct host.json value and WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT=1 you would only have 1 instance processing 1 message at a time.

Even better - @SimonLuckenuik you mentioned you aren't too worried about cost. If you don't use a consumption plan but deploy to app service plan you can fully control scale and fix your function at 1 instance reliably. So you could always do host.json + 1 instance in an app service plan and get this too.

Not sure if a scenario here that a mix of host.json / instance restrictions won't solve for though so let me know. Also worth noting - storage queues don't guarantee ordering so if your scenario is impacted by ordering may want to consider something like service bus.

@SimonLuckenuik

This comment has been minimized.

SimonLuckenuik commented Jun 29, 2018

@jeffhollan, thank you for sharing in detail those different options, I didn't know about the WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT option. What I am really looking after is a cross-instances locking mechanism, to enable atomic startup/creation of Orchestration with Durable Functions.

The advantage of the Singleton attribute vs what you are proposing is that the Singleton impact is very low and localized: no need to have a special "non scalable" Function App for a very specific scenario. With the Singleton attribute, I simply say that I want this Function to be singleton and it works automatically!

The price gap is high between paying to wait for a lock to be released in Consumption plan for a specific Function vs going to App Service Plan running 24/7 on which I will deploy a single Function.

Today, what is the impact of using the Singleton attribute is Consumption plan is it just ignored or not ignored and not working?

@spzSource

This comment has been minimized.

spzSource commented Jul 14, 2018

Hi all, @jeffhollan, @mathewc, @alrod
Reading whole discussion didn't give me a clear understanding about does singleton function (Singleton attribute) officially supported for Azure Function under App Service plan.

Could you clarify can we use [Singleton] attribute for Azure Function under App Service plan in production?

@BowserKingKoopa

This comment has been minimized.

BowserKingKoopa commented Jul 25, 2018

Any update on this? According to Azure/azure-webjobs-sdk#1555 WebJobs as it's own thing is all but abandoned and exists only to serve Azure Functions. And Azure Functions is still missing this basic functionality (needing to process queue items one by one in order is a very common occurrence for me). I'm stuck between abandoned WebJobs and incomplete Azure Functions and I really need to be able to move forward on this. It's been years now.

@jeffhollan

This comment has been minimized.

Member

jeffhollan commented Jul 27, 2018

I'm working to understand current behaviors fully but as called out here it appears Singleton should "just work" for pre-compiled C# functions today, and this specific issue is more about making it possible from other languages (Node, Java, .csx). There is the unknown of if there may be billing implications (time spent waiting for lock to release may be billed) for pre-compiled C# though. I'll work to confirm this though and apologies for confusion (on my part as well) on supportability of this feature

@WonderPanda

This comment has been minimized.

WonderPanda commented Jul 27, 2018

@Schaemelhout

This comment has been minimized.

Schaemelhout commented Aug 17, 2018

In the meantime, would it be a good idea to have 1 queue (A) for dumping a bunch of messages while having another queue (B) to use as the actual trigger?
I could process the messages 1 by 1 by dequeuing a message from (A) and queuing it on the 'trigger' queue (B) when a message has been processed.

@christian-vorhemus

This comment has been minimized.

Member

christian-vorhemus commented Aug 17, 2018

@jeffhollan, I tried to follow your suggestions, however I experience some difficulties. Here is the scenario I'm working on: I'm quickly loading thousands of images into a Storage Blob (A). An Azure Function (AF) gets triggered by a BlobTrigger and adds the Blob image path enriched with some additional information into a ServiceBus Queue (C). Another AF (D) started by a ServiceBusTrigger fetches the messages from the Queue and calls a Third-Party Web API (E) which is very slow. Finally, the results of (E) are stored in (F)

architecture

It is necessary to throttle (D) to not overload (E). However, (D) gets flooded by messages from the Queue Trigger, even though I set maxConcurrentCalls to 1 in host.json and WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1. (D) approximately takes 2 minutes to finish since the call to (E) takes a while.

Maybe my architecture is wrong and there is a clever and easy way to do this?

Edit:
Using the [Singleton] attribute on the AF does throttle the function execution to on after another. However, I'm confused why the maxConcurrentCalls/WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT way is not working?

@simeyla

This comment has been minimized.

simeyla commented Aug 23, 2018

Ugh. So why is it called a QUEUE then.

Should be called a MassiveDumpOfStuffThatNeedsDoingInNoParticularOrder

It's always the simplest of concepts that become the most complex - every time.
I should know this by now.

@simeyla

This comment has been minimized.

simeyla commented Aug 23, 2018

@jeffhollan do we know if Singleton guarantees FIFO order? I suspect that effectively it would (certainly for my needs it would), but I'm not sure if it is guaranteed.

@spzSource

This comment has been minimized.

spzSource commented Aug 23, 2018

@simeyla There is no guarantees for FIFO order even using Singleton attribute. The only way is to set queue.batchSize to 1 along with Singleton attribute.

@simeyla

This comment has been minimized.

simeyla commented Aug 23, 2018

###PSA: Singleton does not work for consumption plan. It just fails silently.

Yes, others in this thread have mentioned it but it's easy to miss.

@BowserKingKoopa

This comment has been minimized.

BowserKingKoopa commented Aug 23, 2018

I use [Singleton] (with batchSize = 1) in WebJobs a lot to process only one message at a time. @simeyla is right. You can't actually use the Azure Storage Queue as an actual Queue (FIFO) unless we get [Singleton] working in Functions.

@jeffhollan

This comment has been minimized.

Member

jeffhollan commented Aug 28, 2018

Few things as I just tested this out. first thing I noticed:

  1. [Singleton] does not appear to work when scaled out. I dropped 1000 items in a queue and after it scaled out to a few instances some instances would be processed at the same time.
  2. It should be noted that Azure Storage Queues don't guarantee ordering regardless of tech you have consuming.

I believe in the interm the only way I can think of this that would work is setting batch size to 1 and setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1. Possible setting batch size to 1 and singleton have same impact, however I don't know if singleton would introduce higher billing as singleton does dequeue a bunch of messages, just holds them for a long time - which holding time I can't confirm if billed or not.

@christian-vorhemus I see you said the above (WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT + batch size) wasn't working but not sure how. The queue getting flooded sounds expected, the main thing is that D is taking ages because you are only processing one item at a time.

As an aside 2 features we are looking at that should help with this longer term:

  1. A more reliable way to limit scale out to just 1 instance (WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT has some quirks, especially if set to a value > 5)
  2. Look into if we can support service bus Sessions. This could enable faster processing across multiple instances while still preserving order.
@spzSource

This comment has been minimized.

spzSource commented Aug 28, 2018

[Singleton] does not appear to work when scaled out. I dropped 1000 items in a queue and after it scaled out to a few instances some instances would be processed at the same time.

@jeffhollan Have you tried [Singleton(Model = SingletonMode.Listener)]? Documentation says that using this mode, the listener will only be running on a single instance (when scalled out).

So, as far as I understood, in case scalled out env function definition must be the following:

[Singleton(Mode = SingletonMode.Function)]
[Singleton(Mode = SingletonMode.Listener)]
public static void Run()
{
}
@jeffhollan

This comment has been minimized.

Member

jeffhollan commented Aug 28, 2018

Thanks - quick correction. I was using some pre-release bits and that appears to be the issue. When I tested Singleton with prod bit it "just worked" -- at least for me. I need to try to repro cause others have mentioned it didn't work for them but for me right now it appears to be ok. Still not sure on the billing effect though

@SimonLuckenuik

This comment has been minimized.

SimonLuckenuik commented Aug 29, 2018

Thanks @jeffhollan these are good news!

@jeffhollan

This comment has been minimized.

Member

jeffhollan commented Sep 8, 2018

All right did another round of investigation. Here's the answer:

  • [Singleton] does work on functions. The Azure Function host will create or wait for a lock in the Azure Storage account. The lock is the host ID which should be the same for all hosts of an app across all instances - so all instances share this lock and will only allow one execution to occur at a time.

To test this I put 1000 queue messages at once on a function with [Singleton]. The function would wake up, emit the invocation ID, sleep, and then emit the invocation ID. After processing all 1000 I looked at logs and never saw invocation IDs overlap. Only one invocation would happen globally at a time.

HOWEVER - in investigating the billing behavior, executions that start and then wait for lock to release are emitting billing meters while waiting. So if a host dequeues a batch of messages (default for Azure Queues I believe is 16), one execution will start and create a lock. The other 15 executions will start but immediately see the lock is taken and wait to try to grab the lock. While waiting they are emitting bills and the GB*sec are adding up. So you will be paying for that idle "wait" time on an instance.

With that said I think the recommendation is: [Singleton] isn't recommended for consumption hosted function plans. If you have a dedicated app service plan it's fine (as you are paying for the instance anyway). If you want to enforce [Singleton] like behavior in a consumption plan you are likely best to:

  1. Set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 so you never scale to more than one instance
  2. Set the host.json file to only allow 1 concurrent execution at a time for that trigger (for instance a batch size of 1 for Azure Queues).

I suspect longer term if we could support something like Service Bus Sessions it could allow for both high throughput across competing consumers and in order enforcement (Event Hubs allows this today).

@spzSource

This comment has been minimized.

spzSource commented Sep 8, 2018

@jeffhollan thanks for exhaustive answer. Can we assume that Singleton functionality will not be removed in future releases?

@SimonLuckenuik

This comment has been minimized.

SimonLuckenuik commented Sep 8, 2018

@jeffhollan I am not sure to understand why the recommendation is not to use it for consumption plan, if it is explicitly said in the documentation that you pay as soon the wait starts. I understand that you might have to keep things in mind for your scenario to see if the billing model fits you (like your 16 messages example), but paying for wait time is still cheaper compared to dedicated plan in a lot of scenarios. For example, we have low concurrency HTTP trigger scenarios where a singleton lock is a great protection to prevent overlapping operations where transactions are not available.

@jeffhollan

This comment has been minimized.

Member

jeffhollan commented Sep 8, 2018

@SimonLuckenuik I suppose you can use it and just pay extra - and I'd have to chat with the team but I don't imagine we'd remove it or block it. Just keep it relatively as-is and document its behavior

@zurcacielos

This comment has been minimized.

zurcacielos commented Sep 14, 2018

@jeffhollan Singleton attribute is not working for me in azure functions under application service plan. When the function is tarted with a trigger like this for example:
[FunctionName("YammerETL"), Singleton("YammerETLLock", SingletonScope.Host)] public static async void Run([TimerTrigger("0 */1 * * * *", RunOnStartup = true)]TimerInfo myTimer, TraceWriter log, Microsoft.Azure.WebJobs.ExecutionContext context) { // do something }

I tried runonstartup false, and other triggers and Singleton at the function level. But if the singletonattribute would work, even with the timertrigger I should not have two instances at the same time running. This happened first at debug time under visual studio code, then it happened when deployed to azure.

@golfalot

This comment has been minimized.

golfalot commented Oct 25, 2018

I'm trying to do something a little simpler. If the function is already being executed, I want to forcibly return HTTP 429 instead of the the request being queued. I have configured as follows but no luck...

[Singleton]

"http": {
"routePrefix": "api",
"maxOutstandingRequests": 0,
"maxConcurrentRequests": 1,
"dynamicThrottlesEnabled": true
}

WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1

@jeffhollan

This comment has been minimized.

Member

jeffhollan commented Oct 25, 2018

@tbdowns strange as this one ideally could be solved without the [Singleton] just with the host configuration you have here. May be worth opening a separate issue on why you aren't getting a 429 with those settings (only thought is potentially disabling dynamic throttles).

@golfalot

This comment has been minimized.

golfalot commented Oct 25, 2018

@tbdowns strange as this one ideally could be solved without the [Singleton] just with the host configuration you have here. May be worth opening a separate issue on why you aren't getting a 429 with those settings (only thought is potentially disabling dynamic throttles).

Thanks @jeffhollan - Sadly same outcome with "dynamicThrottlesEnabled": false. If no one else chips in I'll raise a separate issue later in the week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment