Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Singleton support for Functions to ensure only one function running at a time #912

Open
mathewc opened this issue Nov 11, 2016 · 77 comments

Comments

@mathewc
Copy link
Contributor

commented Nov 11, 2016

We should discuss whether we want to bring this functionality forward for Functions. However, it will be relatively simple for us to do - we just need to expose new properties via function.json for function level singletons. See here for singleton doc.

In addition to Function level singleton support, we might also consider supporting Listener level Singleton as well.

When we do this, we should make sure it works across multiple languages (not just C#)

@lindydonna lindydonna added the feature label Nov 14, 2016
@lindydonna lindydonna added this to the Next - Triaged milestone Nov 14, 2016
@vladkosarev

This comment has been minimized.

Copy link

commented Feb 20, 2017

We would love to have that. Let us specify singleton in different scopes just like in web jobs.

@mathewc

This comment has been minimized.

Copy link
Contributor Author

commented Feb 22, 2017

Thanks @vladkosarev. This issue has been on the back burner, since not many people have been asking for it yet. What specifically is your scenario?

@vladkosarev

This comment has been minimized.

Copy link

commented Feb 22, 2017

Being able to process messages from multiple queues in series as long as they affect the same entity. So if we have two queues and both of them will affect Order/1 (which is encoded in queue messages) we want the two different functions to kick in in series and process messages from those two queues one by one instead of in parallel. The idea here is having serialized access to data storage for a particular entity.

@ajukraine

This comment has been minimized.

Copy link

commented Feb 23, 2017

@vladkosarev sounds like actor model. I would love to see how efficient it's with using scoped singleton and Azure functions. Another option is to use Service Fabric, but it's huge piece of infrastructure to be handled. Prefer to start with something as lightweight as Azure functions

@vladkosarev

This comment has been minimized.

Copy link

commented Feb 23, 2017

That is exactly what I'm trying to achieve. Actor-like model using Azure functions. Now that they announced that functions will support serverless framework this might not be as important but I'd still like this ability in the 'raw'. Service Fabric is great but it's still not serverless. I want to pay for compute resources not for VMs. Consumption functions + actors on top is my path to nirvana. Obviously you can't have in memory state, etc but it would still be a good start to the path of properly infinitely scalable architecture.

@WonderPanda

This comment has been minimized.

Copy link

commented Apr 21, 2017

@lindydonna Hey I'm just wondering if there are any updates in regards to when we might expect support for singleton functionality with Azure Functions. Based on the conversation here it seemed like it might be low hanging fruit. It would be extremely helpful to be able to specify singleton behavior in function.json

@tmakin

This comment has been minimized.

Copy link

commented Apr 30, 2017

I have use case where this would very helpful. I have long running background processing tasks which are triggered via a storage queue. In order to prevent spikes in the database load, I need to ensure that that there is not more than one queue item being processed at a time.

My current solution is to use a timer trigger with a short interval to manually poll the queue, but a singleton flag for queue triggers would be a much tidier option.

@ericleigh007

This comment has been minimized.

Copy link

commented Jun 22, 2017

Yes, I didn't know what this was from the title. Perhaps rename the report to something more descriptive.

My case is just about the same. Would like to guarantee only one queue function running at a time. NOTE with the current time limitations we cannot just wait.

-thanks Donna
-e

@lindydonna lindydonna changed the title Add Singleton support for Functions Add Singleton support for Functions to ensure only one function running at a time Jun 22, 2017
@alohaninja

This comment has been minimized.

Copy link

commented Aug 3, 2017

Seems like you could support locking on your own - you just need shared storage backing it - SQL Azure, Azure Blob/Table/Queue, Redis, etc. Would be great just to add a [SingletonAttribute] to our Azure Functions like webjobs has. host.json has configurable options for singletons, but I don't know if it supports Azure functions (AF) versus Web Jobs. Could just add an environment key which has the storage connection string etc, and assign it in our host.json.

@alexjebens

This comment has been minimized.

Copy link

commented Aug 3, 2017

@lindydonna any updates on a timeline for this feature?

I have a couple of Projects where I would like to switch from WebJobs to Azure Functions as well as some new Projects that need serial processing for queues, which require this functionality.

From my understanding of the way this works for WebJobs is a lock blob with a lease in the storage accounts. Azure Functions appear to already use this mechanism for the Timer-Trigger.

@alexjebens

This comment has been minimized.

Copy link

commented Aug 3, 2017

@alohaninja Supporting locking on our own is not trivial. e.g. In a Queue-Trigger Function you could only throw an exception so that the message is put back in the queue, this may however lead to the message being marked as poison and therefore lost if you cannot process it in time. Additionaly the Function will still be invoked leading to extra costs.

According to this issue there is currently no support for the Singleton Attribute in Azure Functions and the host.json options are therefore moot.

Possible Workarounds:

  • Timer-Trigger Functions appear to run as Singletons. They produce a blob in the storage account under "locks/" and create a lease. This requires implementing the input on your own.
  • ServiceBus-Trigger Functions with serviceBus.maxConcurrentCalls to 1 in host.json. This is however a global setting and I would like use this on a per function basis.

@lindydonna is it possible to confirm that Timer-Trigger Functions run as singletons?

@alohaninja

This comment has been minimized.

Copy link

commented Aug 3, 2017

@aboersch - came here because we are using Timer-Trigger functions and they can run simultaneously, I was looking for a way to ensure you cannot have concurrent timer events - seems to occur during app restart (DLLs change in /bin, app restart via portal).

Configured function.json to ignore trigger firing @ restarts via "runOnStartup": false, but we still see concurrent executions if a function was running before the cycle. Seems like the locking doesn't account well for function updates (restart events) - it fires the trigger even though an existing process is already running. To verify this - use kudu process explorer and you'll see multiple processes for the same function.

For now - I just use kudu process explorer to kill any existing processes before making any app updates or restarting the function host - would be great if the portal allowed you to kill the process for a running azure function.

@alexjebens

This comment has been minimized.

Copy link

commented Aug 3, 2017

@alohaninja I am aware of the troubles with the restart. I usually stop the app before updating because of it, however all my functions are designed to be interrupted at any time.

If you look into your storage account you will see a container called azure-webjob-hosts. There will be several folders here {hostname}-{random-number} which contain a host.Functions.{function-name}.listener file for each timer-trigger function. This file is being used to lock with a blob lease.

Every time your app is (re)started a new folder is created ({hostname}-{random-number}). Since the new folder is empty there is no blob and no lease to check for, hence the parallel execution.

This should perhaps be a separate issue though.

@rossdargan

This comment has been minimized.

Copy link

commented Aug 26, 2017

I could really do with this feature to. The issue I have is that I need to call an external api based to get more data any time a message gets added to a queue. I tend to get around 300+ messages over a few minutes every 8 hours. The issue I'm having is azure spins up 16 servers to handle the spike in messages (which is cool...) however this is utterly destroying the server I'm calling.

I've set batch size and singleton in the host.json but that appears to have no impact (setting batch size just results in more servers being started).

@BowserKingKoopa

This comment has been minimized.

Copy link

commented Sep 1, 2017

I'm in desperate need of this as well. I have a bunch of webjobs I'd like to move to Azure Functions, but I can't because they need to run as singletons. Some are queue based and need to be run in order. Others call external apis that are very sensitive about how rapidly I call them.

@rossdargan

This comment has been minimized.

Copy link

commented Sep 1, 2017

To work around this I made a buffer queue and use a scheduled function to see how many items are in the processing queue and move a few items over depending on the count.

@WonderPanda

This comment has been minimized.

Copy link

commented Sep 28, 2017

@BowserKingKoopa @rossdargan I haven't had time to experiment with it yet but in the configuration settings for host.json there's an option for WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT. Sounds like it's still being worked on but using this in conjunction with batch size might help you achieve singleton semantics for the meantime.

@bwwilliam

This comment has been minimized.

Copy link

commented Oct 5, 2017

Any updates on this please? Has anyone found a way to make enforce Functions run as a single instance max.

@KeenFann

This comment has been minimized.

Copy link

commented Oct 11, 2017

We also have a need for this. Both singleton and scale out limitation at function level.

@ajukraine

This comment has been minimized.

Copy link

commented Oct 11, 2017

I guess, if you can't wait for it to be implemented in SDK, you can use it already in Durable Azure Functions (https://docs.microsoft.com/en-us/azure/azure-functions/durable-functions-overview)

See section Stateful singletons

@AntonChernysh

This comment has been minimized.

Copy link

commented Oct 12, 2017

+1.
Use case: Simple slack bot event handler, should only send message once.

@WonderPanda

This comment has been minimized.

Copy link

commented Oct 12, 2017

@AntonChernysh I think you might be confused about Singleton behavior... There is nothing today preventing you from building a slack bot that only responds to messages only once

@AntonChernysh

This comment has been minimized.

Copy link

commented Oct 12, 2017

@WonderPanda looks like my function is scaling and running multiple times, therefore getting reply as many times as functions started. I have the same function (python 3.6) running on AWS lambda with no problem.
I'd appreciate If you could advice something to make function run only once.

@WonderPanda

This comment has been minimized.

Copy link

commented Oct 12, 2017

@AntonChernysh What triggers your function?

@bwwilliam

This comment has been minimized.

Copy link

commented Oct 12, 2017

Any idea when singleton can be made available please? I'm implementing CQRS pattern with functions. My event publisher needs to be singleton so it can process the events in the right order/sequence. Thanks

@AntonChernysh

This comment has been minimized.

Copy link

commented Oct 12, 2017

@WonderPanda post message to function's HTTPs endpoint. Can we continue in skype? anton.chernysh

@simeyla

This comment has been minimized.

Copy link

commented Aug 23, 2018

@jeffhollan do we know if Singleton guarantees FIFO order? I suspect that effectively it would (certainly for my needs it would), but I'm not sure if it is guaranteed.

@spzSource

This comment has been minimized.

Copy link

commented Aug 23, 2018

@simeyla There is no guarantees for FIFO order even using Singleton attribute. The only way is to set queue.batchSize to 1 along with Singleton attribute.

@simeyla

This comment has been minimized.

Copy link

commented Aug 23, 2018

###PSA: Singleton does not work for consumption plan. It just fails silently.

Yes, others in this thread have mentioned it but it's easy to miss.

@BowserKingKoopa

This comment has been minimized.

Copy link

commented Aug 23, 2018

I use [Singleton] (with batchSize = 1) in WebJobs a lot to process only one message at a time. @simeyla is right. You can't actually use the Azure Storage Queue as an actual Queue (FIFO) unless we get [Singleton] working in Functions.

@jeffhollan

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

Few things as I just tested this out. first thing I noticed:

  1. [Singleton] does not appear to work when scaled out. I dropped 1000 items in a queue and after it scaled out to a few instances some instances would be processed at the same time.
  2. It should be noted that Azure Storage Queues don't guarantee ordering regardless of tech you have consuming.

I believe in the interm the only way I can think of this that would work is setting batch size to 1 and setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1. Possible setting batch size to 1 and singleton have same impact, however I don't know if singleton would introduce higher billing as singleton does dequeue a bunch of messages, just holds them for a long time - which holding time I can't confirm if billed or not.

@christian-vorhemus I see you said the above (WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT + batch size) wasn't working but not sure how. The queue getting flooded sounds expected, the main thing is that D is taking ages because you are only processing one item at a time.

As an aside 2 features we are looking at that should help with this longer term:

  1. A more reliable way to limit scale out to just 1 instance (WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT has some quirks, especially if set to a value > 5)
  2. Look into if we can support service bus Sessions. This could enable faster processing across multiple instances while still preserving order.
@spzSource

This comment has been minimized.

Copy link

commented Aug 28, 2018

[Singleton] does not appear to work when scaled out. I dropped 1000 items in a queue and after it scaled out to a few instances some instances would be processed at the same time.

@jeffhollan Have you tried [Singleton(Model = SingletonMode.Listener)]? Documentation says that using this mode, the listener will only be running on a single instance (when scalled out).

So, as far as I understood, in case scalled out env function definition must be the following:

[Singleton(Mode = SingletonMode.Function)]
[Singleton(Mode = SingletonMode.Listener)]
public static void Run()
{
}
@jeffhollan

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

Thanks - quick correction. I was using some pre-release bits and that appears to be the issue. When I tested Singleton with prod bit it "just worked" -- at least for me. I need to try to repro cause others have mentioned it didn't work for them but for me right now it appears to be ok. Still not sure on the billing effect though

@SimonLuckenuik

This comment has been minimized.

Copy link

commented Aug 29, 2018

Thanks @jeffhollan these are good news!

@jeffhollan

This comment has been minimized.

Copy link
Member

commented Sep 8, 2018

All right did another round of investigation. Here's the answer:

  • [Singleton] does work on functions. The Azure Function host will create or wait for a lock in the Azure Storage account. The lock is the host ID which should be the same for all hosts of an app across all instances - so all instances share this lock and will only allow one execution to occur at a time.

To test this I put 1000 queue messages at once on a function with [Singleton]. The function would wake up, emit the invocation ID, sleep, and then emit the invocation ID. After processing all 1000 I looked at logs and never saw invocation IDs overlap. Only one invocation would happen globally at a time.

HOWEVER - in investigating the billing behavior, executions that start and then wait for lock to release are emitting billing meters while waiting. So if a host dequeues a batch of messages (default for Azure Queues I believe is 16), one execution will start and create a lock. The other 15 executions will start but immediately see the lock is taken and wait to try to grab the lock. While waiting they are emitting bills and the GB*sec are adding up. So you will be paying for that idle "wait" time on an instance.

With that said I think the recommendation is: [Singleton] isn't recommended for consumption hosted function plans. If you have a dedicated app service plan it's fine (as you are paying for the instance anyway). If you want to enforce [Singleton] like behavior in a consumption plan you are likely best to:

  1. Set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 so you never scale to more than one instance
  2. Set the host.json file to only allow 1 concurrent execution at a time for that trigger (for instance a batch size of 1 for Azure Queues).

I suspect longer term if we could support something like Service Bus Sessions it could allow for both high throughput across competing consumers and in order enforcement (Event Hubs allows this today).

@spzSource

This comment has been minimized.

Copy link

commented Sep 8, 2018

@jeffhollan thanks for exhaustive answer. Can we assume that Singleton functionality will not be removed in future releases?

@SimonLuckenuik

This comment has been minimized.

Copy link

commented Sep 8, 2018

@jeffhollan I am not sure to understand why the recommendation is not to use it for consumption plan, if it is explicitly said in the documentation that you pay as soon the wait starts. I understand that you might have to keep things in mind for your scenario to see if the billing model fits you (like your 16 messages example), but paying for wait time is still cheaper compared to dedicated plan in a lot of scenarios. For example, we have low concurrency HTTP trigger scenarios where a singleton lock is a great protection to prevent overlapping operations where transactions are not available.

@jeffhollan

This comment has been minimized.

Copy link
Member

commented Sep 8, 2018

@SimonLuckenuik I suppose you can use it and just pay extra - and I'd have to chat with the team but I don't imagine we'd remove it or block it. Just keep it relatively as-is and document its behavior

@zurcacielos

This comment has been minimized.

Copy link

commented Sep 14, 2018

@jeffhollan Singleton attribute is not working for me in azure functions under application service plan. When the function is tarted with a trigger like this for example:
[FunctionName("YammerETL"), Singleton("YammerETLLock", SingletonScope.Host)] public static async void Run([TimerTrigger("0 */1 * * * *", RunOnStartup = true)]TimerInfo myTimer, TraceWriter log, Microsoft.Azure.WebJobs.ExecutionContext context) { // do something }

I tried runonstartup false, and other triggers and Singleton at the function level. But if the singletonattribute would work, even with the timertrigger I should not have two instances at the same time running. This happened first at debug time under visual studio code, then it happened when deployed to azure.

@golfalot

This comment has been minimized.

Copy link

commented Oct 25, 2018

I'm trying to do something a little simpler. If the function is already being executed, I want to forcibly return HTTP 429 instead of the the request being queued. I have configured as follows but no luck...

[Singleton]

"http": {
"routePrefix": "api",
"maxOutstandingRequests": 0,
"maxConcurrentRequests": 1,
"dynamicThrottlesEnabled": true
}

WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1

@jeffhollan

This comment has been minimized.

Copy link
Member

commented Oct 25, 2018

@tbdowns strange as this one ideally could be solved without the [Singleton] just with the host configuration you have here. May be worth opening a separate issue on why you aren't getting a 429 with those settings (only thought is potentially disabling dynamic throttles).

@golfalot

This comment has been minimized.

Copy link

commented Oct 25, 2018

@tbdowns strange as this one ideally could be solved without the [Singleton] just with the host configuration you have here. May be worth opening a separate issue on why you aren't getting a 429 with those settings (only thought is potentially disabling dynamic throttles).

Thanks @jeffhollan - Sadly same outcome with "dynamicThrottlesEnabled": false. If no one else chips in I'll raise a separate issue later in the week.

@CoenraadS

This comment has been minimized.

Copy link

commented Mar 8, 2019

Would this also work for CronJobs, to prevent another one starting is the previous one runs longer then expected?

@paulbatum

This comment has been minimized.

Copy link
Member

commented Mar 8, 2019

@CoenraadS timer triggers already work this way. You should never see two executions of the same timer trigger happening concurrently - if you're seeing this its either a configuration issue or a bug and I would suggest you file a separate issue about it.

@jeffhollan

This comment has been minimized.

Copy link
Member

commented Mar 8, 2019

@paulbatum I suspect the ask may be if you set a cron expression to fire every 10 seconds, but never want more than one at one time. If execution A fires at time 0:00 and doesn't complete until 0:11. At 0:10 execution B may "trigger" but you may not want it to run unless no other executions in flight (in this case A is in flight). The behavior gets tricky from here because do you want B to "queue up" and start whenever A finishes (potentially just piling on executions if it always takes 10+ seconds), or just skip B and wait for 0:20.

Had some similar requests for Logic Apps back in the day and we went with the latter (skip the execution and wait for next interval), but curious on what, if any, options exist for functions

@paulbatum

This comment has been minimized.

Copy link
Member

commented Mar 8, 2019

I suspect the ask may be if you set a cron expression to fire every 10 seconds, but never want more than one at one time. If execution A fires at time 0:00 and doesn't complete until 0:11. At 0:10 execution B may "trigger" but you may not want it to run unless no other executions in flight (in this case A is in flight).

Sorry I must be missing something, but that exact ask is what I spoke to above. This is how the timer trigger works today.

The behavior gets tricky from here because do you want B to "queue up" and start whenever A finishes (potentially just piling on executions if it always takes 10+ seconds), or just skip B and wait for 0:20.

WebJobs/Functions use the queueing approach rather than the skipping approach, but it will only queue once. So if you had a 10 second interval and you had an execution run for 35 seconds, the system would start another execution immediately, but if that execution only took a second, then the system would go idle until the next normally scheduled execution. As far as I know this behavior is not configurable.

@adrianknight89

This comment has been minimized.

Copy link

commented Jun 13, 2019

Guys, any update on this? For my use case, I need a way to serialize changes to globally unique fields like usernames, and I'm using CosmosDB which doesn't have uniqueness guarantees across partitions. So, the only way to achieve this, afak, is through a singleton function. This feature request should not be put on the backburner since many people demand it.

@fabiocav

This comment has been minimized.

Copy link
Member

commented Jul 19, 2019

@jeffhollan this issue has a lot of activity and evidence that we need to enhance the feature and/or documentation for a lot of the scenarios described above. Can we (perhaps after a sync) define concrete items so this issue is actionable? If we don't believe there's any action to be taken on this (this issue suggests there is), we should close this instead with a clear description and guidance.

@Banchio

This comment has been minimized.

Copy link

commented Sep 5, 2019

I did some tests using functions v2 in linux container. I used a TimerTrigger function for brevity (since it is expected to be a singleton). If I start two docker containers (pointing to the same storage account) they will get two different host id so the timer function gets executed many times. Is this behaviour going to change to permit a singleton across several machines? TIA

@paulbatum

This comment has been minimized.

Copy link
Member

commented Sep 5, 2019

@Banchio If you deploy this to the Azure Functions service on Linux, with an app service plan with two instances, you should see that both running containers get the same Host ID and you get singleton behavior. Similarly if you deploy to the Azure Functions Linux consumption plan and drive enough load to cause the app to scale out to multiple containers, you should still see the correct singleton behavior.

If you're talking about scenarios outside of the Azure Functions service (such as running two containers locally on your machine), then it is expected that you explicitly configure the host IDs if you want those two containers to be considered as part of the same app and share singleton behavior. The reason for this is that we allow multiple different function apps to share the same storage account, so we can't assume that two random function apps using the same storage account should share singelton behavior - we need the host ID to tell us this.

@paulbatum

This comment has been minimized.

Copy link
Member

commented Sep 5, 2019

I forgot to mention, the way you explicitly set a hostid in functions V2 is to set an environment variable:
AzureFunctionsWebHost__hostid = abc123

@Banchio

This comment has been minimized.

Copy link

commented Sep 5, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.