Azure function is not triggering on scheduled time

[karandesai28]

Description
As discussed in #1445 also (and asked to create new issue then after), Azure function is not triggering on scheduled time unless triggered manually or opened from portal.

Expected behavior.

Azure function should trigger at 4.00 am utc every day.

Actual behavior

Azure function did not trigger unless function stays open or triggered manually.

Known workarounds

Function gets executed when opened from portal. If portal is on (logged in), scheduler also runs properly.

Related information

As mentioned in here, last ran execution details are:
utc date time: 2017-05-24T04:41:13.1059288+00:00
Execution id value: 3113d36e-a5d5-4b54-a03d-4cf496ef4632
App Service plan: SouthCentralUSPlan (Consumption)

[karandesai28]

Hello, adding few more info here: Our function app had two functions scheduled to run at 4.00 utc everyday. Yesterday, as a workaround we changed scheduler in one of the function to trigger at every 5 minutes and did few minor changes in function code to get our work done. Interestingly enough, this kept entire function app alive (even after being logged out from portal) and another function (which was scheduled to run at 4.00 utc) also got triggered exactly on time today. This means there seem to be issue of function becoming idle after some threshold time and not being awakened by system listener (which I think it should in consumption plan). I hope you guys are investigating the issue. Thanks.

[davidebbo]

Discussing this on https://stackoverflow.com/questions/44254607/azure-function-is-not-triggering-on-scheduled-time.

[karandesai28]

Hello @davidebbo sir, thanks for response. I have replied to your comments on stack overflow. Kindly check them. Re-quoting the comment here for your reference: "No, it is not the same function which is referred in GitHub issue. We had to change it due to our own commitments of completing the project. And as mentioned here and on GitHub as well, function does get execute for schedules that invoke in less time interval (2 hour), but do not get execute for an interval of day (in my case). We thought the Id we provided (on GitHub) will be enough to let you know the execution time of that function. Anyways, for your reference, I am creating a new function with same scenario to help you investigate."

To add to this:
As mentioned in my previous comment, we had made two functions to run at 4.00 utc. Though currently that second function is deleted, I do have its execution log from kudu, which shows clearly that that function executed on 4.41 instead of 4.00 on 24th May,2017 ( and that too executed because I logged into the portal and navigated to that function.) Here is the link to that log. Also find attached image herewith.

[davidebbo]

@karandesai28 thanks, that should give us the information we need to look deeper.

[davidebbo]

@karandesai28 is seems there were a number of Visual Studio deployments (msdeploy) during that time that could have affected the triggers. e.g. one happened at 2017-05-23 08:19:11. Do you recall anything related to this? Also, were you using the Functions Portal during that time?

[karandesai28]

Yes we have deployed multiple times with changes. and sometimes portal was open. right now we are using it in Indian time (with schedule of executing every 2 hours)

[karandesai28]

So few updates for you:
As discussed, we made a completely new Azure functions in consumption plan set to run on respective schedule:
Function 1- (Made from portal)
-Scheduled to run at 4.00 utc
-Last execution details: 7b9b4c03-41b5-4131-a7f4-4ac9f67ce305 (2017-06-02T04:00:00.0152527+00:00).

Function 2 and 3 (Deployed just once)
-Scheduled to run at 9.30 ist
-Last execution details:
For function 2. 954c264f-db04-4766-a68f-89cf6587b7c7 (2017-06-02T09:30:00.0032148+05:30)
For function 3: 4fbcb234-f1b2-421e-9a44-ed87eb94d27c (2017-06-02T09:30:00.0128314+05:30)

All three functions did got triggered perfectly on time. We have our functions working as per our requirement and have faced no issue in these 2 days. However, we have not tested function 2 and 3 with multiple deployments, as was the case in previous function that had issues.

Will inform you guys again if trigger misbehaved. This is for your info.

[davidebbo]

@karandesai28 ok, let us know. What I saw when I looked earlier is that it appeared that a deployment from VS was clearing the triggers. This could be either because what you deployed was missing the function.json files, or it could be because of an issue on our side. If you see this again, we'll dig deeper and get to the bottom of it.

[karandesai28]

Hi @davidebbo sir, Sure!

We are pretty confident that we had not missed function.json. However, It is important to remind that we faced trigger issue when we had two functions in same function app and were deploying only one at a time. Due to our own commitments that function app is deleted now-but you guys can investigate in that direction.

Latest update:
Yesterday we deployed the function(s) once again. But, today function executed on time. So, I think our function apps in consumption plan are working and stable now. You can close the issue if you want-Thanks for all your support.

[davidebbo]

@karandesai28 thanks for following up. Let's close for now, but feel free to revert if you hit strange behavior again, and we'll try to get to the bottom of it. In the meantime, we're adding additional (internal) logging, which will help future diagnosis.

[karandesai28]

Hello,
After lots of successful daily execution of the azure functions, yesterday we faced strangely the same issue again. Here are the complete details for your reference. I request you to kindly investigate.

We have two function apps (lets name it A and B for convenience), both has two time trigger Azure functions that has trigger time of 9.30 and 10.00 am IST set respectively.

Function App A:
Both function did not run yesterday (22/10/2017 IST) on set triggered time. To find out if there occurred any exception or function execution failure, I logged into portal today (23/10/2017 at around 8:15 am IST), As soon as I logged in, strangely both the function ran automatically (as if both were in some sort of waiting queue)!

Just for the info, No one from our team had logged into portal in last 4 days, does that make the function inactive? or is there any condition that a function goes to idle state after a certain period of in-activeness or explicit non-logging in? Or was there some sort of shutdown or maintenance tasks from your end?

The screenshot of the logs of both functions in Function App A are attached below.

Function 1:

Function 2:

Function App B:
This is another function app in the portal which has same functions with same trigger time. Both functions stopped getting executed two days ago. But when I logged in today and refreshed the function app, both function executed automatically. This is weird again.

Following are the screen shots of logs of both the functions in this Function App:

Function 1:

Function 2:

Since we are using the function in live environment, it becomes critical when function executions misbehave. Therefore, I kindly request you to check the issue. Let me know if you require any other information.

Anticipating your soonest reply. Thanks.

[davidebbo]

@karandesai28 do you by a chance set WEBSITE_TIME_ZONE to a custom time zone? A potential issue came up here, and we are investigating it.

[karandesai28]

Yes, We have set 'India Standard Time' in the app setting's WEBSITE_TIME_ZONE, but it was set since the very early phases of early deployments many months ago. Since then functions ran regularly without any delay on correct Indian Time.

There's one way to check if its time zone issue or not. Lets wait for 15 minutes, the functions are set to run at 9.30 IST daily, so if they don't run on set time, then we can assume that it is due to broken WEBSITE_TIME_ZONE. I'll notify here after 9.30 am IST. (In 15 minutes or so)

[davidebbo]

@karandesai28 there is suspicion that there could have bee a recent regression in this area. We're investigating.

[davidebbo]

@karandesai28 also, can you share your app name directly or indirectly, per this page? That will help us investigate.

[karandesai28]

You were right. The function did not run on 9.30 today!! Strange, few days ago everything was working fine in and they were getting executed.

The UTC execution time and execution Id Value for all four functions are there in the attached images in previous posts. The region is: SouthCentralUSPlan (Consumption). Let me know if any other information you required.

Update: I am manually running two azure functions due to internal requirements. Since you are investigating the functions, I updated it here for your info.

[davidebbo]

Ok, we think we know what the problem is. To work around, can you try setting WEBSITE_USE_PLACEHOLDER to 0 in your Function App Settings?

[karandesai28]

Hi,
We set WEBSITE_USE_PLACEHOLDER to 0 in app settings in Function App A, and did not modify the Function App B. The functions in Function App A triggered on set time of set Time Zone, while the functions in Function App B triggered on some random time zone. So yes, WEBSITE_USE_PLACEHOLDER did work.

Thanks a lot for such a great assistance!

Also, It will be very kind of you if you'd explain a little about WEBSITE_USE_PLACEHOLDER or share some document regarding it. Never heard that before!

[davidebbo]

@karandesai28 thanks for confirming!

Placeholders are a new internal optimization to reduce cold start (very brief mention here). It is not something anyone should need to worry about directly, but apparently it has a bug when used with WEBSITE_TIME_ZONE and we'll need to take care of that. Turning it off gets you back to the same cold start behavior as before, so you should not see any ill effects.

[karandesai28]

Interesting! Thanks for sharing. I didn't get the term 'Cold Start' though. I hope it doesn't mean delay in starting the function. 😅

Oh yes, All the functions triggered on set time today as well. So I think the issue has been resolved. Will give you trouble again if faced any other issue.

Thanks for the help 👍

[nilavghosh]

I am facing the exact issue. My function has suddenly stopped executing on schedule using a TimerTrigger. I am on a App service plan and the web app is set to 'Always On'. It was all working fine. No idea what happened.

[davidebbo]

@nilavghosh Always On is not an option for Consumption. Did you mean App Service Plan? Please provide details per above.

[nilavghosh]

@davidebbo Yes. My bad. I'm on a App Service Plan...yet faced the issue of Functions suddenly not running as per timer trigger schedule.

[davidebbo]

@nilavghosh check if you are hitting this issue.

[nilavghosh]

@davidebbo Just Checked. Nope i'm not using HTTPS . Any other chances of a problem?. The functions are python code FYI.

[davidebbo]

@nilavghosh please provide:

• site details per above
• function name and cron expression
• One UTC time of an invocation that you expected but did not happen

[nilavghosh]

@davidebbo This has to be some automated problem on azure side . All the three functions in that app are supposed to run daily and they all suddenly stopped working after 22nd. I ran one of them manually a few hours back today just to check and it ran without any changes to code or configuration. I assume the others will run too withouth any problem if i run them manually. Is it possible that the storage account got flushed and recreated automagically?

[davidebbo]

@nilavghosh I don't know what could have happened to the storage account. But please check in the App Settings whether the connection string is still valid.

[nilavghosh]

@davidebbo Yes it is still valid just logged in from Azure Storage explorer and rechecked the connection details with the promary and secondar key. All the same!

[davidebbo]

@nilavghosh try restarting your Function App. It could be that storage was down at the time it started, and it got it into some kind of bad state.

[ahamfelt]

Hi, I also have faced this problem.
We have a Function App with several functions included, some of them fires by TimeTrigger once a day, spread out during the nightly hours. These TimeTrigger functions retrieves Id's and put them in queues who later triggers other functions that collects data to these ids.

They have been in production for more then 6 months, and sometimes this problem that they are not firing had cross our way...
Last time was tonight (between 11th-12th Feb), where all functions that's has triggers set to before 01:00 (UTC) fired, but all functions that starts later didn't. And also the QueueTrigger functions stopped after 01:00.
When I restart the Function App they all starts running.
The same happend last weekend. This is, as you can understad, quite a big problem for us and our customer.

One of the functions that didn't fire as it supposted to at the time of 03:00:

Execution time, after manually start: 2018-02-12T10:11:58.0052460+00:00
Execution Id: d9728ca6-54a2-437f-b53a-94449a0ace79
Region: Northern Europe

Thanks.

[davidebbo]

@ahamfelt Looking at your logs, there are some errors like the following:

Host thresholds exceeded: [Connections]. For more information, see https://aka.ms/functions-thresholds.

This is likely due to too much instance concurrency on your queue functions, which then affects other functions. Please try limiting the per-instance concurrency via host.json (see this page).

[ahamfelt]

@davidebbo Thank you for your quick answer, I will look into that!

Just out of curiosity do you have any hypothesis about why this seems to stop the functions just over the weekends? :)
(The pattern I think I've seen from the last few times).

[davidebbo]

@ahamfelt I can't think of anything that would make this specific to weekends, so it may have been a coincidence. Or it could be that you have a certain load pattern that tends to occur around the same time within each week, which could lead to this (but really just guessing).

[AntonyCorbett]

@davidebbo I have a similar problem with a Function App not triggering. The timerTrigger is set for midnight every Wed ("schedule": "0 0 * * 3 *") but never fires unless run manually from within the portal. How can I investigate possible causes please?

2018-02-17T10:23:13.946 Function started (Id=b07a79e3-e737-4931-acee-4a053c91d5b4)
Region: UK South

[davidebbo]

@AntonyCorbett your cron expression is incorrect. What you have means: "every hour during the month of March" :)

What you want is 0 0 0 * * 3. e.g. following 10 occurrences will be (all UTC):

2/21/2018 12:00:00 AM
2/28/2018 12:00:00 AM
3/7/2018 12:00:00 AM
3/14/2018 12:00:00 AM
3/21/2018 12:00:00 AM
3/28/2018 12:00:00 AM
4/4/2018 12:00:00 AM
4/11/2018 12:00:00 AM
4/18/2018 12:00:00 AM
4/25/2018 12:00:00 AM

[AntonyCorbett]

@davidebbo thanks for the correction. I was using https://en.wikipedia.org/wiki/Cron as a reference under "CRON expression" which seemed to me to indicate otherwise. Also http://www.nncron.ru/help/EN/working/cron-format.htm which lists "0 0 * * 3 *" as "Each Wednesday at midnight". Seems like I need to dig a bit deeper. Thanks again.

[davidebbo]

@AntonyCorbett cron has many flavors unfortunately. See https://github.com/atifaziz/NCrontab/wiki/Crontab-Expression for the one we use, except we use the flavor that has an addition 'seconds' field on the left side.

[AntonyCorbett]

@davidebbo many thanks for your help.

[karthikshekaran]

Hi,
I am facing an issue that my timer trigger isn't getting triggered.
My old function - created in VS 2015 - still works fine.
The new one i.e. precompiled one created in VS 2017 does not get fired at all.
The CRON expression in both the versions is the same: ""0 */15 * * * *""

Even though I have not set anything to do with time zone, I tried adding the setting WEBSITE_USE_PLACEHOLDER and set it to 0 but to no avail.

Could you please help me? I could share anything else too which you may need.

Thanks.

[davidebbo]

@karthikshekaran please provide info as above.

[npiasecki]

I had a function today that didn't run at its scheduled time of 6 p.m. Eastern time. It's been running fine for many months. WEBSITE_TIME_ZONE is set to Eastern Standard Time.

Site: https://fn-sv-bills-prod.azurewebsites.net/
Function Name: IssueInvoices
CRON expression: 0 0 18 * * 5
Expected Invocation: 2018-03-16T18:00:00-04:00 (2018-03-16T22:00:00Z)

When I logged into the portal, I ran the function manually and it worked (that's the invocation at 2018-03-16T18:23:09.0252356-04:00). Curious, I restarted the site via the portal. A few minutes later, Functions seemed to have realized it had missed the previously scheduled invocation, and ran it again at 2018-03-16T18:28:06.8759704-04:00.

I added WEBSITE_USE_PLACEHOLDER in attempt to prevent it from happening again, though I'm not sure if the defect @davidebbo mentioned above with WEBSITE_TIME_ZONE is still at play here.

As an aside, in a separate functions app, I have a function that normally runs at 5 p.m. Eastern every weekday:

Site: https://fn-sv-labels-prod.azurewebsites.net/
Function Name: DispatchVolumePostageReport
CRON expression: 0 0 17 * * 1-5

On Monday, it executed at 6 p.m. instead of 5 p.m., an hour later than usual. On Tuesday, it was back on its 5 p.m. schedule, and I hadn't made any changes.

What has me curious is that the next invocation following the switch to Daylight Saving Time was an hour late, but after that next invocation got itself back on track. I almost wonder if my IssueInvoices job that didn't run as expected also would have been an hour late since it would have been the first invocation after the DST change. However, I'm not sure how Azure is scheduling and keeping track of the next invocation internally for that to be a possible source of a bug?

[davidebbo]

@npiasecki I think this is probably a bug related to Daylight Saving transition. Something to do with the fact that the next occurrence was scheduled before the transition, and things get confused.

Would you mind opening a separate issue so we can properly track (and link from here)? This one has been a bit of a catch all for all time issues, and this is a very distinct situation. Thanks!

[ghost]

I am also having issue in Webjobs in Azure. I have setted scheduler for running my services. But its not running as per defined schedule. I have setted timezone in application settings. also Always On setting is also enabled.

I dont understand what is the issue. help me on this.
It showing completed but if i set time as 9:30am it is showing 10 hrs ago , means running at 6am. time difference and also job are also not well executed

[davidebbo]

@rakeshshindeeternus This thread is about Azure Functions, while you are using WebJobs which is very different. Please open new issue on https://github.com/projectkudu/kudu/issues instead.

[danny8002]

My functions also run into this problem in Consumption price tier, and then l migrate the app with 'always on' host plan. but this issue still happens.

could you please take a look ?
CRON: 0 */5 * * * * (runs every 5 mins)
http://satoriportalfuncs.azurewebsites.net
function name: DatasetDashboardSummaryTimer
No WEBSITE_TIME_ZONE setting
AppServicePlan: SatoriPortalFuncs (Standard) (always on)

[davidebbo]

@danny8002 I checked the system logs and your function ran consistently every 5 minutes. The reason you don't see it in the logs is a result of App Insights sample. See this doc for more info.

[danny8002]

@davidebbo, thanks. by the way, is it possible to expose the system logs to user ? so it can save a lot of efforts for your guys.

[thedoupub]

We are facing some similar issue.

for us our function is getting offline on its own after 30 minutes or so.

We even commented out all of our code to see what is causing the issue. And we were surprised that the function went offline still.

I cannot verify why this is happening. As in the logs there is no error happening at all.

What information should i provide to look into this issue?

[thedoupub]

For the people that might have a similar issue:

What did the trick for me was to delete the function apps and recreate them after i have installed the latest version of azure function core tools.

[sthewissen]

I'm facing similar issues to this as well. A timer not firing anymore, but as soon as I log in to the Azure Portal it starts processing the function with IsPastDue=true. Not very reliable that way, so if you could help me out with what might be the root cause of this I would be very grateful.

CRON: 30 0 0,4,8,12,16,20 * * * (30 seconds after every 4 hours)
http://biffy-4h-btc.azurewebsites.net
function name: Orchestrator4H_Timer (its not actually an orchestrator function)
No WEBSITE_TIME_ZONE setting

Also from time to time I get this on the portal:

I'm guessing it could be related, but the storage connection string is correct and I can reach it through Azure Storage Explorer. It seems like functions somewhat randomly drops the connection to it.