Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random errors across every single Python HttpTrigger Azure Function in Function App #1758

Open
htstan0 opened this issue Aug 7, 2017 · 23 comments

Comments

@htstan0
Copy link

htstan0 commented Aug 7, 2017

We have started encountering random, intermittent errors across every single one of our HttpTrigger Azure Functions which are coded in Python. The log streaming service shows only this vague Exception when the issue occurs while executing a given function:

Functions.. Microsoft.Azure.WebJobs.Script: .

In looking at the logs under Monitoring, I was able to dig out this call stack:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.<my function name> ---> System.ApplicationException
   at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.ExecuteScriptAsync(String path,String arguments,Object[] invocationParameters,FunctionInvocationContext context) at C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Script\\ScriptFunctionInvoker.cs : 0
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Script\\ScriptFunctionInvoker.cs : 55
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs : 171
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`1.InvokeAsync[TReflected](Object[] arguments)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,ILogger logger,CancellationTokenSource functionCancellationTokenSource)
   at System.Runtime.Exception…
@silencev
Copy link

silencev commented Aug 8, 2017

I experienced the similar issue.
My function app is coded in Python for service bus queue trigger and terminated unexpectedly.

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.<XXXXXX> ---> System.ApplicationException
   at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.ExecuteScriptAsync(String path,String arguments,Object[] invocationParameters,FunctionInvocationContext context) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 114
   at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 50
   at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs : 171
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`1.InvokeAsync[TReflected](Object[] arguments)
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,ILogger logger,CancellationTokenSource functionCancellationTokenSource)
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
   at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??) 
   End of inner exception

@christopheranderson
Copy link
Contributor

Could we please have an invocation id and a region so we can take a look at the app?

Please note that Python is experimentally supported and not fit for production scenarios.

@silencev
Copy link

@christopheranderson
my function app info:

  • Invocation ID: 2a7db84d-0ebb-46ca-93ce-60d523082113
  • Datacenter: centralus
  • Execution time: around 2017-08-07 06:10:34

@Frankygupta
Copy link

Frankygupta commented Oct 23, 2017

I have developed an Azure function that has ran into exactly same error.

"Exception while executing function: Functions.RadiaPythonPoc
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.RadiaPythonPoc ---> System.ApplicationException
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.ExecuteScriptAsync(String path,String arguments,Object[] invocationParameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 114
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 55
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs : 95
at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments) at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync…"

The region is South Central US & with invocation id, do you mean the subscription id?

@StephenThomson
Copy link

StephenThomson commented Oct 24, 2017

I'm having the same issue, previously working code just stopped working.

Invocation: 12a6e044-5ab1-40ee-9705-e81ad59671f5

Region: UK West

Exception while executing function: Functions.elevate
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.elevate ---> System.ApplicationException
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.ExecuteScriptAsync(String path,String arguments,Object[] invocationParameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 114
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 55
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs : 95
at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments) at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
End …

@StephenThomson
Copy link

@fabiocav Would you be able to confirm this issue is still under investigation please?

@nchrisr
Copy link

nchrisr commented Oct 25, 2017

I am also facing the same exact issue:

Location: Canada Central

Exception while executing function: Functions.getMissingStocks
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.getMissingStocks ---> System.ApplicationException
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.ExecuteScriptAsync(String path,String arguments,Object[] invocationParameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 114
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 55
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs : 95
at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments) at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource)

@silversmurfer
Copy link

silversmurfer commented Oct 25, 2017

Ditto - I am seeing the same thing in UKSouth from this morning - any update?

Exception while executing function: Functions.HttpTriggerPython31
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.HttpTriggerPython31 ---> System.ApplicationException
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.ExecuteScriptAsync(String path,String arguments,Object[] invocationParameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 114
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 55
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs : 95
at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments) at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLo…

@Frankygupta
Copy link

I was able to resolve the issue once I created a virtual environment and imported libraries inside it. You would require to navigate to the folder of your function on the debug console and do so. Below are the steps:

  1. Navigate to Site/wwwroot/Function
  2. create virtual env by 'python -m virtualenv myvenv'
  3. Navigate to Site/wwwroot/Function/myenv/scripts and run the command 'activate.bat'. This would change the env to the newly created virtual env and your command prompt will be prefixed with (myvenv)

Now, you can go ahead and install all the packages that you need via 'python -m pip install '.

The only change would be in the way you are trying to read the packages in the script. They should now be imported with a different statement as below:
import sys, os.path
sys.path.append(os.path.abspath(os.path.join(os.path.dirname( file ), 'myvenv/Lib/site-packages')))

I took reference from https://stackoverflow.com/questions/43970307/azure-functions-installing-python-modules-and-extensions-on-consumption-plan

@sergei3000
Copy link

sergei3000 commented Oct 27, 2017

This thing affects us too on a time triggered function on consumption plan. Makes using Azure Functions impossible. But it used to work for some time.
After getting this error the whole applications stops (all of the functions inside it). And it does not restart. Pressing Restart of Stop/Start does not help.
The only way we can restart an app is to delete all files from data\Functions\secrets and then restart. But it's not good since some functions need those keys, and when they are regenerated the connected service just breaks. And of course it's not right to build a monitoring and restarting service around this problem.

The function's output is like so:

Exception while executing function: Functions.RecountRecommendations

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.RecountRecommendations ---> System.ApplicationException

at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.ExecuteScriptAsync(String path,String arguments,Object[] invocationParameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 114

at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 55

at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs : 95

at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments)

at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)

at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)

at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource)

at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)

at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.Execute…

@sergei3000
Copy link

sergei3000 commented Oct 30, 2017

Hi everyone,
It seems that changing runtime version of your Functions app to the earlier one may fix this. It did work for my function.
I set the version to the one that worked before the release that was pulled before this issue was opened. Namely I am using version 1.0.11027 of July 14.
Here's how to deal with runtime versions in Functions apps:
https://docs.microsoft.com/en-us/azure/azure-functions/functions-versions
And here you can find info on the Functions releases:
https://github.com/Azure/azure-webjobs-sdk-script/releases
Hope this helps.

@StephenThomson
Copy link

@sergei3000 Thanks! This fixed it for me too. Were you using a custom installed version of python?

@nchrisr
Copy link

nchrisr commented Oct 30, 2017

@sergei3000 Thhanks alot that worked for me

@sergei3000
Copy link

@StephenThomson I did not do that personally, but a later version of Python and some additional libraries were installed into the app I am using by a colleague of mine. This is what he used as a reference:
https://prmadi.com/running-python-code-on-azure-functions-app/

@fabiocav
Copy link
Member

@StephenThomson apologies for the delay on this. The notification was completely missed.

@sergei3000 so you started experiencing this on the version immediately after 1.0.11027?

@StephenThomson based on your initial report, I'm assuming this issue started more recently for you, correct?

@fabiocav
Copy link
Member

@StephenThomson would you be able to provide a repro for this?

@sergei3000
Copy link

sergei3000 commented Oct 31, 2017

@fabiocav I started experiencing this several weeks ago, but I did not use Functions so much before that. So I should say I started experiencing this error right after starting to use Azure Functions extensively. I chose version 1.0.11027, because it was the second release back from the date this issue was opened.

Also I found that the error mentions WebJobs in logs, and there was WebJobs SDK release on August 2, which is the next release after 1.0.11027. This could be a foolish thing to make this kind of a connection, but I just tried to find some clues to make things work for me :).

@StephenThomson
Copy link

StephenThomson commented Oct 31, 2017

@fabiocav : Thanks for getting back to me :)

Yes I started experiencing this last week. It seems to be solved by setting FUNCTIONS_EXTENSION_VERSION=1.0.11296

I am currently trying to reproduce the error but it is tricky. When I first encountered it I redeployed onto a fresh function-app. This ran initially but then stopped working the next day. I've created a new function app today and am trying to get it to break. I did notice that the bug only happens for me when I am using python 3.6 from D:\home\site\tools.

My steps so far are:

New Function App - Consumption - UK West
New Python Http trigger
"import os
import sys
response = open(os.environ['res'], 'w')
response.write("hello world from "+sys.version)
response.close()"

Output:

"hello world from 2.7.8 (default, Jun 30 2014, 16:03:49) [MSC v.1500 32 bit (Intel)]"

Kudu:
nuget.exe install -Source https://www.siteextensions.net/api/v2/ -OutputDirectory D:\home\site\tools python362x86
mv /d/home/site/tools/python362x86.3.6.2/content/python362x86/* /d/home/site/tools/

Rerun function:

"hello world from 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)]"

I will update as soon as I can.

UPDATE:

Approx 4h later rerun function:
"{
"id": "6e9bffa1-834c-41ae-b5f9-a17d2bfbdaad",
"requestId": "451df001-90ea-4848-9b45-a106f8797205",
"statusCode": 500,
"errorCode": 0,
"message": "Exception while executing function: Functions.HttpTriggerPython31 -> "
}"

TEST LOGS:
2017-10-31T13:53:57 Welcome, you are now connected to log-streaming service.
2017-10-31T13:54:04.924 Function started (Id=feebf67b-64a7-46b4-98a6-0cc053b363f8)
2017-10-31T13:54:06.018 Exception while executing function: Functions.HttpTriggerPython31. Microsoft.Azure.WebJobs.Script: .
2017-10-31T13:54:06.049 Function completed (Failure, Id=feebf67b-64a7-46b4-98a6-0cc053b363f8, Duration=1122ms)

Invocation context : feebf67b-64a7-46b4-98a6-0cc053b363f8

Monitor Logs:
Exception while executing function: Functions.HttpTriggerPython31
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.HttpTriggerPython31 ---> System.ApplicationException
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.ExecuteScriptAsync(String path,String arguments,Object[] invocationParameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 114
at async Microsoft.Azure.WebJobs.Script.Description.ScriptFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Script\ScriptFunctionInvoker.cs : 55
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs : 90
at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments) at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLo…

@StephenThomson
Copy link

Set FUNCTIONS_EXTENSION_VERSION=1.0.11296 and it started working. Will have to wait to see if they continue to work but my production code was changed over ~30h ago and is still working.

@StephenThomson
Copy link

The test code stayed working on 1.0.11296 for >24h. Switched it to ~1 and it stop functioning in a few hours.

@fabiocav fabiocav removed this from the Active Questions milestone Nov 2, 2017
@fabiocav fabiocav removed their assignment Nov 2, 2017
@fabiocav
Copy link
Member

fabiocav commented Nov 2, 2017

Clearing milestone and assignment for triage and further investigation

@sergei3000
Copy link

sergei3000 commented Nov 3, 2017

Hi everyone.
There seems to be another solution to this problem.
I created a function in a Functions app that is working on the App Service plan (I am using a B1 instance). It's being working fine for a day already, and on the current version of runtime (version ~1).

The drawback here is that for a time/http trigger to work the app should be always on. But if the price is not too high for you, it's a good solution actually, my code works much faster there. Also it imports additional libraries very fast, unlike functions working on the Consumption plan which take way too long to do that.

At the same time, I really hope that Microsoft will fix this bug soon. It is really critical, since it is affecting so many people and makes it impossible using the very attractive Consumption plan.

@christopheranderson christopheranderson added this to the Triaged milestone Jan 16, 2018
@paulbatum
Copy link
Member

Hello, I believe that using a custom version of python is interacting poorly with our "placeholder" feature which is designed to optimize cold start scenarios. Thanks to @StephenThomson for clear repro steps.

One of the important points to note is that placeholders are disabled when you specify a custom function version. So while on first look it might appear that 1.0.11296 solves the problem, actually all that is happening is that you are disabling placeholders. In fact, if you go to "Function App Settings" you'll probably see that your host version is something newer like this:

image

This is because we retire older versions, and 11296 is already retired.

You can confirm my theory by disabling placeholders directly. Remove your app setting for the function version, and replace it with WEBSITE_USE_PLACEHOLDER=0.

This setting is documented here - please let me know how it goes.

I do not know why placeholders are interacting poorly with a custom python version. However I want to point out that we are no longer investing in the experimental implementation of python that is currently available for functions v1. For more information about language support, see here.

So we are unlikely to get to the bottom of this issue, but it looks like we have a workaround - disabling placeholders as described above.

I will close this issue as "won't fix, but workaround available" once I hear from a few of you.

@paulbatum paulbatum removed this from the Triaged milestone Jan 20, 2018
@paulbatum paulbatum added this to the Active Questions milestone Jan 20, 2018
@paulbatum paulbatum self-assigned this Jan 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants