Isolated Process Azure function v4 with .net 6 fails on Azure but works local #747
Comments
ghost
added
the
|
I experienced a similar issue but mine is related to using a Linux App Service Plan for the Azure Function with .NET 6 isolated |
|
Mine is running on Windows, both local and on Azure. The previous version os this function was .net5. The only difference between the 2 are the migration to .net6 and the use of some C# new features. |
|
Did you check the Log stream (verbose) on your Function App? because at least for me it showed a more revealing error |
|
That's where I got the messages above (the screenshot). The other message I got from Application Insights. Not relevant at all. I did it all exactly the same way we did for the .net5 versions. I tried both upgrading the app and also deleting and creating a new one. Same results. Tomorrow I'll try a simpler version of the app, with no logic, just to see if it works. If not, I'll try downgrading back to net5 for now. |
|
Today I was able to find the problem. We use UserSecrets in our development to prevent from commiting database passwords and other settings to source control. We have a library in our internal Nuget feed that sets our defaults to Azure Functions at startup, and it add the UserSecrets as a configuration source. The problem ended up being that something changed with AddUserSecrets. The code that worked in .net 5 had to be changed to include the parameter that defines UserSecrets as an optional configuration source. It was working on my local environment because I have a secrets.json file, but on Azure it failed because it couldn't find the file. The difficulty in identifying the problem happened because the error message was not registered on Application Insights as an error, but as and information log. It should've been an error. The host process was not able to start the worker process and it logged the startup exception as information (actually, as multiple information logs, one for each line of the exception). After I figured that out and found the relevant error message, I was able to update our code to conform with this behavior change and the app now works. Anyway, I see a problem here with how the startup failures are being reported to Insights (and not appearing at all in the Log Stream of the app). It should've been logged as an error and it should also have appeared in the Log Stream. Looks like one place where dotnet-isolated functions still have a long road ahead in with regard to logs.
|
|
Looks like this was a change in @brettsam @gzuber Is it possible to log these exceptions as error and not information. And is there anything that needs to be done for this to be surfaced in the errors API? @soninaren /cc @liliankasem who is looking into worker initialization errors |
|
Yup I'm currently investigating issues that lead to a work init errors. I think this ties in to my design doc around improved logging, but also is slightly different in the sense that in this instance there was a reason why the worker process crashed. @kelps do you mind sharing a timestamp, invocation id and region so I can look at the trace logs on our side, this should help me address the logging problem |
@liliankasem Sure. I'll search my logs again. How do I share those with you? Is it ok to post that kind of data here? |
Yup you're good to share that information here, nothing identifiable :) |
|
Sorry for the delay, long meetings today.... timestamp of the first occurrency I found: 2021-12-09T13:26:43.8280758Z Edit: invocation id: 2a3a9490-5021-4b98-9a72-f7ed75b92e27 |
|
Found the invocation ID and updated your comment to remove the app name (region, invocation id and time is fine, but defo don't want to post app name 😄). |
Poke on this @soninaren, @brettsam @gzuber |
|
I have a similar problem, but I don't use Migrated from .net 5 functionapp v3 to .net 6 functionapp v4. Works local, not remote on Azure (both Windows). The message I get from the logs: |
|
Hello folks, Is there any update? I have faced a similar issue. The exception was logged.
|
|
This is still occuring and we have recreated the function app with dotnet-isolated and correct linuxFxVersion DOTNET-ISOLATED|6.0 The issue seems to be missing ASPNETCORE APP installation in the latest Linux image? 2022-02-14T20:56:50Z [Error] Final functionDispatcher state: WorkerProcessRestarting. Initialization timed out and host is shutting down |
|
in my case (Function v4, VNet, Docker, Linux) - it helps to remove this line:
|
|
in my case I was running |
|
I had the same errors as above (Initialization timed out and host is shutting down) for an isolated function running on .NET 6 and Linux from a deployed zip. Id The fix for me was based on what @ds-evo mentioned. I remove the call to |
|
I had the same problems because of mixing .net 5 and .net 6 functionality, libraries and tooling. These are my lessons learned: Az runtime Create an azure function app with the right runtime (dotnet-isolated) and right version (4): az functionapp create \
--resource-group *** \
--app-insights *** \
--consumption-plan-location westeurope \
--runtime dotnet-isolated \
--functions-version 4 \
--name *** \
--storage-account *** \
--os-type LinuxProgram.cs HostBuilder()
.ConfigureFunctionsWorkerDefaults()
.Build()
.Run();Function attributes Use the right Azure function attributes from the [Function("TestFunction")]
public async Task<Response> Run([HttpTrigger(AuthorizationLevel.Function, "post", Route = "v1/action")] HttpRequestData request)
{
var response = new Response()
{
Test = "test",
Data = request.CreateResponse()
};
response.Data.StatusCode = HttpStatusCode.Accepted;
return response;
}
internal class Response
{
public HttpResponseData Data { get; internal set; }
[QueueOutput("Queue", Connection = "AzureWebJobsStorage")]
public string Test { get; internal set; }
}Deploy with the right tools Deploy the function with azure-functions-core-tools. npm i -g azure-functions-core-tools@4 --unsafe-perm true
mv local.settings.json.release local.settings.json
func azure functionapp publish $FUNCTION_APP |
@zyofeng Could you add this reference to the csproj and try it out again? |
|
Facing the same issue for isolated, net6, v4, Function, windows We tried to wrap up the call to But our call stack is a bit different from what has been posted above: |
I have the same issue using docker and Linux on Azure Portal |
|
Running into this issue ("Did not find any initialized language workers") on one of our two .net 6 isolated functions (Linux Consumption Plan). Figured out that the one failing (and Http Triggered function) was using Since it's an Http Triggered function, I couldn't find any way to set a Service Bus output binding as the returned result for Http trigger is already bound to the Http Response (am I correct on this?), so needed to go this approach instead. Everything runs perfectly fine locally, but fails deployed to Azure. So commented out the We have this exact same code being used in .Net 3.1 v3 Functions perfectly fine. I saw mentioned in one of the many issues logged about this very generic error that people had found that certain Nuget Packages aren't compatible with the Isolated Process functions. is Here's the code that causes the function to not startup. Tried both the Managed Identity and Connection String auth approaches thinking one might have an issue, but same result for both: services.AddAzureClients(clientsBuilder =>
{
//clientsBuilder.AddServiceBusClientWithNamespace(serviceBusNamespace)
// .WithName(serviceBusClientName)
// .WithCredential(new DefaultAzureCredential(options));
clientsBuilder.AddServiceBusClient(configuration["TempSBConnectionString"])
.WithName(serviceBusClientName);
}); |
|
@willyt150 Just to confirm, did you have the |
|
@kshyju Yes, I double and triple checked to make sure that I wasn't going crazy, and all the settings I have in my In the case of |
|
@kshyju I apologize for wasting your time. I have confirmed the issue was totally of my doing with the Managed Identity auth using the service bus namespace. Still not sure why it also failed w/ the Connection String approach though.... I'm reusing the "ServiceBusConnection__fullyQualifiedNamespace" app setting, since it's needed with the new Service Bus Trigger to support Managed Identity auth just keeping with the pattern, as the setting to retrieve the namespace from to pass into the However, in the C#, I used the "__" instead of ":" for the configuration key, so on my Windows machine locally it worked just fine, but that caused the config value I retrieved up on the Linux machine to be an empty string.... so then the Add* call blew up. Is there any way to expose those error messages during the DI process? Cause looking through app insights/log stream, I didn't see any of that information. Didn't figure this out until I wrote my own simple implementation of a service bus client factory, so when it hit my code and errored I had the ILogger available and got a very nice error telling me Value '' is not a valid Service Bus namespace. |
|
@willyt150 Glad you got it figured out. We are working on improving the logging on worker startup failure scenario so that it will show detailed cause of failure in app-insights/log stream. |
cloudmelon
added this to Backlog ( Under consideration )
in .NET on Azure functions Roadmap (Public)
|
I've been trying to work around this issue for 4 days now without success. Until I stumbled across a SO comment about dependencies being on |
|
I too am having this problem. This is an Azure Function .NET6 isolated mode. The function works locally, works when published to an Azure Function, but does not work when published to a deployment slot (in either Consumption Mode or App Service Plan). You may be thinking the deployment slot is missing a config/setting from the primary slot...I double-checked and I tested it. I published a different function that requires no configuration (the default Azure Function NET6 isolated template) and it too resulted in the error. The next test was to spin up a new function and add a deployment slot. The second deployment slot also produced the same error. A conclusion is that NET6 isolated Azure Functions do not work with deployment slots. The error is below: |
I resolved my issue...The function name was too long. The default name provided by Visual Studio when publishing makes the name too long for Azure as described here: Create an Application Setting with the key of "AzureFunctionsWebHost:hostid" then give the host id a unique name less than 32 characters. Make it a deployment slot setting. |
|
I had this issue a year ago when running on .NET 5 and v3 Functions, upgraded to .NET 6 and v4, thinking this surely has to be fixed by now and clearly it isnt. I applied this work around again (as I did originally) and it's working again. |
|
I just update the project.csproj with following
After deploying and restart again I see functions are available Can checkout the following docs |
Another solution is to publish in x64 and set the platform to 64-bit |
and others
We have a few Azure Functions developed using isolated-process in .net 5 and we are migrating to .net 6.
The migration worked without problems and runs perfectly local, but fails when deployed to azure. We are using Http, ServiceBus and Timer triggers and all fail the same way:
It shows an "Executing" log in the "Log stream", but after a few seconds it fails with "[Error] Final functionDispatcher state: WorkerProcessRestarting. Initialization timed out and host is shutting down". After that the host is restarted.
All functions fail the same way, but only when running on Azure. They work perfectly local and there are no other logs on Azure to find out what could be going on.
The app was not created from scratch, but was updated from the previous version to use .net 6 and updating/upgrading all required dependencies.
Edit:
Application Insights shows the following 2 error log messages, aprox. 50 seconds after any attempts:
Final functionDispatcher state: WorkerProcessRestarting. Initialization timed out and host is shutting down and Did not find any initialized language workers
The text was updated successfully, but these errors were encountered: