.Net8 Isolated worker Service Bus Trigger "No Job functions found" error when deploying code to azure functions

[PFXBen]

Description

Hi there,

Getting the following error with a .Net8 Isolated worker project when deployed to a Consumption based Windows based Azure Function in West Europe:

"No job functions found. Try making your job classes and methods public. If you're using binding extensions (e.g. Azure Storage, ServiceBus, Timers, etc.) make sure you've called the registration method for the extension(s) in your startup code (e.g. builder.AddAzureStorage(), builder.AddServiceBus(), builder.AddTimers(), etc.)."

The code I am trying to deploy works perfectly fine locally and the service bus trigger is correctly registered and triggered. However, after deployment, the no job function error occurs in application insights and the trigger does not show up as a function in the azure portal, as if its just been ignored.
I'm using all the latest nuget packages however ill list the relevant ones below:

• Microsoft.Azure.Functions.Worker@1.21.0
• Microsoft.Azure.Functions.Worker.Sdk@1.17.2
• Microsoft.Azure.Functions.Worker.Extensions.ServiceBus@5.17.0

I've verified that the connection string has been placed in the environment variable section of the azure function correctly and that the azure function is set to zip deploy.

Csproj

host.json

Relevant local.settings.json (AzureWebJobsStorage points to Azure Storage in cloud)

Relevant service bus trigger in code

Relevant Program.cs startup code

(Note, this project has a requirement to connect to multiple service buses so im using a named azure client approach)

Any help would be greatly appreciated!

Thanks,
Ben

Steps to reproduce

Create a .net8 isolated worker project, with a single service bus trigger as described above. Use the latest sdks in nuget for azure worker projects. Run and verify the code works locally then deploy to a windows based azure function.

[satvu]

Hi @PFXBen can you share your app privately using these instructions so we can investigate the deployed app?

And can you see if adding the MSBuild prop <FunctionsEnableWorkerIndexing>false</FunctionsEnableWorkerIndexing> to your project fixes the problem in deployment?

[iqoOopi]

Same issue here.... .net 8 service bus triggered azure function works perfectly on my local. The release pipeline by using zip deployment also succeed. However, no functions found in azure portal.

@satvu Adding that MSBuild prop fixed the problem. Wondering what is that prop doing? if this prop is needed, why it is not included in the VS template (I created the service bus triggered azure function via VS new project)

[PFXBen]

Hi @PFXBen can you share your app privately using these instructions so we can investigate the deployed app?

And can you see if adding the MSBuild prop <FunctionsEnableWorkerIndexing>false</FunctionsEnableWorkerIndexing> to your project fixes the problem in deployment?

Adding the MSBuild prop unfortunately hasn't made any difference. As for sharing the app privately, as there are no azure functions shown, i can only do option 2, and when i try to do that i just get a 404 error back and nothing logged on the azure function. Is there a way i can privately share the function app URL or other identifying information with you guys so I don't have to post it here?

[satvu]

Same issue here.... .net 8 service bus triggered azure function works perfectly on my local. The release pipeline by using zip deployment also succeed. However, no functions found in azure portal.

@satvu Adding that MSBuild prop fixed the problem. Wondering what is that prop doing? if this prop is needed, why it is not included in the VS template (I created the service bus triggered azure function via VS new project)

The prop is true by default as part of some performance improvements (see here). @iqoOopi, can you share your function app privately with these instructions so we can inspect it in prod?

[satvu]

Hi @PFXBen can you share your app privately using these instructions so we can investigate the deployed app?
And can you see if adding the MSBuild prop <FunctionsEnableWorkerIndexing>false</FunctionsEnableWorkerIndexing> to your project fixes the problem in deployment?

Adding the MSBuild prop unfortunately hasn't made any difference. As for sharing the app privately, as there are no azure functions shown, i can only do option 2, and when i try to do that i just get a 404 error back and nothing logged on the azure function. Is there a way i can privately share the function app URL or other identifying information with you guys so I don't have to post it here?

@PFXBen Can you provide a minimal repro and share a GH link to it? I will see if I can repro the issue myself deploying it to the same region.

[PFXBen]

Hi @PFXBen can you share your app privately using these instructions so we can investigate the deployed app?
And can you see if adding the MSBuild prop <FunctionsEnableWorkerIndexing>false</FunctionsEnableWorkerIndexing> to your project fixes the problem in deployment?

Adding the MSBuild prop unfortunately hasn't made any difference. As for sharing the app privately, as there are no azure functions shown, i can only do option 2, and when i try to do that i just get a 404 error back and nothing logged on the azure function. Is there a way i can privately share the function app URL or other identifying information with you guys so I don't have to post it here?

@PFXBen Can you provide a minimal repro and share a GH link to it? I will see if I can repro the issue myself deploying it to the same region.

@satvu sure ok ill try and sort that out today for you.

EDIT: Heres a repro repo of the code in use, essentially its a message forwarder that listens on one service bus, and dependant on the type of message, forwards it on to a different queue on another service bus to be picked up by other azure functions (All in west europe, all having the same issue and all .net8 isolated worker using same packages) https://github.com/PFXBen/net8servicebusrepro

[alanmk]

FYI - I'm experiencing this exact issue, and the MSBuild property isn't working for me either.

[PFXBen]

Any update @satvu ?

[satvu]

@PFXBen We are still investigating- will post more updates when we are more confident on a root cause.

[satvu]

@PFXBen @alanmk We investigated another similar incident where we had the full app payload, and the issue was due to the folder structure of the deployed zip. How are you building/deploying your app? Are either of you MacOS users by any chance? Additionally, if you could open a support case, it may be easier/quicker for us to investigate as there are many root causes that can lead to this scenario.

EDIT: Also, if you have a last known working version of the app that has successful prod behavior, what's different?

[PFXBen]

@PFXBen @alanmk We investigated another similar incident where we had the full app payload, and the issue was due to the folder structure of the deployed zip. How are you building/deploying your app? Are either of you MacOS users by any chance? Additionally, if you could open a support case, it may be easier/quicker for us to investigate as there are many root causes that can lead to this scenario.

EDIT: Also, if you have a last known working version of the app that has successful prod behavior, what's different?

Certainly in my case, its a zip deployed application built/deployed in azure devops, and the original project was created in MacOS and commited to the repo from there, so that may be a factor. Can create a support ticket sure, do you have a link to where the best place to do that is? Not sure if you want it via azure support or .Net. As for the edit, this was a new project so no working prod version, only that it works ok locally on my macbook.

[alanmk]

@satvu/@PFXBen We've managed to resolve our issues after a little more investigation.

After a bit of digging, we found that the following device gave us considerably more diagnostic information to work with:

 using eventSourceListener = new AzureEventSourceListener(
            (args, message) => Logger.LogInformation("[{0:HH:mm:ss:fff}][{1}] {2}", DateTimeOffset.Now, args.Level, message),
            level: EventLevel.Verbose);

We used it to bookend our host configuration and were able to deduce that we were encountering a number of issues relating to network and MSI configuration at startup. Once corrected, everything seem to work as intended.

It's probably worth mentioning that we also needed to include the MSBuild property in our csproj files.

<FunctionsEnableWorkerIndexing>false</FunctionsEnableWorkerIndexing>

I hope this helps.

[satvu]

@PFXBen You should be able to open one from the Azure Portal - if you navigate to "Support + Troubleshooting" you should be able to open a support case and fill out the fields as best you can (not 100% sure what the UI is currently, but you should be able to link the exact resource to the case in a dropdown box specific to you).

[PFXBen]

@satvu Tried logging a support ticket but it wont let me proceed with opening that ticket until ive address the exception its raising, which is the same one at the start of this ticket...so it wont let me open it

[shashank3kapoor]

@PFXBen , @satvu
We are facing this exact same issue, when we tried upgrading Azure Function from .Net-6 to .Net-8

Please share if we have solution for it now?

[longmatthew]

I'm adding a comment here for anyone who may come across this issue and how we potentially solved it on our end. Like many others we started experiencing this issue when we had upgraded our azure functions from in-process to isolated worker (and from .net 6.0 to .net 8.0). We had followed many of the other solutions which I will mention here as to consolidate our attempts and elaborate on the actual fix for us.

Potential reason(s) for not seeing deployed functions:

The structure of the files for each azure function in its deployed state needs to be in a certain 'format' or 'layout'. More specifically, if anything subverts that layout you may not see the azure function at all or may see errors already mentioned here. This means any method for 'zip deploying' could alter the structure of that layout.

That and the application settings needs to have very specific options selected.

The steps to produce a resolution

From previous research on this same topic we obtained a solution as follows:

1. We initially released from the publishing option by right clicking the Azure function within our visual studio solution and going through the 'publish wizard' to release the function to its original deployed location. In place of this step I have also verified that creating the azure function in the portal itself set to the isolated worker model with all needed resources and then deploying in the same manner should be equivalent to this step. This step was needed because the previous application settings are pointing to a previous version of dotnet and not the isolated model which we explain in the next few steps. DO NOT HIT PUBLISH YET. You may receive errors upon publishing that give no real vital information if you do.
   

2. The 'publish wizard' updates the application settings and generally points to the dotnet-isolated version of functions along with putting in a placeholder for the isolated worker model. Navigate to the azure function in the portal and ensure that these settings are correct in our environment variables/application settings:
   

Whether you created the function in the portal then deployed or used the 'publish wizard' you must ensure these settings exist before continuing.

To note:
For Continuous Deployments of Azure Function App using GitHub Actions, you have to enable the setting 'SCM_DO_BUILD_DURING_DEPLOYMENT' to true in the Application Configuration.

3. Next click 'publish' from the final step in the 'publish wizard' which will deploy the function to the location of your choosing. If the settings are correct from the previous step the deployment should succeed and you should see the functions not in the 'Overview' window in Azure portal. To some varying degrees I've seen some errors when clicking 'publish' and then checked the error message to only get something akin to:

System.AggregateException: One or more errors occurred. ---> Microsoft.WebTools.Shared.Exceptions.WebToolsException: Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details. 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) Microsoft.WebTools.Shared.Exceptions.WebToolsException: Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details. <---

Microsoft.WebTools.Shared.Exceptions.WebToolsException: Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details. 

Which I believe generally means the settings from the previous step did not gel correctly with the publishing step. If the error continues try going through the publishing process again and removing connections to resources. I noted that once I had just the 'bare-bones' of deploying the azure function this worked as well.

CI/CD Deployment Fix

Besides the manual fix to this issue we also noticed that our CI/CD deployment through Azure Devops caused the function to disappear or not be found in some manner. This is because all mentions of build steps that zip or archive the project files before creating a release caused the files to be zipped twice.

Previously we had a task to publish the files and archive them like so:

...
- task: DotNetCoreCLI@2
  displayName: 'Publish Azure Functions'
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: '$(System.DefaultWorkingDirectory)/$(functionProjectAppName)/$(functionProjectAppName).csproj'
    arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)/publish/$(environmentName)/$(functionProjectAppName)'
    zipAfterPublish: true
...

We entirely removed zipAfterpublish - which is the main culprit or generally any task or job that zips the files before release (via publishing or a publish command). Following the Microsoft documents you can explicitly set it to false so the task should look like this now (along with a few other options explicitly set):

...
- task: DotNetCoreCLI@2
  displayName: 'Publish Azure Functions'
  inputs:
    command: 'publish'
    projects: '$(System.DefaultWorkingDirectory)/$(functionProjectAppName)/$(functionProjectAppName).csproj'
    arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)/publish/$(environmentName)/$(functionProjectAppName)'
    publishWebProjects: false
    modifyOutputPath: false
    zipAfterPublish: false
...

Then our release is just a standard Azure Function Deployment Release - ensure you can see the zip file in the location you have chosen. I was extra precautious by having a task that would extract the files from the zip location and then move them into a separate folder location. Then from that location we would deploy the zip file.

The deployment method allowed was to leave it as 'Auto-Detect'.

Potential Pitfalls

I've seen that the 'Auto-Detect' feature will choose 'publish using the package' which it will zip and use a positional argument that publishes the versions that are already in the package.json files. If the settings in Azure portal do not reflect the method in which you are attempting to deploy you will receive another error on publish. As part of the CI/CD pipeline the release step for deploying the actual azure function will set this variable to either zero or one in the portal.

Note that if your release becomes out of sync then it will not deploy. I have seen that the 'manual' publish through visual studio will sometimes change the application settings and then other times it won't.

There are also sources listed in the next section that mention removing the archiving of files from the CI/CD pipeline. I have tried both versions of leaving them in and also removing them and was able to see the functions in their deployed location. It could just be a matter of being conscious of the steps taken to zip your deployment.

Again be warry of your settings.

Sources

• https://stackoverflow.com/questions/66662364/deploying-azure-function-isolated-process-to-azure-using-azure-devops-is-not-w
• https://stackoverflow.com/questions/76255239/deployed-isolated-function-app-returns-404
• https://stackoverflow.com/questions/75503474/azure-functions-deployment-from-git-does-not-work-with-net-7
• https://stackoverflow.com/questions/55623529/azure-function-deployed-successfully-but-not-working
• https://learn.microsoft.com/en-us/azure/azure-functions/functions-how-to-azure-devops?tabs=csharp%2Cyaml&pivots=v1

[PFXBen]

I'm adding a comment here for anyone who may come across this issue and how we potentially solved it on our end. Like many others we started experiencing this issue when we had upgraded our azure functions from in-process to isolated worker (and from .net 6.0 to .net 8.0). We had followed many of the other solutions which I will mention here as to consolidate our attempts and elaborate on the actual fix for us.

Potential reason(s) for not seeing deployed functions:

The structure of the files for each azure function in its deployed state needs to be in a certain 'format' or 'layout'. More specifically, if anything subverts that layout you may not see the azure function at all or may see errors already mentioned here. This means any method for 'zip deploying' could alter the structure of that layout.

That and the application settings needs to have very specific options selected.

The steps to produce a resolution

From previous research on this same topic we obtained a solution as follows:

1. We initially released from the publishing option by right clicking the Azure function within our visual studio solution and going through the 'publish wizard' to release the function to its original deployed location. In place of this step I have also verified that creating the azure function in the portal itself set to the isolated worker model with all needed resources and then deploying in the same manner should be equivalent to this step. This step was needed because the previous application settings are pointing to a previous version of dotnet and not the isolated model which we explain in the next few steps. DO NOT HIT PUBLISH YET. You may receive errors upon publishing that give no real vital information if you do.
   
2. The 'publish wizard' updates the application settings and generally points to the dotnet-isolated version of functions along with putting in a placeholder for the isolated worker model. Navigate to the azure function in the portal and ensure that these settings are correct in our environment variables/application settings:
   

Whether you created the function in the portal then deployed or used the 'publish wizard' you must ensure these settings exist before continuing.

To note: For Continuous Deployments of Azure Function App using GitHub Actions, you have to enable the setting 'SCM_DO_BUILD_DURING_DEPLOYMENT' to true in the Application Configuration.

3. Next click 'publish' from the final step in the 'publish wizard' which will deploy the function to the location of your choosing. If the settings are correct from the previous step the deployment should succeed and you should see the functions not in the 'Overview' window in Azure portal. To some varying degrees I've seen some errors when clicking 'publish' and then checked the error message to only get something akin to:

System.AggregateException: One or more errors occurred. ---> Microsoft.WebTools.Shared.Exceptions.WebToolsException: Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details. 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) Microsoft.WebTools.Shared.Exceptions.WebToolsException: Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details. <---

Microsoft.WebTools.Shared.Exceptions.WebToolsException: Publish has encountered an error. We were unable to determine the cause of the error. Check the output log for more details. 

Which I believe generally means the settings from the previous step did not gel correctly with the publishing step. If the error continues try going through the publishing process again and removing connections to resources. I noted that once I had just the 'bare-bones' of deploying the azure function this worked as well.

CI/CD Deployment Fix

Besides the manual fix to this issue we also noticed that our CI/CD deployment through Azure Devops caused the function to disappear or not be found in some manner. This is because all mentions of build steps that zip or archive the project files before creating a release caused the files to be zipped twice.

Previously we had a task to publish the files and archive them like so:

...
- task: DotNetCoreCLI@2
  displayName: 'Publish Azure Functions'
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: '$(System.DefaultWorkingDirectory)/$(functionProjectAppName)/$(functionProjectAppName).csproj'
    arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)/publish/$(environmentName)/$(functionProjectAppName)'
    zipAfterPublish: true
...

We entirely removed zipAfterpublish - which is the main culprit or generally any task or job that zips the files before release (via publishing or a publish command). Following the Microsoft documents you can explicitly set it to false so the task should look like this now (along with a few other options explicitly set):

...
- task: DotNetCoreCLI@2
  displayName: 'Publish Azure Functions'
  inputs:
    command: 'publish'
    projects: '$(System.DefaultWorkingDirectory)/$(functionProjectAppName)/$(functionProjectAppName).csproj'
    arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)/publish/$(environmentName)/$(functionProjectAppName)'
    publishWebProjects: false
    modifyOutputPath: false
    zipAfterPublish: false
...

Then our release is just a standard Azure Function Deployment Release - ensure you can see the zip file in the location you have chosen. I was extra precautious by having a task that would extract the files from the zip location and then move them into a separate folder location. Then from that location we would deploy the zip file.

The deployment method allowed was to leave it as 'Auto-Detect'.

Potential Pitfalls

I've seen that the 'Auto-Detect' feature will choose 'publish using the package' which it will zip and use a positional argument that publishes the versions that are already in the package.json files. If the settings in Azure portal do not reflect the method in which you are attempting to deploy you will receive another error on publish. As part of the CI/CD pipeline the release step for deploying the actual azure function will set this variable to either zero or one in the portal.

Note that if your release becomes out of sync then it will not deploy. I have seen that the 'manual' publish through visual studio will sometimes change the application settings and then other times it won't.

There are also sources listed in the next section that mention removing the archiving of files from the CI/CD pipeline. I have tried both versions of leaving them in and also removing them and was able to see the functions in their deployed location. It could just be a matter of being conscious of the steps taken to zip your deployment.

Again be warry of your settings.

Sources

• https://stackoverflow.com/questions/66662364/deploying-azure-function-isolated-process-to-azure-using-azure-devops-is-not-w
• https://stackoverflow.com/questions/76255239/deployed-isolated-function-app-returns-404
• https://stackoverflow.com/questions/75503474/azure-functions-deployment-from-git-does-not-work-with-net-7
• https://stackoverflow.com/questions/55623529/azure-function-deployed-successfully-but-not-working
• https://learn.microsoft.com/en-us/azure/azure-functions/functions-how-to-azure-devops?tabs=csharp%2Cyaml&pivots=v1

Thanks for the reply @longmatthew , this worked for me and has solved the issue above, service bus triggers are now showing as expecting. Hopefully this can help @satvu find the root cause.

[satvu]

@longmatthew Thanks for the excellent, detailed writeup! We will review this flow to create more specific work items for improving it and adding instrumentation to better diagnose these kinds of issues for customers.

[satvu]

Closing this as it is primarily a host issue - you can track our first improvement task in the host: Azure/azure-functions-host#10021.