[BUG]: 404 Failed to deploy web package to App Service (Linux)

[Eazyed]

Task name

AzureRmWebAppDeployment

Task version

4

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows

Task log

##[debug][POST]https://$contoso-project-int-ase-front:***@contoso-project-int-ase-front.scm.azurewebsites.net/api/zipdeploy?isAsync=true&deployer=VSTS_ZIP_DEPLOY&message=%7B%22type%22%3A%22deployment%22%2C%22commitId%22%3A%22ca18ed0cf5370f2079a1256627325eb63b909842%22%2C%22buildId%22%3A%226477%22%2C%22buildNumber%22%3A%22Extranet_release-v1.1.7_2023.08.01.3%22%2C%22repoProvider%22%3A%22TfsGit%22%2C%22repoName%22%3A%22contoso-Extranet%22%2C%22collectionUrl%22%3A%22https%3A%2F%2Fdev.azure.com%2Fcompany-unit%2F%22%2C%22teamProject%22%3A%220d311dd0-de9f-4189-b738-60f14ef6869d%22%2C%22buildProjectUrl%22%3A%22https%3A%2F%2Fdev.azure.com%2Fcompany-unit%2F0d311dd0-de9f-4189-b738-60f14ef6869d%22%2C%22repositoryUrl%22%3A%22https%3A%2F%2Fcompany-unit%40dev.azure.com%2Fcompany-unit%2Fcontoso-Extranet%2F_git%2Fcontoso-Extranet%22%2C%22branch%22%3A%22release-v1.1.7%22%2C%22teamProjectName%22%3A%22contoso-Extranet%22%2C%22slotName%22%3A%22production%22%7D
##[debug]ZIP Deploy response: {"statusCode":202,"statusMessage":"Accepted","headers":{"content-length":"0","connection":"close","date":"Tue, 01 Aug 2023 08:08:46 GMT","server":"Kestrel","location":"https://contoso-project-int-ase-front.scm.azurewebsites.net:443/api/zipdeploy?isAsync=true&deployer=VSTS_ZIP_DEPLOY&message=%7B%22type%22%3A%22deployment%22%2C%22commitId%22%3A%22ca18ed0cf5370f2079a1256627325eb63b909842%22%2C%22buildId%22%3A%226477%22%2C%22buildNumber%22%3A%22Extranet_release-v1.1.7_2023.08.01.3%22%2C%22repoProvider%22%3A%22TfsGit%22%2C%22repoName%22%3A%22contoso-Extranet%22%2C%22collectionUrl%22%3A%22https%3A%2F%2Fdev.azure.com%2Fcompany-unit%2F%22%2C%22teamProject%22%3A%220d311dd0-de9f-4189-b738-60f14ef6869d%22%2C%22buildProjectUrl%22%3A%22https%3A%2F%2Fdev.azure.com%2Fcompany-unit%2F0d311dd0-de9f-4189-b738-60f14ef6869d%22%2C%22repositoryUrl%22%3A%22https%3A%2F%2Fcompany-unit%40dev.azure.com%2Fcompany-unit%2Fcontoso-Extranet%2F_git%2Fcontoso-Extranet%22%2C%22branch%22%3A%22release-v1.1.7%22%2C%22teamProjectName%22%3A%22contoso-Extranet%22%2C%22slotName%22%3A%22production%22%7D/api/deployments/latest?deployer=VSTS_ZIP_DEPLOY&time=2023-08-01_08-08-46Z","scm-deployment-id":"a075012a-fa1d-4691-9f2e-c7f5555b1eba","retryafter":"30"}}
##[debug]Polling for ZIP Deploy URL: https://contoso-project-int-ase-front.scm.azurewebsites.net:443/api/zipdeploy?isAsync=true&deployer=VSTS_ZIP_DEPLOY&message=%7B%22type%22%3A%22deployment%22%2C%22commitId%22%3A%22ca18ed0cf5370f2079a1256627325eb63b909842%22%2C%22buildId%22%3A%226477%22%2C%22buildNumber%22%3A%22Extranet_release-v1.1.7_2023.08.01.3%22%2C%22repoProvider%22%3A%22TfsGit%22%2C%22repoName%22%3A%22contoso-Extranet%22%2C%22collectionUrl%22%3A%22https%3A%2F%2Fdev.azure.com%2Fcompany-unit%2F%22%2C%22teamProject%22%3A%220d311dd0-de9f-4189-b738-60f14ef6869d%22%2C%22buildProjectUrl%22%3A%22https%3A%2F%2Fdev.azure.com%2Fcompany-unit%2F0d311dd0-de9f-4189-b738-60f14ef6869d%22%2C%22repositoryUrl%22%3A%22https%3A%2F%2Fcompany-unit%40dev.azure.com%2Fcompany-unit%2Fcontoso-Extranet%2F_git%2Fcontoso-Extranet%22%2C%22branch%22%3A%22release-v1.1.7%22%2C%22teamProjectName%22%3A%22contoso-Extranet%22%2C%22slotName%22%3A%22production%22%7D/api/deployments/latest?deployer=VSTS_ZIP_DEPLOY&time=2023-08-01_08-08-46Z
##[debug]setting affinity cookie 
##[debug]CLIENT_RESETSTREAMONRETRY=undefined
##[debug][GET]https://contoso-project-int-ase-front.scm.azurewebsites.net:443/api/zipdeploy?isAsync=true&deployer=VSTS_ZIP_DEPLOY&message=%7B%22type%22%3A%22deployment%22%2C%22commitId%22%3A%22ca18ed0cf5370f2079a1256627325eb63b909842%22%2C%22buildId%22%3A%226477%22%2C%22buildNumber%22%3A%22Extranet_release-v1.1.7_2023.08.01.3%22%2C%22repoProvider%22%3A%22TfsGit%22%2C%22repoName%22%3A%22contoso-Extranet%22%2C%22collectionUrl%22%3A%22https%3A%2F%2Fdev.azure.com%2Fcompany-unit%2F%22%2C%22teamProject%22%3A%220d311dd0-de9f-4189-b738-60f14ef6869d%22%2C%22buildProjectUrl%22%3A%22https%3A%2F%2Fdev.azure.com%2Fcompany-unit%2F0d311dd0-de9f-4189-b738-60f14ef6869d%22%2C%22repositoryUrl%22%3A%22https%3A%2F%2Fcompany-unit%40dev.azure.com%2Fcompany-unit%2Fcontoso-Extranet%2F_git%2Fcontoso-Extranet%22%2C%22branch%22%3A%22release-v1.1.7%22%2C%22teamProjectName%22%3A%22contoso-Extranet%22%2C%22slotName%22%3A%22production%22%7D/api/deployments/latest?deployer=VSTS_ZIP_DEPLOY&time=2023-08-01_08-08-46Z
##[debug]Could not parse response: {}
##[debug]Response: undefined
##[error]Failed to deploy web package to App Service.
##[debug]Processed: ##vso[task.issue type=error;]Failed to deploy web package to App Service.
##[debug]Deployment Failed with Error: Error: Error: Failed to deploy web package to App Service. Not Found (CODE: 404)
##[debug]task result: Failed
##[error]Error: Error: Failed to deploy web package to App Service. Not Found (CODE: 404)

....

##[debug][PUT]https://$contoso-project-int-ase-front:***@contoso-project-int-ase-front.scm.azurewebsites.net/api/deployments/64771690877327941
##[debug]Could not parse response: {}
##[debug]Response: undefined
##[debug]updateDeployment. Data: {"statusCode":409,"statusMessage":"Conflict","headers":{"connection":"close","content-type":"text/plain; charset=utf-8","date":"Tue, 01 Aug 2023 08:08:47 GMT","server":"Kestrel","transfer-encoding":"chunked"},"body":"The 'Performing deployment' operation conflicts with the pending 'Deployment' operation started at 08/01/2023 08:08:46.  Please retry operation later."}
##[warning]Error: Failed to update deployment history. Error: Conflict (CODE: 409)

Relevant log output

##[error]Failed to deploy web package to App Service.
##[error]Error: Error: Failed to deploy web package to App Service. Not Found (CODE: 404)

...

##[warning]Error: Failed to update deployment history. Error: Conflict (CODE: 409)

Aditional info

The deployment actually works and is visible in the deployment log. 
However the task fails with this 404 error. It looks like the polling for deployment status is failing

This error appeared suddenly across environments. (Across different App plan instances/SKU)

[leivavictor]

Hi Eazyed, I am facing the same issue. In my case I also redeployed an environment and it worked just for 3 deployments but now it came back to the same behavior.

[vivelys]

Hi Eazyed and leivavictor,
I'm facing the same problem.
The deployment appears to have failed, but when I look in my web app log, the correct version is deployed.
However, as the deployment appears to have failed, my pipeline stops.

[Eazyed]

Hello @leivavictor and @vivelys.

The task is still usable (kind of) , as this is only the polling for deployment status that fails, deployment is actually launched. So if you don't have any task that depends directly on the deployment in your pipeline you can simply set the task to continue on error for now.

By the way, this problem did not appear in another similar project.

[GabrielKropp]

Hello @Eazyed , I've exactly the same issue in one of my app service deployment, so I'll watch if a resolution will be found here.

[richsage]

Same thing here - except we're using AzureWebApp - same errors though, so wonder if there is something underlying supporting these

[mandaniainarandriampanana]

same error for me too

[borjasanes]

Same issue:

Run azure/webapps-deploy@v1
Package deployment using ZIP Deploy initiated.
Error: Failed to deploy web package to App Service.
Error: Deployment Failed with Error: Error: Failed to deploy web package to App Service.
Not Found (CODE: 404)
Error: Error: Failed to deploy web package to App Service.
Not Found (CODE: 404)

[sbergot]

We had the exact same issue. He deleted the app service and recreated a new one and it fixed the issue.

[DariusDavidavicius]

Deploy WebApp to Azure
1 Run azure/webapps-deploy@v2
11 Package deployment using ZIP Deploy initiated.
12 Error: Failed to deploy web package to App Service
13 Error: Deployment Failed, Erro: Failed to deploy web package to App Service.
14 Not Found (CODE: 404)
15 App Service Application URL: http://

problems started from 2023.08.01 it was all fine for 2 years.

[jammyzt]

We deploy our app services in dev via bicep each morning and release the code via ado. They’re torn down each night. This has worked beautifully for a few months and started failing on Tuesday.

We had the exact same issue. He deleted the app service and recreated a new one and it fixed the issue

Unfortunately, we’re not having this joy. Our brand new Functions on brand new ASPs are “failing” to deploy. Same story as others, the function does deploy but ado thinks it has failed

[jammyzt]

Hello all!
I've just reached the office and all 30 linux function apps have deployed correctly!

There was the odd hiccup on two of them but they immediately retried and successfully deployed.

@Eazyed How are things for you today? :)

[DariusDavidavicius]

@jammyzt so what helped? this tip:
"deleted the app service and recreated a new one and it fixed the issue"?

[jammyzt]

@jammyzt so what helped? this tip: "deleted the app service and recreated a new one and it fixed the issue"?

No change on our side. Our app services were always destroyed every night and recreated in the morning.

We suddenly saw the issues on Tuesday morning and they continued until yesterday. This morning, everything is working after no change on our side.

We have an open support ticket with our friends at Microsoft but have had no response beyond acknowledgement of the ticket. I suspect something has been fixed on their side.

[Eazyed]

Hello @jammyzt,

Just did a test run and it seems to be working correctly for an app that was consistently failing until then. I'm leaving the issue open in case this has not been fixed for everyone.

[vivelys]

The "delete/recreate" tip did the trick for me.
The web app slots where I didn't apply this workaround still reply the deployment failed

[richsage]

No change here just now despite deleting & recreating both app service (including additional slot) and app service plan - still fails :(

[alphaousbah]

Same here. My deployments used to work, and they don't anymore since yesterday :
Deployment Failed, Error: Failed to deploy web package to App Service. Site Not Found (CODE: 404)

The delete/recreate app service trick doesn't work for me either

[v-gayatrij]

Does this issue still persist or is it resolved? Can anyone please confirm?

[alphaousbah]

Hi, the problem was gone for me the day after my post. Sorry I didn’t give an update.
I coding in Python, push my code to GitHub, and I have set a CI CD connection in the deployment center of the azure app service.
I think the problem was on azure’s side because I didn’t update my code or anything

[v-mohithgc]

Thanks for confirming, will be closing this for now. re-open if the issue reoccurs from the task end.

[martin-greentrax]

I have the same issue suddenly on unchanged config while this deployment job worked for 2 years :(
Deleting+recreating the app service is not an option for me as this is a live prod application.

[v-mohithgc]

I have the same issue suddenly on unchanged config while this deployment job worked for 2 years :( Deleting+recreating the app service is not an option for me as this is a live prod application.

Hi @martin-greentrax , can you please let us know the previous successful run task version and the recent failure run task version ( eg: 4.233.2 )

[martin-greentrax]

@v-mohithgc sorry, I m not sure if this is the correct repository to report the issue. We did not change the deployment job so I m not aware of any version changes.

In our github workflow yml we are using this since a long time: Azure/functions-action@v1

While debugging now we also tried v1.5.1, but no success with it either.

Here is how the github job log starts with debug logging:

Current runner version: '2.311.0'
Operating System
Runner Image
  Image: ubuntu-22.04
  Version: 20240116.3.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240116.3/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240116.3
Runner Image Provisioner
  2.0.324.1

And this is how it fails:

##[debug]Deployment status: 0 'Building and Deploying '***-4e62-871a-d8bdc4ef87f0'.'. retry after 5 seconds
##[debug]setting affinity cookie ["ARRAffinity=****a443cf3f6c7c2e4daaee92c2062af1b69e3649bf26d43fe66fe700a7de;Path=/;HttpOnly;Secure;Domain=***.scm.azurewebsites.net","ARRAffinitySameSite=****443cf3f6c7c2e4daaee92c2062af1b69e3649bf26d43fe66fe700a7de;Path=/;HttpOnly;SameSite=None;Secure;Domain=***.scm.azurewebsites.net"]
##[debug][GET] [https://***.scm.azurewebsites.net:443/api/deployments/latest?deployer=GITHUB_ZIP_DEPLOY_FUNCTIONS_V1&time=2024-01-23_11-20-47Z](https://***.scm.azurewebsites.net/api/deployments/latest?deployer=GITHUB_ZIP_DEPLOY_FUNCTIONS_V1&time=2024-01-23_11-20-47Z)
##[debug]Could not parse response body.
##[debug]{}
Error: Failed to deploy web package to App Service.
Error: Execution Exception (state: PublishContent) (step: Invocation)
Error:   When request Azure resource at PublishContent, zipDeploy : Failed to use /home/runner/work/_temp/temp_web_package_8472975363497814.zip as ZipDeploy content
Error:     Failed to deploy web package to App Service.
Not Found (CODE: 404)
Error:       Error: Failed to deploy web package to App Service.
Not Found (CODE: 404)
    at Kudu.<anonymous> (/home/runner/work/_actions/Azure/functions-action/v1/lib/appservice-rest/Kudu/azure-app-kudu-service.js:235:41)
    at Generator.throw (<anonymous>)
    at rejected (/home/runner/work/_actions/Azure/functions-action/v1/lib/appservice-rest/Kudu/azure-app-kudu-service.js:6:65)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Error: Deployment Failed!
##[debug]Node Action run completed with exit code 1

[martin-greentrax]

The issue has gone away after experimenting for a long time. Hope this helps someone else googling:

• I did add the environment variables that are suggested in other threads about the issue but can't say that they helped
• Go to your function on portal.azure.com and in the Overview blade at the bottom click on your Function "app"
• I see this message there:
  "Your app is currently in read only mode because you are running from a package file. To make any changes update the content in your zip file and WEBSITE_RUN_FROM_PACKAGE app setting."
• I clicked on Disable
• Then added WEBSITE_RUN_FROM_PACKAGE=0 in the blade Environment variables (it was not part of the variables before)
• Restarted the function, enabled the "app" again and my github job went through again finally

Update: The issue came back a week later :( repeated the above steps but also deleted the "app". On the Overview blade on the bottom right there is a "..." menu next to "app" where you can delete it. On re-starting/re-building it will come back. Somehow that may have helped this time. I had 6 failed builds and then a green one again. Annoying.

[Blaze404]

One thing I have noticed is- If I change any setting in my webapp while its being deployed it usually gives that error. And it gets deployed correctly If I just do a simple redeploy ( In my case add a little space somewhere in code and git push again )

[JaggerH]

I discover this problem when I diff files one by one.

git diff <commit-hash> -- <file-path>

And I found this line below.

diff --git a/requirements.txt b/requirements.txt
Binary files a/requirements.txt and b/requirements.txt differ

I don’t think it reasonable for a .txt type of file identify as binary.
I think this problem is because requirements.txt is encoded in UTF-16 LE in this commit.

I just checkout to previous version, and everything work fine again.

git checkout <commit-hash> -- <file-path>