New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
app_offline.htm file not being consistently removed by Kudu #2949
Comments
@ahmelsayed, @suwatch, can we take a look at this? We also received a CRI about this and multiple customers have experienced it. |
Count me as another customer affected by this. |
We will investigate and update .. |
Here is the code here and here creating and removing app-offline during deployment. There should be a trace in the deployment log that we create and remove the app-offline file. There is one condition where we may not remove the app_offline file if content has changed (which should be rare). If you have the site repro in that state - we would like to have a look and get to the bottom. If you don't really care about this app_offline (you can opt out by adding this appSettings). |
Thanks for the response and code samples. I'll be sure to report in when we next have a site actively in that state (Saturday morning now, but should be early next week). I have considered avoiding the app_offline file altogether but it's big value is stopping the staging site to reduce file lock errors during deployment. We still get those sometimes even with the app_offline present (perhaps graceful shutdowns are not completed first?) but I'm eager to learn more if there's a better suggestion for increasing consistency of the builds. |
FWIW we haven't seen the issue this week, so we're going back to auto-swap for production. I'd like to leave this issue open since others have seen it too in case anybody else can offer the request from suwatch to see a site that is actively in that state. I'll certainly come back to report it when/if we next see the issue. |
Hello there, I am trying to deploy an angular app to azure and I am having difficulties with the app_offline.htm file as well. I feel like there is a little bit of magic going on here, can someone elaborate how app_offline fits into existing kudu-scripts, where is this declared and created? The angular app I created resides here, as well as the kudu script: https://github.com/BigBubbus/beraterboard-test How can I instruct Kudu to remove app_offline after deployment, I still go on and delete it manually. Thanks for every tip here :) |
The app_offline creation and removal is part of KuduSync.Net. It is supposedly removed after we are done with deploying bit to destination. You can opt out by setting appSettings SCM_CREATE_APP_OFFLINE=0. We are interested to see why it was left there. Do provide us the UTC time of incident and webapp, we can help taking a look. @ankitkumarr for fyi. |
We had this issue last night on our production instance. Even after multiple deploys, it still would not remove w/o us manually removing the file. Regrettably this put us in a severe outage for hours before technical teams thought to check for the file. What can be done to find out why this didn't remove and prevent in the future? Is someone actively looking into this issue? |
A customer of mine has also encountered this issue. It took several days of a premier ticket to figure out the cause, but thankfully was in a pre-production environment. What is the resolution/status of this issue? |
To help guide others here to the root cause Evenlog.xml messages that appear over and over: FREB Log message: |
We have experienced this as well with an automatic deployment from GitHub, just now. |
I apologize that this still was an issue. Usually when this happens, it is likely that the deployment itself failed because we were unable to remove that |
@ankitkumarr Thank you for such a quick and helpful response :) |
Are there any updates on this? This is causing major issues for us every time we deploy our .net core production site. We've tried deploying from VS for mac, VS for windows, and through Devops (including swapping slots) and every time we have this issue. MS support is recommending we set |
We're also seeing this issue. |
I apologize for how long it took me to get back. We are starting to work on a fix to avoid such issues and this will be completed by the end of this month. As Suwat mentioned above, the way kudu deployments work today is that before deployment we create an We will do our best to make this resilient and work on ways to ensure the file is always cleaned, but it would help to look at case by case on why this file was not removed in the first place. If you do see this issue, please share with us the app name and a time-frame so we can investigate. Setting |
Me too, please fix soon. |
@nicholastic and @badescuga, can you share you app name and the time period you saw the issue? |
This is critical issue not solved for months ! |
@djsowa Do consider SCM_CREATE_APP_OFFLINE = 0 on staging slot - make it slot settings. |
@suwatch Yes, we are making this change with SCM_CREATE_APP_OFFLINE. |
This just happened again on help-keyman.azurewebsites.net (I was notified at 2020-01-11T01:16:11.0610722Z) after a deployment started at 12:09:32 PM GMT+11 (which succeeded) (commit id 2b836a8). I have manually removed app-offline.htm in order to make sure the site is available, but nothing else has currently changed on the site. Nothing visible in the logs. |
This happened to us on two separate web app services last weekend. Both times were Kudu Azure CI builds from a bitbucket repo. Issue is still happening. Nothing visible in the logs/UI, the UI indicates the build has succeeded as per normal. Hair-pulling for sure. |
The fix for this is completed and merged in. The release will be starting this week and should be deployed by the end of next week. |
@ankitkumarr Is this fix released now? If so, in what version, or how can we check that we got the fix? Thanks! |
Yes, the deployment was just completed today. If you go to your kudu, you should see the version = (or >) "85.11226.4297.0" That means you should have the fix, where leftover "app_offline.htm" by a failed deployment is auto deleted by kudu. |
Closing as this is resolved and deployed. If you do see this again, please feel free to "@" me here, with necessary information (site name, timestamp) and we can take a look. |
@ankitkumarr I am seeing this issue still on an Azure Functions app being deployed through ADO with the "Take app offline" box selected. Our setup is to deploy to a slot, then swap the slots. |
@ankitkumarr This happened for me today. Azure App Service, fl-cc-am-beta.azurewebsites.net. Deployed through CI/CD for years at this point w/o issue. Last update on CI pipeline was 5/21/21, and the last release (ran as part of PR check) did fail w/ ERROR_FILE_IN_USE. Kudu build 94.30524.5227.0, so maybe this was left over when the deployment failed? |
We started noticing a couple weeks ago that occasionally the
app_offline.htm
file is not being removed at the end of the Kudu process. This results in the site remaining offline with a 503 Site under construction error.We mentioned it over at Azure/app-service-announcements#137 and several other people chimed in they were seeing it as well.
The issue is not happening consistently, but it is happening fairly regularly for us (~20% of the time). We've temporarily disabled auto-swap and are manually checking for the
app_offline.htm
file after a deploy, manually deleting it when present and proceeding with a swap.Please let me know what information you would like for me to provide (the template didn't seem completely applicable) and thanks for any help you can offer!
The text was updated successfully, but these errors were encountered: