New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Discussion] Run From Package: a new way to deploy your Azure Web Apps and Function Apps (in Preview) #32

Open
davidebbo opened this Issue Feb 15, 2018 · 175 comments

Comments

Projects
None yet
@davidebbo
Contributor

davidebbo commented Feb 15, 2018

Please use this issue to discuss Azure/app-service-announcements#84

@Suchiman

This comment has been minimized.

Suchiman commented Feb 15, 2018

Sounds great!

  • Whats the performance like?
    • Are reads always deflating or is it cached in locally?
    • How does this affect local cache?
  • Is it compatible with all technologies (.NET Core, .NET Framework, etc.)?
  • What happens when you point it to a new zip file?
    • The announcement says "its atomic", is it smooth like slot swap, or
    • Is the zip downloaded, app halted, zip remounted, app restarted?
  • Is the transition to the new zip file synchronous or asynchronous?
    • If async, is there a way to detect when the transition to an updated version of the zip file is completed? (e.g., when slot swapping, the powershell cmdlets return, but requests may hit the old page for about another 30 seconds which is unfortunate for automation)
@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@Suchiman great questions:

  • It does some local caching
  • It shouldn't be used with the Local Cache feature (probably conflicts, untested)
  • It should in theory work with all stacks
  • The App Setting change will restart the app. So it is atomic in the sense that you can't be 'half-baked', but it is not smooth like slot. That's a potential future improvement, at least for apps that have multiple instances
  • It's the same as modifying an Azure App Setting, which is async.
@shibayan

This comment has been minimized.

shibayan commented Feb 15, 2018

It's cool feature.

  • Is there a way to notify that deployment is complete?
    • Similarly, when the deployment fails?
  • Is there a limit on the size of zip file?
  • If I change App Setting while asynchronously deploying, will the deployment abort?
  • Just an idea, How about the feature to deploy from a local machine use Azure CLI?
    • Transparently upload zip to storage and deploy it to App Service
@anthonychu

This comment has been minimized.

Member

anthonychu commented Feb 15, 2018

Does it re-pull the zip file from the specified location on restart or scale out? Just wondering if we need to worry about the SAS URL expiring.

@devlead

This comment has been minimized.

devlead commented Feb 15, 2018

@davidebbo Will/is there be an option in the kudu API to do this kind of deploy this?
Would be nice if similar to zip deploy something like post {"uri": "=https://davidebbostorage.blob.core.windows.net/content/.....D"} to /api/RunFromZip

to get a 202 or 200 if

  1. Request received
  2. Can access uri
  3. Settings changed
  4. Zip downloaded and mounted
@martinpeck

This comment has been minimized.

Member

martinpeck commented Feb 15, 2018

This is a great feature! Thank you!

Is there some guidance for how this changes what should be deployed for your app? For example, I'm assuming that the zip file should include the node_modules folder, right?

@securityvoid

This comment has been minimized.

securityvoid commented Feb 15, 2018

Is there any way to set the appropriate environmental variable from a visualstudio.com build? If so, this should be able to be automated now since you can already zip up the source.

@securityvoid

This comment has been minimized.

securityvoid commented Feb 15, 2018

Alternatively, can this be used with git deploy? e.g. Zip file is returned to git, auto-deploys to environment; ZIP variable points to the file path where the Zip sits in the environment?

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@shibayan Deploying completing is basically equivalent to the site restarting after changing an App Setting. If something goes wrong during zip download, you'll see a file called "FAILED TO DOWNLOAD ZIP FILE.txt" in wwwroot from Kudu with error details.

There is no strict size limit, but it does use the local storage, so in Free/Shared/Consumption really large zips could hit the limit.

The tooling part like Azure CLI is something that would come later and build on top of it.

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@anthonychu Yes, it does repull the zip each time. So:

  • Make the SAS valid as long as you want the site to work
  • Use a storage account in the same region
@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@devlead Note that the zip is not uploaded a part of deployment, but instead that happens at runtime. There is no deployment per se, other than setting the App Setting.

Potentially, Kudu could build the zip from a source project, and do the upload. Though that does raise some questions about it needing write access to the blob storage.

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@martinpeck Yes, it should absolutely include the node_module folder, and basically everything your app needs at runtime. Think of it this way: if you were to deploy via good ol' FTP, you'd copy a bunch of files. That exact same bunch of files is what you want in your zip.

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@securityvoid similar to previous comments. Basically, right now we just have the low level intrinsic that knows how to handle the App Setting pointing to a zip. There can be all kind of deployment workflows built on top of that to create the zip, host it and set the App Setting. I don't know VSTS well, but presumably it's flexible enough that you could be this today with enough custom build logic.

@securityvoid

This comment has been minimized.

securityvoid commented Feb 15, 2018

@davidebbo Thank-you for the response. I'm going to play around with this some. As long as the variable supports pointing to the local file-path there are a lot of work flows that might work here. This should be a great feature!

FYI, in case someone is looking how to set an environment variable from VSTS:
https://marketplace.visualstudio.com/items?itemName=pascalnaber.PascalNaber-Xpirit-WebAppConfiguration

@tony-gutierrez

This comment has been minimized.

tony-gutierrez commented Feb 15, 2018

How would this zip contain node_modules that were built to be compatible with the environment? If I am on a mac and build a node module that uses gyp or is anything other than simple js, and deploy that zip, there is no way it is going to be compatible.

@devlead

This comment has been minimized.

devlead commented Feb 15, 2018

Note that he zip is not uploaded a part of deployment

@davidebbo I get that, but just setting the setting doesn't guarantee the web app can access the zip file. If an api could provide at least some minimal validation, that would give more confidence in the deployment.

And epic if an api could "like" the zip deploy api wait and give a 2xx when web is up or error if failed, that would give immediate feedback to fail or retry from CI.

@devlead

This comment has been minimized.

devlead commented Feb 15, 2018

How would this zip contain node_modules that were built to be compatible with the environment?

@tony-gutierrez Personally I would build and deploy from CI which would be using a Windows agent.

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@tony-gutierrez Yes, that's true. This issue also exists today if you use FTP or msdeploy. The only case where it doesn't exist is when doing Kudu git deployment, since that has a build step that restores packages.

There is also the possibility of solving this via a CI server like VSTS. In such world

  • your sources don't have the packages
  • VSTS restores them at build time and has knowledge about the target environment
  • It zips that up and all is well, without your client having to worry about it

So it may be that 'raw' Run-From-Zip' is a little too low level in some scenarios if used by itself. But the right tooling can make it work correctly (but really, this is no different from msdeploy/FTP today).

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@devlead yes, I think this is something that can be built on top of the low level intrinsics. In fact, you could easily write a command line tool that takes the zip, uploads it to blob, sets the URL, and hits the Web App to make sure everything works.

@Suchiman

This comment has been minimized.

Suchiman commented Feb 15, 2018

@davidebbo but how does hitting the web app make sure everything works? Since its async, it sounds like it would need a Thread.Sleep(30000) to approximiate when the changes took effect.

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@Suchiman is is no different from changing an App Setting in App Service today. You have FOO=1 and change it to FOO=2, and it can take 10 or 20 seconds before the change is live. The very same applies here.

@Suchiman

This comment has been minimized.

Suchiman commented Feb 15, 2018

@davidebbo Not really the answer i hoped for but i guess i can put a version.txt into the zip to workaround it and just poll it every second in the release process to block until the new release is live.

@securityvoid

This comment has been minimized.

securityvoid commented Feb 15, 2018

@Suchiman Why not simply return the environmental variable that points to the zip? If its the new value, you know the new release is in place. Then you don't need anything specifically in the release.

@Suchiman

This comment has been minimized.

Suchiman commented Feb 15, 2018

@securityvoid mh interesting trick, cool yeah that should work, thanks 👍

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 15, 2018

@securityvoid @Suchiman one small twist is that if you have multiple instances, they will not necessarily all big ready at the exact same time. So you might still want to wait a few seconds even if one instance tells you it's ready.

@dfinke

This comment has been minimized.

dfinke commented Feb 16, 2018

The deploy worked. I can seen the function in the function apps blade. When I click on the function, get the error below. I deployed a PowerShell run.ps1

image

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Feb 16, 2018

@dfinke Yep, that's a known issue in the Functions Runtime (not in this new feature) that just got fixed. Until the fix is deployed, just make sure to include a host.json at the root of your zip. It should just contain {}

@dfinke

This comment has been minimized.

dfinke commented Feb 16, 2018

@davidebbo Thanks, yup, I just included it, was coming back to delete my comment and saw your response. This is pretty sweet and will simply function app DevOps.

@arichiardi

This comment has been minimized.

arichiardi commented Aug 22, 2018

I have some objection about this particular feature, and I know that it might be odd to raise it here, but I think it is the only place I feel I can have a conversation going.

If I understand correctly, the new zip will be containing the whole set of functions for the function app. I see the use case there - deployments should be self-contained and atomic - but still I cannot avoid thinking that this does not give me one thing: function deployment isolation.

This feature assumes that all functions have the same dev lifecycle but I am thinking this is not the case: we have different folks working on different functions and it would be awesome if all our deployments would not interfere with each other, effectively atomically uploading and releasing one function at the time.

Have you folks considered this? If yes, can you explain please why you decided against? I would be really curious, maybe the workflow I am imagining is not really a thing 😄

Finally, thank you for this feature, it improves radically the way we deploy still ☮️

@robdmoore

This comment has been minimized.

robdmoore commented Aug 22, 2018

@arichiardi Can you get each person to take on their own separate function app? You can run all of them on the same app service plan so the compute/cost is the same. If you need all of them off a single URL you can use Application Gateway or API Management.

@robdmoore

This comment has been minimized.

robdmoore commented Aug 22, 2018

@davidebbo is run from zip supported on App Service Environments yet?

@arichiardi

This comment has been minimized.

arichiardi commented Aug 22, 2018

Thanks for the prompt response and yes that is what we do but I thought of it as a workaround. If you think that is the right path though, maybe it is not 😄

@robdmoore

This comment has been minimized.

robdmoore commented Aug 22, 2018

@arichiardi I'm not on the App Service team, I'm just a long time user of Azure, but from my perspective a single function app makes sense to be the boundary of a single deployment. I wouldn't call that a workaround.

@JohnReeson

This comment has been minimized.

JohnReeson commented Aug 23, 2018

@davidebbo Got one question. With this new feature, if we want to delete or add functions to one existing function app, we need to redeploy the whole function app, right? Or did I miss some other basic ways?

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Aug 23, 2018

@arichiardi @JohnReeson Generally, Function apps should be deployed as a unit, and not piecemeal. This is true even when not running from a Package. See https://github.com/Azure/Azure-Functions/wiki/Deploying-Function-Apps-vs-individual-Functions for more details.

@robdmoore Yes, I would expect it to work on ASE as well.

@davidebbo davidebbo changed the title from [Discussion] Run-From-Zip: a new way to deploy your Azure Web Apps and Function Apps (in Preview) to [Discussion] Run From Package: a new way to deploy your Azure Web Apps and Function Apps (in Preview) Aug 23, 2018

@JohnReeson

This comment has been minimized.

JohnReeson commented Aug 23, 2018

@davidebbo Sorry for my terrible expressions. I will add some details to explain why there were such confusions.

When I use Web Deploy in VS, saying I only have one function in my local function app, while there are some created before in the destination Azure function app. I can deploy my local function app to Azure with those existing ones retained, as long as I don't click Remove additional files at destination and there's no function name conflict.

Hence Web Deploy leaves me the tricky impression that I am able to add functions without the whole function app content, which is apparently against the deployment principle.

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Aug 23, 2018

@JohnReeson while you can technically do that, it is not a reliable way to deploy because it doesn't propagate deletions. i.e. if you don't check Remove additional files at destination, you can only ever add files, and can never delete any files.

e.g. maybe in some existing function you'd like to rename a helper file, effectively adding a file and removing one. But you just won't be able to do it (you'll end up with both files), and will just accumulate garbage over time, leading to inconsistent behavior.

So really, the only valid way to use WebDeploy with Functions is to remove unknown files. Not doing this is asking for trouble.

@JohnReeson

This comment has been minimized.

JohnReeson commented Aug 23, 2018

@davidebbo Totally agree. Looks like VS team has also realized this and enabled Remove additional files at destination by default. I mentioned delete before because I can still delete functions on portal/kudu after Web Deploy. While right now function app is truly read-only mode set by run from package, every operation should be done by redeployment. This is the better way indeed, thanks again!

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Aug 23, 2018

@JohnReeson You're absolutely correct: running from a package makes it impossible to do this at all. While with alternative, as much as we're trying to discourage/prevent it, you can always find a way to modify individual files outside of a complete deployment.

@jakenuts

This comment has been minimized.

jakenuts commented Sep 11, 2018

Loading files from the zip with external tools?

As a very-low-priority question, I had been using the main function .csx to call "dotnet subfolder\mycoreapp.dll" to run a console app (similar to 'Run Console Apps on Azure Functions' but for .NET core). Obviously this might not fly when the console app is contained in a zip file as the path is mounted not standard file system. Is that something that can be overcome by using a different path (i.e. is there a path I can pass to dotnet.exe to identify & load a dll within the zip)?

Thanks!

James

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Sep 11, 2018

@jakenuts not sure I follow. Once the zip is mounted, I would expect it to work just like a normal file system, except for the fact that it's read-only.

@ishepherd

This comment has been minimized.

ishepherd commented Sep 12, 2018

Please confirm this works for GA Azure Functions 1.x, as well as 2.0?
(Not got through the whole thread yet, but I saw a '2.0' comment on related #2230)

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Sep 12, 2018

@ishepherd Yes, it is really agnostic of what you're running: Functions App of any version, Web App of any stack, etc...

@graemechristie

This comment has been minimized.

graemechristie commented Sep 14, 2018

I just implemented this on our App Service Environment, using zip deploy and WEBSITE_RUN_FROM_PACKAGE=1. On our V2 App Service Environment, it works fine. On our V1 ASE, the deployment actually works, however - when we explore the file system using Kudu/Command Prompt (so essentially using the vfs api), we still see the hostingstart.html file in site/wwwroot - rather than the files mounted from the zipfile. Everything else seems to work fine - we can browse the .Net Core site and it works as expected - but the vfs api will return the following:

image

@estei

This comment has been minimized.

estei commented Sep 14, 2018

I observed the same as @graemechristie above in my really short testing the other day.
This was not in ASE though, but in a regular web app.

We use a newly created staging slot when deploying.
After some deployments it would it would show the hostingstart.html file in wwwroot.

After others it would show the contents of the zip file.

Just like Graeme though, the node.js app was working as it should, we just didn't see the right content in wwwroot when coming in through kudu.

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Sep 14, 2018

@graemechristie @estei that is odd. To help isolate:

  • does restarting the app consistently make it work?
  • does it happen regardless of the zip? e.g. try on a test app with a zip that just has a hello world HTML
@graemechristie

This comment has been minimized.

graemechristie commented Sep 15, 2018

I'll be able to do some more testing next week with other content - however I can confidently say that restarting the app does not make any difference. From @estei's comments it also looks like the ASE v2 vs ASE v1 situation may just be a coincendence (I have only deployed 2 apps so far).

Oh - And we are also using staging slots and swaps to deploy.

@tomasherceg

This comment has been minimized.

tomasherceg commented Sep 15, 2018

Are there any ways to improve error messages that occur when the application wants to write in the filesystem? Since the Zip Deploy is the default option in Azure App Service Deploy task in VSTS and I bet most users haven't heard about Run from Package, so this may be very confusing.

For example, Directory.CreateDirectory and mkdir returns Could not find file 'D:\home\site\wwwroot\Temp'..

Creating a directory in Kudu in wwwroot returns 409 Conflict: Cannot delete directory. It is either not empty or access is not allowed.

I have written a blog post about that - I spent a couple of hours until I discovered that the website runs from a package and that VSTS uses Zip Deploy by default.

@jsheetzati

This comment has been minimized.

jsheetzati commented Oct 2, 2018

@davidebbo Does Run From Package support self-contained deployments for .Net Core? I could not find anything in the documentation nor this GitHub issue.

@davidebbo

This comment has been minimized.

Contributor

davidebbo commented Oct 2, 2018

@jsheetzati yes, that should work. In theory, it should support any scenario where nothing tries to write to wwwroot at runtime.

@alicejgibbons

This comment has been minimized.

alicejgibbons commented Oct 19, 2018

Agreeing with @tomasherceg here...

After updating my Azure App Service Deploy Task to version 4 Preview it was not obvious that the wwwroot folder was going to be overwritten and then changed to Read Only.

@davidebbo it would be great if we could have some better notifications around this on both the Kudu side and in the Deployment Task itself as it is a drastic change from version 3 to version 4.

@c-eliasson

This comment has been minimized.

c-eliasson commented Oct 23, 2018

@davidebbo I'm using the 2.0 version of the runtime, where support for things like queue triggered functions have to be installed separately as extensions.

Since wwwroot is replaced with every deploy, I'm thinking I would need to include extensions.csproj and the /obj and /bin directories in the zip. Would that be the right way to approach it, or is there a better way?

@devlead

This comment has been minimized.

devlead commented Oct 23, 2018

@c-eliasson you don't need to include obj or csproj , extensions are nuget packages and endup in bin.

If you publish to a directory either through studio or .NET CLI dotnet publish .\funcv2.csproj --output ./output regardless if you're using csx or not.

This will precompile the function and the output will look something like this
image

And the wwwroot will look something like this
image

The bin folder will contain all the extensions specified by the csproj. And your function app will look something like this
image

@bennybauer

This comment has been minimized.

bennybauer commented Oct 23, 2018

@devlead doesn't it depend on which OS I'm publishing the extensions? Can I publish it on linux (my CI host) and then deploy it to windows (my functions host)?

@c-eliasson

This comment has been minimized.

c-eliasson commented Oct 23, 2018

@devlead Sorry, I failed to mention that I'm using Node not .NET

Currently I'm deploying the project through Azure DevOps pipes.

This is basically the build process:

  1. Install npm dependencies
  2. Build the project (compile TypeScript etc)
  3. Install azure-functions-core-tools@core through npm
  4. Install extensions with func extensions install
  5. Zip the directory
  6. Upload the zip to blob storage

And the release process:

  1. Create SAS for the zip in blob storage
  2. Set the complete path to the zip in application settings

This seems to work fine. I'm curious as to whether this is the best way to do it for a NodeJS Functions App?

@devlead

This comment has been minimized.

devlead commented Oct 23, 2018

@c-eliasson Ah I see, my process is similar for node apps, I use Azure Pipelines and a build script for node functions, basically the steps are

  1. dotnet build extensions.csproj -o bin
  2. build js app
  3. zip bin, function folders and host.json
    Zip looks something like this
    image
    and wwwroot
    image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment