Extension does not play well with Run-From-Zip deployment

[smoelker]

Microsoft has introduced a new way to deploy your WebApps to Azure called Run-From-Zip that will be out of preview at the end of Q3. This method basically allows you to mount a zip file to the wwwroot folder of your app at startup. This comes with some great benefits but also a big downside: the wwwroot folder becomes read-only.

The Let's Encrypt site extension relies on writing the to the wwwroot folder (or whatever folder is specified in letsencrypt:WebRootPath) for serving the answer of the ACME challenge.

I've tried using the extension with Run-From-Zip deployment but the process of requesting a certificate fails. The Let's Encrypt error report tells me that the request to https://my-site.ext/.well-known/acme-challenge/{challenge} failed because of a HTTP 404 (Not Found).

I would expect this process to fail since the wwwroot folder is frozen (well, actually I would expect the process to fail at writing the answer to the wwwroot folder but I might not completely understand the internals of the Run-From-Zip method).

My current workaround is to have the extension write the answers to a different folder (d:\home\data\letsencrypt\challenges) by setting the letsencrypt:WebRootPath and have the application serving any requests to /.well-known/acme-challenge/{challenge}.

A better solution would be if the extension did not rely on writing to the wwwroot folder for serving the answers but handled it internally by having a handler listening to the path /.well-known/acme-challenge/{challenge} instead.

[joeri-juramento]

Thanks for leaving this here. I found your message a bit later than I wished I did.

My problem was indeed that the error I received was that ACME "could not reach the token". But the token was never written in the .well-known folder because - as you mentioned - the wwwroot folder is read-only in RunFromZip.
I got wind of this new method within the deploy step of VSTS pipeline. My webapp deploy step was v4 and the little information icon referred me to a page explaining run-from-zip method with read-only and in Azure I found the application setting RUN_FROM_ZIP set to 1 so I put 1 and 2 together.

However, setting a 0 to that setting does not solve the problem ;).

On top of that, my angular app was responsible for routing so I got a bit confused if this should be solved before angular or within angular.

So, before I got to trying out your solution, I got it to work by downgrading the deployment method.

This finally worked after many different combinations:

1. No special web.config in the root folder.
2. Letencrypt will generate a web.config in the Acme-Challenge subfolder containing the snippet bellow.
3. Downgraded my deploy step from v4 to v3. (So VSTS-agent will just unpack the zip.)
4. Removed the run_from_zip setting in application settings of webapp in azure.

web.config.in.subfolder.txt - When the lets encrypt extension has write access to its acme_challenge folder, I think it generates this web.config file. This is not a file for in the root wwwroot folder.

I will keep your option in mind, while keeping an eye on this thread.

[xt0rted]

I just updated one of my sites to use the run from package setup and the way I worked around the .well-known folder issue is like so (taken from projectkudu/kudu#2849 (comment)):

1. Create a folder called LetsEncrypt under D:\home\site
2. Create a folder called .well-known under D:\home\site\LetsEncrypt
3. In the site's Application settings create a new virtual path called /.well-known which points to site\LetsEncrypt\.well-known
4. Add an App Setting called letsencrypt:WebRootPath with a value of D:\home\site\LetsEncrypt
5. Once the site's using run from package uninstall and reinstall the Let's Encrypt Site extension, this looks to force the webjob to be setup outside of the wwwroot folder

My site doesn't use the .well-known folder right now, but on my test site I did and had that working by:

1. Creating a virtual path called /.well-known which points to site\wwwroot\wwwroot\.well-known (or where ever your real folder is stored)
2. Creating a second virtual path called /.well-known/acme-challenge which points to site\LetsEncrypt\.well-known\acme-challenge

[Alexei-B]

I didn't want to use a virtual path as @xt0rted did, however, I managed to get this to work by:

1. Setting the letsencrypt:WebRootPath environment variable to a different location under the home directory.
2. Modifying the "letsencrypt" function to read from that directory.

Sample Code (.NET Standard):

{
    log.LogInformation($"Processing Let's Encrypt Request for Code: {code}");

    var content = File.ReadAllText(@"D:\home\lets-encrypt\.well-known\acme-challenge\"+code);
    return new OkObjectResult(content);
}

This approach worked just fine for me and it is much easier to automate without the extra virtual directory.

[alastairs]

@Alexei-B's approach worked for us too, after a lot of frustration today. @sjkp Please could the wiki be updated with these instructions? 😄 I can't see any way on the Functions wiki page to contribute this change myself.

[iyerusad]

This was instrumental in getting me unblocked and cert issued. Thanks everyone.

Couple of additional wiki notes/my findings as a new comer:

• Proxy doesn't appear to be preview feature anymore, can safely remove that section on how to enable
• Is %WEBSITE_HOSTNAME% documented somewhere? Presumably this is an application setting that a user would set in azure function application settings, or perhaps its available by default? Unclear if that is variable that needs to be defined or is some standard variable.
• Suggestion: Switch %WEBSITE_HOSTNAME% to localhost to be inline with official docs.
• Suggestion: Switch {*rest}/{rest} entries to {*restOfPath}/{restOfPath} to be more easily understood, as exampled by official docs.
• Not a C# dev, but it seems c# example is missing route component Route = "letsencrypt/{code}" in trigger?

public static class letsencrypt
{
    [FunctionName("letsencrypt")]
    public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = "letsencrypt/{code}")]HttpRequestMessage req, string code, TraceWriter log)
    {
        log.Info($"C# HTTP trigger function processed a request. {code}");
        var content = File.ReadAllText(@"D:\home\letsencrypt\.well-known\acme-challenge\"+code);
        var resp = new HttpResponseMessage(HttpStatusCode.OK);
        resp.Content =  new StringContent(content, System.Text.Encoding.UTF8, "text/plain");
        return resp;
    }
}

Additionally, in light of shifting limitations of wwwroot (mine was not locked/frozen, but deployment path seems to be embracing that more for Azure Web Apps), general installation should consider default recommending to installing outside of wwwroot folder. e.g. set letsencrypt:WebRootPath appplication setting to d:\home\letsencrypt

[sjkp]

@iyerusad a few comments, thanks for your feedback.

%WEBSITE_HOSTNAME% is well supported, it just expands an environment variable. The WEBSITE_HOSTNAME environment variable is always present it is set by the app service. It is something newer that they support localhost.

The route in the C# is not needed as it is implicit from the function and parameter name.

I can't really change the default behavior of writing to wwwroot, as it would be a breaking change, unless the extension in some otherway can ensure that it work without the users configuring anything extra. I can't think of a way to do that atm. Suggestions are welcome :)

Also please take a look at release 0.8.9 it now supports not writing the challenge file to disk, but instead writing it to blob-storage, that could maybe make life easier for those of you that use deploy from zip.

[danzel]

Looks like VSTS deploys default to run from zip now, the Kudu error message has changed too (pasting to make this issue easier to find in search)

Server Error in '/letsencrypt' Application.
Response status code does not indicate success: 409 (Conflict).

And the stack trace starts with:

[HttpRequestException: Response status code does not indicate success: 409 (Conflict).]
   System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() +224
   LetsEncrypt.Azure.Core.<PutFile>d__11.MoveNext() in D:\a\1\s\LetsEncrypt.SiteExtension.Core\KuduRestClient.cs:71

[sjkp]

@danzel thanks for letting me know. Guess I have to play around with deploy from zip, to make sure it works as smooth as possible.

[Justincale]

@xt0rted Thanks so much, this works a treat after a few days of turmoil. !! :)

[tomaszzmuda]

In the version 0.9.3 right now

[sjkp]

@tomaszzmuda nothing have been done to fix the problem from the extension side of things.

[Misaka-0x447f]

azure pipeline user, meet the same problem.

[Misaka-0x447f]

plus, microsoft maybe going to work on it? https://feedback.azure.com/forums/170024-additional-services/suggestions/16957756-add-integration-with-let-s-encrypt

[sjkp]

Have you heard anything new from Microsoft? It sounds like radio silence to me still.

[Misaka-0x447f]

Emmmmm nope. It's one of the top feedback but opened 2 yrs ago.

[sjkp]

Fixed in 0.9.4

[mmartain]

I have the same problem. Does not work out of the box...