Using IIS ARR Proxy (instead of nginx/docker/vm) fails on pull with 502.3 "The server returned an invalid or unrecognized response"

[pbering]

Hi, I'm trying to use IIS ARR proxy for ACR (locally for now, the plan was to deploy on Azure App Service). I can login fine, do a 'GET http://localhost/v2' but docker pull fails and so does 'GET http://localhost/v2/sitecore/blobs/sha256:74999096977bb95e29d7368cb1b84070895c0e35f64515959a70b7b9db21688d'.

My web.config looks like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <system.webServer>
        <httpErrors errorMode="Detailed" />
        <rewrite>
            <rules>
                <rule name="Proxy" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="https://someregistry.azurecr.io/{R:1}" />
                    <serverVariables>
                        <set name="HTTP_ACCEPT_ENCODING" value="" />
                        <set name="HTTP_HOST" value="someregistry.azurecr.io" />
                        <set name="HTTP_X_REAL_IP" value="{REMOTE_ADDR}" />
                        <set name="HTTP_X_FORWARDED_FOR" value="{REMOTE_ADDR}" />
                        <set name="HTTP_X_FORWARDED_PROTO" value="https" />                        
                    </serverVariables>
                </rule>
            </rules>
            <allowedServerVariables>
                <add name="HTTP_HOST" />
                <add name="HTTP_ACCEPT_ENCODING" />
                <add name="HTTP_X_REAL_IP" />
                <add name="HTTP_X_FORWARDED_FOR" />
                <add name="HTTP_X_FORWARDED_PROTO" />
            </allowedServerVariables>
        </rewrite>
    </system.webServer>
</configuration>

Is this even possible to do with IIS ARR?

[pbering]

Here's a screenshot of the failed request tracing log:

[pan-wang]

@pbering ARR uses WinHttp APIs to communicate with the backend server. This error is returned by async callback from WinHttp. It seems WinHttp could not understand the response from the backend. Could you please provide an iDNA trace so that we can take a look at it? Thanks!

[pbering]

@pan-wang I'm not familiar with "iDNA" - found some old KB here https://support.microsoft.com/en-us/help/2716889/-sdp-3-3fc3cd2a-d297-4e37-a501-35cf7303e436-time-travel-tracing-diagnostic but with no download links. Can you point to some place where I can get the tool?

[pan-wang]

I reproduced the failure locally. Per the FREB log, it seems ACR returned 307 response with non-zero content-length header value but empty body. ACR team is working on the fix. Let's wait for this fix to see whether it solves the problem.

[yuwaMSFT2]

The ACR server is forwarding the 307 status code on blob GET request. However, there is a bug that the body of the response may not be forwarded to client.

Docker client seems to be more resilient since the body content is normally only be used by very old client. But IIS ARR module is more strict. We are working on a fix.

[yuwaMSFT2]

@pbering

I think the problem is the proxy configuration. You need to unselect the following option (which is selected by default I think):
Reverse rewrite host in response header

Our redirect is to the Storage SAS Url. With this being selected, IIS will override the Location header to point back to the Host (someregistry.azurecr.io). Our registry receives these calls and returns 404. So upload sees unknown blob.

Now after I unselected it, I could Pull/Push with ARR with no problem. Please try it and see if it can unblock you.

[pbering]

@yuwaMSFT2 I've try with that setting both on an off with the same results. Also tried to toggle the other settings. My config looks like so:

[yuwaMSFT2]

This is the configuration working for me:

This is the web.config
<?xml version="1.0" encoding="utf-8"?> <configuration> <system.webServer> <httpErrors errorMode="Detailed" /> <rewrite> <rules> <rule name="Proxy" stopProcessing="true"> <match url="(.*)" /> <action type="Rewrite" url="https://yuwatestcan1.azurecr.io/{R:1}" logRewrittenUrl="true" /> <serverVariables> <set name="HTTP_ACCEPT_ENCODING" value="" /> <set name="HTTP_HOST" value="yuwatestcan1.azurecr.io" /> <set name="HTTP_X_REAL_IP" value="{REMOTE_ADDR}" /> <set name="HTTP_X_FORWARDED_FOR" value="{REMOTE_ADDR}" /> <set name="HTTP_X_FORWARDED_PROTO" value="https" /> </serverVariables> </rule> </rules> <allowedServerVariables> <add name="HTTP_HOST" /> <add name="HTTP_ACCEPT_ENCODING" /> <add name="HTTP_X_REAL_IP" /> <add name="HTTP_X_FORWARDED_FOR" /> <add name="HTTP_X_FORWARDED_PROTO" /> </allowedServerVariables> </rewrite> </system.webServer> <location path="" overrideMode="Allow"> <appSettings /> </location> </configuration>
One of our engineer also tried it. After turning off the reverseHostOverride, he could pull/push.

What is the error you are getting? If you can share with me your registry name, I can also help checking our server logs.

[pbering]

@yuwaMSFT2 mmm still not working with the exact same proxy setting and your web.config. It seems that I have a few more setting than in your screenshot, for example "stealthMode", which IIS / OS are you trying on? I'm on IIS 10 (Windows 10 CU Enterprise) and have ARR 3.0 x64 installed (also tryied to reinstall it). My ACR is running on chreg-pmbcodehouse392.azurecr.io.

The error from the docker client is:

PS C:>docker pull chreg-proxy.ad.codehouse.com/sitecore:8.2.170407.0
8.2.170407.0: Pulling from sitecore
error pulling image configuration: received unexpected HTTP status: 502 Bad Gateway

The error from IIS and FRT is in the beginning of this issue.

[yuwaMSFT2]

@pbering Sorry you are right.

I could pull successfully because I already had the image on my machine. So docker pull only pulls the manifest but not the layer image.

If there is no layer image already on the local client, docker client will send GET request; our server will return 307 to redirect to Azure storage...But due to a bug I mentioned earlier, we may send positive content-lenght but not body. IIS is strict on that and will fail.

Our fix is checked in...but you may need to wait for the server to be patched to make your scenario working.

[pbering]

Glad that the issue is confirmed and fixed! I would prefer to keep this issues open so others that try the same, can see that a fix is on its way. Could you please update this issue when fix in "live"?

[yuwaMSFT2]

Sure. Will update here once the fix is applied to the service in all regions.

[yuwaMSFT2]

@pbering We have updated our service. Could you please try again and let us know? Thanks!

[pbering]

Verified from both local IIS 10 and IIS on Azure App Service as well, thank you :)

[minusdavid]

This looks like an old issue but it really helped me out a bit. In my case, I'm using an IIS ARR Proxy, but I had to explicitly remove the ":" character from the requestPathInvalidCharacters attribute in the httpRuntime.

IIS 8 would proxy the manifest but not the blobs and that was why.

Anyway, just leaving that here in the event that someone else with the same problem comes across this particular Github issue just like me.

I added the following under the element in my web.config:

<system.web> <!-- <customErrors mode="Off"/> --> <httpRuntime requestPathInvalidCharacters="&lt;,&gt;,%,&amp;,\,?" /> </system.web>

(I used the customErrors element to figure out what the problem was.)