-
Notifications
You must be signed in to change notification settings - Fork 109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using IIS ARR Proxy (instead of nginx/docker/vm) fails on pull with 502.3 "The server returned an invalid or unrecognized response" #45
Comments
@pbering ARR uses WinHttp APIs to communicate with the backend server. This error is returned by async callback from WinHttp. It seems WinHttp could not understand the response from the backend. Could you please provide an iDNA trace so that we can take a look at it? Thanks! |
@pan-wang I'm not familiar with "iDNA" - found some old KB here https://support.microsoft.com/en-us/help/2716889/-sdp-3-3fc3cd2a-d297-4e37-a501-35cf7303e436-time-travel-tracing-diagnostic but with no download links. Can you point to some place where I can get the tool? |
I reproduced the failure locally. Per the FREB log, it seems ACR returned 307 response with non-zero content-length header value but empty body. ACR team is working on the fix. Let's wait for this fix to see whether it solves the problem. |
The ACR server is forwarding the 307 status code on blob GET request. However, there is a bug that the body of the response may not be forwarded to client. Docker client seems to be more resilient since the body content is normally only be used by very old client. But IIS ARR module is more strict. We are working on a fix. |
I think the problem is the proxy configuration. You need to unselect the following option (which is selected by default I think): Our redirect is to the Storage SAS Url. With this being selected, IIS will override the Location header to point back to the Host (someregistry.azurecr.io). Our registry receives these calls and returns 404. So upload sees unknown blob. Now after I unselected it, I could Pull/Push with ARR with no problem. Please try it and see if it can unblock you. |
@yuwaMSFT2 I've try with that setting both on an off with the same results. Also tried to toggle the other settings. My config looks like so: |
@yuwaMSFT2 mmm still not working with the exact same proxy setting and your web.config. It seems that I have a few more setting than in your screenshot, for example "stealthMode", which IIS / OS are you trying on? I'm on IIS 10 (Windows 10 CU Enterprise) and have ARR 3.0 x64 installed (also tryied to reinstall it). My ACR is running on chreg-pmbcodehouse392.azurecr.io. The error from the docker client is:
The error from IIS and FRT is in the beginning of this issue. |
@pbering Sorry you are right. I could pull successfully because I already had the image on my machine. So docker pull only pulls the manifest but not the layer image. If there is no layer image already on the local client, docker client will send GET request; our server will return 307 to redirect to Azure storage...But due to a bug I mentioned earlier, we may send positive content-lenght but not body. IIS is strict on that and will fail. Our fix is checked in...but you may need to wait for the server to be patched to make your scenario working. |
Glad that the issue is confirmed and fixed! I would prefer to keep this issues open so others that try the same, can see that a fix is on its way. Could you please update this issue when fix in "live"? |
Sure. Will update here once the fix is applied to the service in all regions. |
@pbering We have updated our service. Could you please try again and let us know? Thanks! |
Verified from both local IIS 10 and IIS on Azure App Service as well, thank you :) |
This looks like an old issue but it really helped me out a bit. In my case, I'm using an IIS ARR Proxy, but I had to explicitly remove the ":" character from the requestPathInvalidCharacters attribute in the httpRuntime. IIS 8 would proxy the manifest but not the blobs and that was why. Anyway, just leaving that here in the event that someone else with the same problem comes across this particular Github issue just like me. I added the following under the element in my web.config:
(I used the customErrors element to figure out what the problem was.) |
Hi, I'm trying to use IIS ARR proxy for ACR (locally for now, the plan was to deploy on Azure App Service). I can login fine, do a 'GET http://localhost/v2' but docker pull fails and so does 'GET http://localhost/v2/sitecore/blobs/sha256:74999096977bb95e29d7368cb1b84070895c0e35f64515959a70b7b9db21688d'.
My web.config looks like this:
Is this even possible to do with IIS ARR?
The text was updated successfully, but these errors were encountered: