Unable to connect to HTTPS endpoint through WebProxy

[pmadusud]

Unable to connect to HTTPS endpoint through WebProxy

I am unable to connect to an https endpoint from behind the enterprise firewall. I get the below error.
"Only the 'http' scheme is allowed for proxies"

General

I have created a Azure Function Core App which uses .NET Core 2.1 and functions runtime v2. The code tries to get data from a https endpoint. During debugging I am not able to connect to the https endpoint due to the enterprise proxy issue.
To resolve that, I am using the WebProxy to configure the enterprise firewall and adding the same to the httpclient before connecting to the endpoint.

However, I get the error "Only the 'http' scheme is allowed for proxies".

The code sample is given below for reference.

var proxy = new WebProxy()
{
                Address = new Uri($""), //proxy address
                UseDefaultCredentials = false,
                Credentials = System.Net.CredentialCache.DefaultNetworkCredentials
};

var httpClientHandler = new HttpClientHandler()
{
                UseProxy = true,
                PreAuthenticate = true,
                Proxy = proxy,
 };

 HttpClient client = new HttpClient(httpClientHandler);
 client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));

HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, Uri); //Send request to endpoint.
                
HttpResponseMessage response = await client.SendAsync(request);

Any pointers on how to resolve this issue will be highly appreciated.

[sirblenheim]

What URL are you using for the proxy address? The error seems to indicate that you are trying to use a proxy over the HTTPs protocol.

[karelz]

HTTPS proxies are not supported on .NET. The support in the industry is not wide-spread either (it just started appear in last couple of years).

However, it is NOT the same thing as connecting to HTTPS endpoint via HTTP proxy (which is IMO the primary problem you want to solve). That is totally valid scenario, fully supported and should be working.
Can you please double-check that you can use HTTP proxy URL, then use it even for HTTPS traffic?

[pmadusud]

      What URL are you using for the proxy address? The error seems to indicate that you are trying to use a proxy over the HTTPs protocol.

The URL provided for the proxy address is the proxy server address along with port. For example. :port. As you can see above, I am not trying to connect to proxy server over https portal.

[pmadusud]

Karelz, as mentioned in my above comment, I am not using a https Proxy. The proxy address is just the server name & port. Using this proxy server, I am connecting to a https end point. Please let me know if I am missing something here. Also, let me know if you would need any more information.

[karelz]

That is a bit suspicious. Can you please double check that? Because the error you're quoting is reported only if you use HTTPS URL for the proxy itself:
https://github.com/dotnet/corefx/blob/a10890f4ffe0fadf090c922578ba0e606ebdd16c/src/System.Net.Http/src/Resources/Strings.resx#L375-L377
https://github.com/dotnet/corefx/blob/449190d7c873b56337ba6a7f2d9b53f5232ad16d/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs#L294-L297

Things to check on top of that:

• Does the same code work on the same machine from .NET Framework?
• Does it fail locally or only in the cloud as Azure function?
• Which OS do you use? Did you check if it is specific failure to the OS?

[pmadusud]

Responses to your questions below.
Does the same code work on the same machine from .NET Framework? - Yes with .NET Framework it works. In fact i don't need to use WebProxy itself as I can use the Defaultproxy in the app.config to ensure the application works with the local enterprise proxy.
Does it fail locally or only in the cloud as Azure function? This fails only locally because the issue is with the local enterprise proxy.
Which OS do you use? Did you check if it is specific failure to the OS? I use Windows 7 and I am not sure if this is a OS specific issue.

I also checked the URI which i am sending to the WebProxy.
If the URI is provided like the below
Address = new Uri($"servername:port")
then i get the error "Only the 'http' scheme is allowed for proxies".
If the URI is provided like the below
Address = new Uri($"http://servername/proxy.pac")
then i get the error System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.

Any pointers to resolve this will be great.

[karelz]

I don't think we support PAC files in URL for proxy. What is the inner exception in the second case?
Can you grab the exception in the first case? What is the callstack? Can you try to debug into CoreFX itself?

cc @davidsh

[davidsh]

I also checked the URI which i am sending to the WebProxy.
If the URI is provided like the below
Address = new Uri($"servername:port")
then i get the error "Only the 'http' scheme is allowed for proxies".

It is invalid to specify a proxy using just the "servername:port" format. That is not a valid URI since it doesn't have a scheme. You must specify it as "http://servername:port"

then i get the error "Only the 'http' scheme is allowed for proxies".

The error that you're getting is an incorrect error message, unfortunately. That is probably a different bug of sorts.

I would suggest that you retry this scenario using the latest .NET 2.1.6 release which contains all known fixes for proxies and enterprise authentication scenarios.
https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1.6/2.1.6-download.md

[davidsh]

HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, Uri); //Send request to endpoint.

Also, can you try an experiment? If you change this from 'POST' to 'GET' does it work?

[pmadusud]

I also checked the URI which i am sending to the WebProxy.
If the URI is provided like the below
Address = new Uri($"servername:port")
then i get the error "Only the 'http' scheme is allowed for proxies".

It is invalid to specify a proxy using just the "servername:port" format. That is not a valid URI since it doesn't have a scheme. You must specify it as "http://servername:port"

then i get the error "Only the 'http' scheme is allowed for proxies".

The error that you're getting is an incorrect error message, unfortunately. That is probably a different bug of sorts.

I would suggest that you retry this scenario using the latest .NET 2.1.6 release which contains all known fixes for proxies and enterprise authentication scenarios.
https://github.com/dotnet/core/blob/master/release-notes/2.1/2.1.6/2.1.6-download.md

By changing the Uri as http://servername:port, i am able to get the proper response. Thanks for the tip.

I would like to know if there is some way of having an equivalent of in .NET Framework in .NET Core. This is to have a better solution to deal with Local Enterprise proxy which will not be a scenario when it is deployed in Azure Functions. Any inputs on this?

[davidsh]

I would like to know if there is some way of having an equivalent of in .NET Framework in .NET Core.

What "equivalent" are you referring to? I don't understand.

[pmadusud]

oops.. sorry... setting in App.config which is available in ,NET Framework. Is there an equivalent of that in .NET Core.

[davidsh]

oops.. sorry... setting in App.config which is available in ,NET Framework. Is there an equivalent of that in .NET Core.

Currently there are no default config files (app.config/web.config) for .NET Core apps that are similar to .NET Framework. So, it will not be possible to map this concept directly.

There is generalized support for configuration such as that provided by ASP.NET for example:
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/index?view=aspnetcore-2.1

There are also configuration patterns using 'appsettings.json' as well:
https://adamrussell.com/2018/11/05/appsettings-json-in-a-net-core-console-application/

But those configuration mechanisms are not something that CoreFx itself could use.

We are considering adding environment variable support on Windows .NET Core for static proxies similar to 'http_proxy' etc. that is available in the Linux implementation of .NET Core. See issues such as https://github.com/dotnet/corefx/issues/29934.

[pmadusud]

@davidsh thanks for this. I did try to use http.proxy setting in Visual Studio Code. However, .NET Core doesn't seem to use that setting. I guess that is the one which you referred in the above note about Windows .NET core doesn't support environment variable like http_proxy. Please correct me if I am wrong.

[davidsh]

Windows .NET core doesn't support environment variable like http_proxy. Please correct me if I am wrong.

Correct. Windows .NET Core doesn't support environment variables. We are considering adding that feature in a future release.

However, Windows. NET Core supports the standard IE settings for proxies including automatic discovery of PAC files, explicit PAC files or explicit proxy server settings. This is the recommended thing to use for enterprise scenarios where proxies are used on the network.

[pmadusud]

@davidsh thanks. However, it doesn't seem to use the standard IE setting. I get a 407 proxy authentication error. I am using the latest release of Visual Studio Code for development.

[davidsh]

I get a 407 proxy authentication error.

How are you passing in credentials to the proxy? If you are using the "default system" proxy, then the HTTP stack will read the proxy settings from IE settings. But you are required to tell HttpClient to use some credentials to pass to the proxy. In the case of the "default system" proxy, you need to write code like this:

var handler = new HttpClientHandler();
handler.DefaultProxyCredentials = System.Net.CredentialCache.DefaultNetworkCredentials;
var client = new HttpClient(handler);

// ...

I'm assuming in the example above that your enterprise networking topology uses an authenticating proxy and that the proxy use Windows auth schemes (such as Negotiate or NTLM). Using the code above will use the default credentials of the logged in user and pass those to the authenticating proxy. Thus, it should prevent getting the 407 from the proxy.

[pmadusud]

I have tried both the options below.
Option 1

var httpClientHandler = new HttpClientHandler()
{
                DefaultProxyCredentials = CredentialCache.DefaultNetworkCredentials
};

HttpClient client = new HttpClient(httpClientHandler);

Option 2
HttpClient client = new HttpClient();

Both the options either returns the below error or in some cases 407 Proxy Authentication error.
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

I am able to resolve the above error only by explicitly providing the proxy like the below.

var proxy = new WebProxy()
{
                Address = new Uri($"http://{servername}:{port}"),
                UseDefaultCredentials = false,
                BypassProxyOnLocal = true,

                Credentials = System.Net.CredentialCache.DefaultNetworkCredentials
};

var httpClientHandler = new HttpClientHandler()
{
                Proxy = proxy,
                DefaultProxyCredentials = CredentialCache.DefaultNetworkCredentials
};

HttpClient client = new HttpClient(httpClientHandler);

As mentioned, the standard IE setting for Proxy doesn't seem to be used and also the http.proxy user setting in Visual Studio Code.

[davidsh]

Option 2
HttpClient client = new HttpClient();

This option will never work because you are not specifying credentials for the proxy at all. So, if the default system proxy requires credentials, you will always get a 407.

Let's start with collecting some information.

• What version of Windows OS are you running? Please open up a CMD.EXE window and type "ver".
• What version of .NET Core are using? Type "dotnet --info" from a CMD.EXE window.
• Please show a screenshot of your IE settings dialog for proxy settings.

Can you attach a Wireshark trace of the failing scenario?

What happens if you switch to the older HTTP stack implementation? Add the following line of code to your application:

AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);

[pmadusud]

Responses for the questions...

What version of Windows OS are you running? Please open up a CMD.EXE window and type "ver".
Microsoft Windows [Version 6.1.7601]
What version of .NET Core are using? Type "dotnet --info" from a CMD.EXE window.
.NET Core SDK (reflecting any global.json):
Version: 2.1.403
Commit: 04e15494b6
Runtime Environment:
OS Name: Windows
OS Version: 6.1.7601
OS Platform: Windows
RID: win7-x64
Base Path: C:\Program Files\dotnet\sdk\2.1.403
Host (useful for support):
Version: 2.1.5
Commit: 290303f510
.NET Core SDKs installed:
2.1.403 [C:\Program Files\dotnet\sdk]
Please show a screenshot of your IE settings dialog for proxy settings.

What happens if you switch to the older HTTP stack implementation? Add the following line of code to your application:

AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);

There is no difference by adding the above line of code. I still get the same error.

[davidsh]

Thank you for the information.

What happens if you change the IE settings so that instead of specifying "Use automatic configuration script", you instead type the name of the server and the port number into the "Use a proxy server for your LAN" section. Then uncheck the "Use automatic configuration script" and check the "Use a proxy server for LAN" checkbox.

In all cases, make sure you use the following pattern for setting up HttpClientHandler, HttpClient:

var httpClientHandler = new HttpClientHandler()
{
    DefaultProxyCredentials = CredentialCache.DefaultNetworkCredentials
};

HttpClient client = new HttpClient(httpClientHandler);

[davidsh]

Also, are you able to do a Wireshark trace as well?

[pmadusud]

Thank you for the information.

What happens if you change the IE settings so that instead of specifying "Use automatic configuration script", you instead type the name of the server and the port number into the "Use a proxy server for your LAN" section. Then uncheck the "Use automatic configuration script" and check the "Use a proxy server for LAN" checkbox.

In all cases, make sure you use the following pattern for setting up HttpClientHandler, HttpClient:

var httpClientHandler = new HttpClientHandler()
{
    DefaultProxyCredentials = CredentialCache.DefaultNetworkCredentials
};

HttpClient client = new HttpClient(httpClientHandler);

I still get the same issue if I explicitly provide the proxy address in LAN Settings. Please note, I am not able to remove the configuration script setting as it is controlled by corporate admin.

[davidsh]

Can you provide Wireshark traces?

Also, can you provide PerfView traces as well? See:
https://github.com/dotnet/corefx/blob/master/Documentation/debugging/windows-instructions.md#using-perfview

[davidsh]

@pmadusud Were you able to take additional traces? Otherwise, we will unable to diagnose further. Thank you.

[karelz]

@pmadusud any update?

[pmadusud]

@davidsh and @karelz, sorry I have been away and I have not yet managed to get any logs yet. Also, the current IT environment doesn't allow me to install software, I will check if I can provide Wireshark traces. However, if not, is there any other way we can resolve this?

[karelz]

The only other way would be to debug through the code on your machine ... that is, however, rather involved.

[eltiare]

I'm getting this error when trying to connect to Crawlera. Their proxies use HTTPS. :(

[karelz]

Looks like not actionable with current information, closing.
Let us know if there is more info and we can reopen.

[arun-subramanian-1]

With .net core 6.0 I am getting same error. HTTPS pages with HTTP proxy works, But not with HTTPS proxy.
From the exception it appears to be thrown probably from here.
https://github.com/dotnet/runtime/blob/main/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPoolManager.cs#L409

IsSupportedProxyScheme checks for socks & http. so does this mean .net core doesnt support https proxy servers ?

[karelz]

HTTPS proxy is not supported in .NET / .NET Core yet: dotnet/runtime#31113