Implement global proxy configuration for HttpClient

[davidsh]

[edit: this top post has been edited to show the final draft proposal for a new API]

API Proposal

namespace System.Net.Http
{
    public class HttpClient
    {
        ...
        public static IWebProxy DefaultProxy { get; set; } 
        ...
    }
}

Behaviors

The property will default to a non-null value representing the platform/system proxy. With PR 37328, all platforms can use environment variables as the first possible choice of proxy settings. If those variables are set, then an IWebProxy interface of an instance of the internal HttpEnvironmentProxy class is returned.

If the environment variables are not set, then the following happens for the default value of this property:

• On Windows, the IWebProxy interface of the internal HttpSystemProxy class is returned. HttpSystemProxy wraps the IE/Wininet settings.
• On OSX, the IWebProxy interface of the internal MacProxy class is returned. MacProxy wraps the OSX system proxy configuration.
• On Linux, if environment variables are not set, then a non-null instance of an IWebProxy object is returned. It will return 'true' for IsBypassed() for any Uri passed in.

The property will never return null.

The property can be set to any object implementing the IWebProxy interface. This property cannot be set to 'null'. Doing so will throw an exception, ArgumentNullException.

Sample API usage

// I want to pass in default credentials to the system proxy so my any HttpClient request
// will work on my corporate network.
// This includes any HttpClient objects that I might not create as long as they use the
// default settings of HttpClientHandler.
HttpClient.DefaultProxy.Credentials = CredentialCache.DefaultCredentials;

// I want to use a special proxy everywhere I use HttpClient
HttpClient.DefaultProxy = new MyCustomProxy();

History of proxy support in .NET

https://gist.github.com/davidsh/f8768c02faf714de9a3029cbf0166f18

Background

HttpClient has support for reading system proxy configuration (IE settings on Windows, Environment variables on Linux, etc). However, there is no mechanism for setting credentials (i.e. CredentialCache.DefaultCredentials) for a system configured proxy. Currently, this requires creating an HttpClientHandler instance and setting its DefaultProxyCredentials property.

For cases where libraries are creating the HttpClient object, there is no way for consumers of the library to set these credentials easily.

The .NET Framework supports using app.config/web.config settings with a system.net section:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.net>
    <defaultProxy useDefaultCredentials="true" />
  </system.net>
</configuration>

This allows applications to inject a global proxy configuration for any HttpClient that is created. However, .NET Core doesn't natively support configuration files.

.NET Framework currently has the WebRequest.DefaultWebProxy property to control this for all HttpWebRequest objects (and affects HttpClient) as well. On .NET Core, the WebRequest class is considered legacy. But we need a similar way to set global proxy config on .NET Core for HttpClient usage.

A static API on HttpClient (or other object) created for global proxy configuration for the application could be used by upstack components to create a similar configuration file (i.e. app.config) behavior.

[davidsh]

The next step for implementing this will be a new API proposal issue to be opened soon.

[lobster2012-user]

Don't forget about UseDefaultCredentials .
This parameter is important for the intranet and by default it has value false.

[tebeco]

@davish thx a lot for this

in a previous issue a proposed solution was something LIKE this (the issue with this one is that it's aspnetcore specific)

https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.http.features.ifeaturecollection?view=aspnetcore-2.1

Another idea could be something like this
AppContext.SetSwitch(switchName, state)
but it can be hard to have "logic" in it

Another way (that i find more interresting as a consumer) would be an Api that kinda works like powershell :
Get-Proxy destinationUrl
allowing developper to do a "Last Look" on request and return for example :

• the url of the proxy to use
• an httphandler to add to that call

it could be on theHttpClientFactory.HERE(Uri destination) itself but i have no idea if the creation of an HttpClient in a third part nuget already "interact" with HttpClientFactory

the goodness of the last one is that we could inject a default callback using WindowsCompatPack, check if windows, hit the Registry, retrieve proxy settings etc ...
that would allow :

• old behavior for global settings
• default credential if we get the hand on the handler
• dynamism (because we have multiple proxy in LAN)
• no huge impact if the value are "re-used" instead of "per call"

[karelz]

@tebeco HttpClientFactory is higher-layer library. It does not make sense to make HttpClient complex and make it depend on it - beside complexity, which is hard to explain, it would bring also layering violation which is often sign of bad design.
We should stay in HttpClient and solve the problem here, then let HttpClientFactory / ASP.NET Core use the lower-layer mechanism here in HttpClient.

Rather than shooting for full setting compatibility with .NET Framework, we should IMO focus on extensibility of the mechanisms in HttpClient, so that users and/or libraries can provide additional logic. Then anyone can write smart Registry-reading / config file-reading functionality ...

[epignosisx]

I would love to see this happening. We have a corporate proxy that causes a lot of pain for us because of the reasons that @davidsh explained.

What if instead of adding a new API to the already complicated HttpClientHandler, we make the windows implementation aware of the HTTP_PROXY environment variable and friends like the Linux version.

Node.js, Rust, and Go all honor these environment variables in Windows, probably most developers running behind a proxy already have these environment variables defined. Another thing that bit us in the past is that our test environments were inside the company network, however the production environment was in a DMZ with no proxy. We learned the difference only after going to prod and finding out that all our calls were failing. In many companies these environment intricacies are only known to the networking or sysadmin teams. Environment variables allow sysadmins to configure the servers making it transparent to the applications running in them.

[karelz]

@epignosisx enc vars are on the table AFAIK.
We should add new API anyway, to enable config-style settings by host (e.g. ASP.NET). Someone can then even implement a simple library to read .NET Framework-style config for full compat (it should not be in the platform itself as .NET Core does not have config concept at the moment).

[epignosisx]

@karelz thanks for the feedback, makes sense. Since adding a new API will have to go through the review process, I’m afraid it will miss the 3.0 train. If that’s the case, could the team work first on honoring the env vars? I’m assuming this will be simpler to implement and can make it for 3.0.

[karelz]

It is not going to miss 3.0, unless something very unexpected happens, don't worry.
3.0 date is not announced yet - it will be announced at Build conference in May. If you look at number of remaining 3.0 bugs (eg via my tweets on progress), you can do some guesswork as well.

[davidsh]

@epignosisx

What if instead of adding a new API to the already complicated HttpClientHandler, we make the windows implementation aware of the HTTP_PROXY environment variable and friends like the Linux version.

I just implemented environment variables support for Windows to match Linux/OSX with PR dotnet/corefx#37238.

We still will be adding a new API.