New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement global proxy configuration for HttpClient #29147
Comments
The next step for implementing this will be a new API proposal issue to be opened soon. |
Don't forget about UseDefaultCredentials . |
@davish thx a lot for this in a previous issue a proposed solution was something LIKE this (the issue with this one is that it's aspnetcore specific) Another idea could be something like this Another way (that i find more interresting as a consumer) would be an Api that kinda works like powershell :
it could be on the the goodness of the last one is that we could inject a default callback using
|
@tebeco HttpClientFactory is higher-layer library. It does not make sense to make HttpClient complex and make it depend on it - beside complexity, which is hard to explain, it would bring also layering violation which is often sign of bad design. Rather than shooting for full setting compatibility with .NET Framework, we should IMO focus on extensibility of the mechanisms in HttpClient, so that users and/or libraries can provide additional logic. Then anyone can write smart Registry-reading / config file-reading functionality ... |
I would love to see this happening. We have a corporate proxy that causes a lot of pain for us because of the reasons that @davidsh explained. What if instead of adding a new API to the already complicated Node.js, Rust, and Go all honor these environment variables in Windows, probably most developers running behind a proxy already have these environment variables defined. Another thing that bit us in the past is that our test environments were inside the company network, however the production environment was in a DMZ with no proxy. We learned the difference only after going to prod and finding out that all our calls were failing. In many companies these environment intricacies are only known to the networking or sysadmin teams. Environment variables allow sysadmins to configure the servers making it transparent to the applications running in them. |
@epignosisx enc vars are on the table AFAIK. |
@karelz thanks for the feedback, makes sense. Since adding a new API will have to go through the review process, I’m afraid it will miss the 3.0 train. If that’s the case, could the team work first on honoring the env vars? I’m assuming this will be simpler to implement and can make it for 3.0. |
It is not going to miss 3.0, unless something very unexpected happens, don't worry. |
SocketsHttpHandler was not using the proxy bypass list specified in IE settings in cases where there is a combination of both 'AutoDetect' and 'Manual' settings in the IE settings dialog. SocketsHttpHandler uses WinHttpHandler's WinInetProxyHelper class as part of the HttpSystemProxy class. But it was consuming it in an incorrect way. WinInetProxyHelper was originally written to be used only with the rest of the WinHTTP stack for WinHttpHandler. When WinHTTP GetProxyForUrl() was returning a ProxyBypass string, HttpSystemProxy was ignoring it. It was assuming that the string for Proxy was correct. But in cases where ProxyBypass is returned, the Proxy string is only used if the destination uri doesn't match any of the strings in the ProxyBypass list. That logic would normally be handled automatically by WinHttpHandler. But HttpSystemProxy was simply discarding the ProxyBypass string returned by WinHTTP GetProxyForUrl(). In order to address this fix, I added new tests for HttpSystemProxy. I utilized the existing mock layers for registry and WinHTTP interop that are contained in the WinHttpHandler Unit Tests. It might seem weird to have tests for the internal HttpSystemProxy class located in the WinHttpHandler folder. But, we already pull in the source code for WinInetProxyHelper from WinHttpHandler into the SocketsHttpHandler compilation. Using these Unit Tests with the mocking layers allows us to test a broad range of scenarios (registry settings and network failures) that are normally difficult to test. I have a plan, though, to refactor the system proxy code and tests as part of implementing #36553. That will result in the system proxy code and tests ending up in a more logical place. As part of this fix, I also optimized how SocketsHttpHandler is calling IWebProxy. I explained this in the comments of HttpSystemProxy.IsBypassed(). In summary, IWebProxy.IsBypassed() shouldn't be used. In most cases it is the same amount of work than calling IWebProxy.GetProxy(). And the latter can be used to return a valid proxy uri, or null if a proxy shouldn't be used for that particular destination uri. Fixes #33866
SocketsHttpHandler was not using the proxy bypass list specified in IE settings in cases where there is a combination of both 'AutoDetect' and 'Manual' settings in the IE settings dialog. SocketsHttpHandler uses WinHttpHandler's WinInetProxyHelper class as part of the HttpSystemProxy class. But it was consuming it in an incorrect way. WinInetProxyHelper was originally written to be used only with the rest of the WinHTTP stack for WinHttpHandler. When WinHTTP GetProxyForUrl() was returning a ProxyBypass string, HttpSystemProxy was ignoring it. It was assuming that the string for Proxy was correct. But in cases where ProxyBypass is returned, the Proxy string is only used if the destination uri doesn't match any of the strings in the ProxyBypass list. That logic would normally be handled automatically by WinHttpHandler. But HttpSystemProxy was simply discarding the ProxyBypass string returned by WinHTTP GetProxyForUrl(). In order to address this fix, I added new tests for HttpSystemProxy. I utilized the existing mock layers for registry and WinHTTP interop that are contained in the WinHttpHandler Unit Tests. It might seem weird to have tests for the internal HttpSystemProxy class located in the WinHttpHandler folder. But, we already pull in the source code for WinInetProxyHelper from WinHttpHandler into the SocketsHttpHandler compilation. Using these Unit Tests with the mocking layers allows us to test a broad range of scenarios (registry settings and network failures) that are normally difficult to test. I have a plan, though, to refactor the system proxy code and tests as part of implementing #36553. That will result in the system proxy code and tests ending up in a more logical place. As part of this fix, I also optimized how SocketsHttpHandler is calling IWebProxy. I explained this in the comments of HttpSystemProxy.IsBypassed(). In summary, IWebProxy.IsBypassed() shouldn't be used. In most cases it is the same amount of work than calling IWebProxy.GetProxy(). And the latter can be used to return a valid proxy uri, or null if a proxy shouldn't be used for that particular destination uri. Fixes #33866
SocketsHttpHandler was not using the proxy bypass list specified in IE settings in cases where there is a combination of both 'AutoDetect' and 'Manual' settings in the IE settings dialog. SocketsHttpHandler uses WinHttpHandler's WinInetProxyHelper class as part of the HttpSystemProxy class. But it was consuming it in an incorrect way. WinInetProxyHelper was originally written to be used only with the rest of the WinHTTP stack for WinHttpHandler. When WinHTTP GetProxyForUrl() was returning a ProxyBypass string, HttpSystemProxy was ignoring it. It was assuming that the string for Proxy was correct. But in cases where ProxyBypass is returned, the Proxy string is only used if the destination uri doesn't match any of the strings in the ProxyBypass list. That logic would normally be handled automatically by WinHttpHandler. But HttpSystemProxy was simply discarding the ProxyBypass string returned by WinHTTP GetProxyForUrl(). In order to address this fix, I added new tests for HttpSystemProxy. I utilized the existing mock layers for registry and WinHTTP interop that are contained in the WinHttpHandler Unit Tests. It might seem weird to have tests for the internal HttpSystemProxy class located in the WinHttpHandler folder. But, we already pull in the source code for WinInetProxyHelper from WinHttpHandler into the SocketsHttpHandler compilation. Using these Unit Tests with the mocking layers allows us to test a broad range of scenarios (registry settings and network failures) that are normally difficult to test. I have a plan, though, to refactor the system proxy code and tests as part of implementing #36553. That will result in the system proxy code and tests ending up in a more logical place. As part of this fix, I also optimized how SocketsHttpHandler is calling IWebProxy. I explained this in the comments of HttpSystemProxy.IsBypassed(). In summary, IWebProxy.IsBypassed() shouldn't be used. In most cases it is the same amount of work than calling IWebProxy.GetProxy(). And the latter can be used to return a valid proxy uri, or null if a proxy shouldn't be used for that particular destination uri. Fixes #33866
SocketsHttpHandler was not using the proxy bypass list specified in IE settings in cases where there is a combination of both 'AutoDetect' and 'Manual' settings in the IE settings dialog. SocketsHttpHandler uses WinHttpHandler's WinInetProxyHelper class as part of the HttpSystemProxy class. But it was consuming it in an incorrect way. WinInetProxyHelper was originally written to be used only with the rest of the WinHTTP stack for WinHttpHandler. When WinHTTP GetProxyForUrl() was returning a ProxyBypass string, HttpSystemProxy was ignoring it. It was assuming that the string for Proxy was correct. But in cases where ProxyBypass is returned, the Proxy string is only used if the destination uri doesn't match any of the strings in the ProxyBypass list. That logic would normally be handled automatically by WinHttpHandler. But HttpSystemProxy was simply discarding the ProxyBypass string returned by WinHTTP GetProxyForUrl(). In order to address this fix, I added new tests for HttpSystemProxy. I utilized the existing mock layers for registry and WinHTTP interop that are contained in the WinHttpHandler Unit Tests. It might seem weird to have tests for the internal HttpSystemProxy class located in the WinHttpHandler folder. But, we already pull in the source code for WinInetProxyHelper from WinHttpHandler into the SocketsHttpHandler compilation. Using these Unit Tests with the mocking layers allows us to test a broad range of scenarios (registry settings and network failures) that are normally difficult to test. I have a plan, though, to refactor the system proxy code and tests as part of implementing #36553. That will result in the system proxy code and tests ending up in a more logical place. Fixes #33866
SocketsHttpHandler was not using the proxy bypass list specified in IE settings in cases where there is a combination of both 'AutoDetect' and 'Manual' settings in the IE settings dialog. SocketsHttpHandler uses WinHttpHandler's WinInetProxyHelper class as part of the HttpSystemProxy class. But it was consuming it in an incorrect way. WinInetProxyHelper was originally written to be used only with the rest of the WinHTTP stack for WinHttpHandler. When WinHTTP GetProxyForUrl() was returning a ProxyBypass string, HttpSystemProxy was ignoring it. It was assuming that the string for Proxy was correct. But in cases where ProxyBypass is returned, the Proxy string is only used if the destination uri doesn't match any of the strings in the ProxyBypass list. That logic would normally be handled automatically by WinHttpHandler. But HttpSystemProxy was simply discarding the ProxyBypass string returned by WinHTTP GetProxyForUrl(). In order to address this fix, I added new tests for HttpSystemProxy. I utilized the existing mock layers for registry and WinHTTP interop that are contained in the WinHttpHandler Unit Tests. It might seem weird to have tests for the internal HttpSystemProxy class located in the WinHttpHandler folder. But, we already pull in the source code for WinInetProxyHelper from WinHttpHandler into the SocketsHttpHandler compilation. Using these Unit Tests with the mocking layers allows us to test a broad range of scenarios (registry settings and network failures) that are normally difficult to test. I have a plan, though, to refactor the system proxy code and tests as part of implementing #36553. That will result in the system proxy code and tests ending up in a more logical place. Fixes #33866
I just implemented environment variables support for Windows to match Linux/OSX with PR dotnet/corefx#37238. We still will be adding a new API. |
cc: @karelz @stephentoub @dotnet/ncl I have updated the top post of this issue and will be using it as the API proposal issue. |
if you got a nightly or something you want me to test at work (except between 04/05 and 11/05), i will gladly test it
|
In order to provide a consistent experience with the HttpClient IWebProxy objects, the returned internal proxy object which represent the system/platform proxy settings should never be null. If the platform's settings indicate that no proxy is being used, then return an instance of the internal HttpNoProxy object. Note that even if the platform settings indicate that a proxy could be used, any particular Http request might still not go thru a proxy. The final determination of what proxy is being used for a request is still governed by the return of the IWebProxy.IsBypassed and IWebProxy.GetProxy methods. Contributes to #36553
In order to provide a consistent experience with the HttpClient IWebProxy objects, the returned internal proxy object which represent the system/platform proxy settings should never be null. If the platform's settings indicate that no proxy is being used, then return an instance of the internal HttpNoProxy object. Note that even if the platform settings indicate that a proxy could be used, any particular Http request might still not go thru a proxy. The final determination of what proxy is being used for a request is still governed by the return of the IWebProxy.IsBypassed and IWebProxy.GetProxy methods. Contributes to #36553
This PR implements the new static HttpClient.DefaultProxy property which was approved during API review. Modify the SystemProxyInfo.ConstructSystemProxy method to a Singleton. Modify SocketsHttpHandler to use the HttpClient.DefaultProxy property. Rename the HttpSystemProxy class to HttpWindowsProxy. Add some HttpClient tests for the new property. Closes #36553
This PR implements the new static HttpClient.DefaultProxy property which was approved during API review. Modify the SystemProxyInfo.ConstructSystemProxy method to a Singleton. Modify SocketsHttpHandler to use the HttpClient.DefaultProxy property. Rename the HttpSystemProxy class to HttpWindowsProxy. Add some HttpClient tests for the new property. Closes #36553
@davidsh SignalR creates websocket in order to connect to Azure SignalR also as CosmosDb/KeyVault and other Api manually creates their own httpclient manually with custom handler |
If they are creating a .NET Core websocket via System.Net.WebSockets.Client, then they will benefit from it. The implementation of .NET Core websocket uses the common HTTP stack on .NET Core (which uses SocketsHttpHandler) to start the websocket. SocketsHttpHandler will query HttpClient.DefaultProxy.
If those components are using HttpClient, then they will benefit from the API indirectly. But if they have custom proxy requirements, they should make sure to set that property (or HttpClient.DefaultProxy.Credentials) appropriately. If you think there are new issues to be investigated here, please open a new issue for that in the appropriate repo. |
The main issue for us now is to be able to benefits from "outside" resource from :
the nightmare for us today is that, in order to simply press F5, we have to:
these "outside" resources are mainly :
but for now the use of a such "hack" does not go well for developper so it's kinda hard to move on azure if we can't just
it's probably time to review Azure Sdk public Api to allow developper to inject httpClient handler / expose it / etc ... but again i have some issue with just if this is not the case i would love this issue to be re-open as "on premise" apps need to connect to both LAN (internal resource) or Azure for example that's why a "callback" that could be registered like
it could be a "type" with an interface to implements instead of a straight callback this would be way more flexible from consumer PoV |
All environment variables are now working across all platforms. See: dotnet/corefx#37200, dotnet/corefx#37238 |
i edited the previous comment, do you want me to edit and put the rest here for clarity for further readers ? |
i feel bad i did not even read the title of the PR ;) (jumped too fast to part of the code) |
Not sure I understand. But if you feel that there is some new issue not addressed here regarding configuration with either .NET Core or other components, please open up a new issue. For things outside of .NET Core, it would be best to open up the issue in those repo's. Thx. |
Yes you are right is it technically possible to get the nightly of 3.0 running against current azure sdk ? |
@davidsh When you wrote https://github.com/dotnet/corefx/issues/34466#issuecomment-452845198 i kinda expected a completely new API. Given that the new design still sticks with |
@Suchiman one of the key reasons is that |
@karelz Now you can set two different "global" proxies. The
which is bad and in on itself not a reason to break it because you can. Maybe it makes sense to forward |
Good point. I didn't know that. Either way, it is scoped to the obsoleted APIs I don't think we should encourage anyone to use |
While i believe it's redundant to have two properties doing the same thing, just in different circumstances, i can get behind the idea of trying to hide public static IWebProxy DefaultWebProxy
{
get
{
return HttpClient.DefaultProxy;
}
set
{
HttpClient.DefaultProxy = value;
}
} similiar to how it's done for many things on the obsolete |
That would just encourage more people to use it and it would confuse more users. I don't think it is a good idea. |
time to remove That will fix any potential confusion |
We don't remove APIs. Especially when they are useful for compat and migration from Framework. |
While i'm not a fan of having users find out the hard way that this API is intentionally broken in certain ways, i'm out of arguments ¯_(ツ)_/¯ |
I don't think anyone answered your specific question here. I'm not sure what you mean by "current azure sdk". But if you want to try out .NET Core 3.0, you can download the latest preview release. https://devblogs.microsoft.com/dotnet/announcing-net-core-3-0-preview-5/ |
Hello again,
so i tried :
Is this PR included in this SDK ?
Final touch, we use a
EDIT: |
@tebeco It's not efficient to comment on this closed issue. This issue was closed since it reflected the new HttpClient.DefaultProxy API that was added, dotnet/corefx#37333. But your scenario is trying to use environment variables to set global proxy. It's not using the new HttpClient.DefaultProxy API (which would have to be used with dotnet restore itself). We did add support for using environment variables to set global proxy information, dotnet/corefx#37200. If there are problems with that working for your scenario, please open a new issue. We can then investigate it. Please also describe how your environment is set up in terms of authenticated proxy and what authentication schemes are being used. Thanks! |
[edit: this top post has been edited to show the final draft proposal for a new API]
API Proposal
Behaviors
The property will default to a non-null value representing the platform/system proxy. With PR 37328, all platforms can use environment variables as the first possible choice of proxy settings. If those variables are set, then an IWebProxy interface of an instance of the internal HttpEnvironmentProxy class is returned.
If the environment variables are not set, then the following happens for the default value of this property:
The property will never return null.
The property can be set to any object implementing the IWebProxy interface. This property cannot be set to 'null'. Doing so will throw an exception,
ArgumentNullException
.Sample API usage
History of proxy support in .NET
https://gist.github.com/davidsh/f8768c02faf714de9a3029cbf0166f18
Background
HttpClient has support for reading system proxy configuration (IE settings on Windows, Environment variables on Linux, etc). However, there is no mechanism for setting credentials (i.e. CredentialCache.DefaultCredentials) for a system configured proxy. Currently, this requires creating an HttpClientHandler instance and setting its DefaultProxyCredentials property.
For cases where libraries are creating the HttpClient object, there is no way for consumers of the library to set these credentials easily.
The .NET Framework supports using app.config/web.config settings with a system.net section:
This allows applications to inject a global proxy configuration for any HttpClient that is created. However, .NET Core doesn't natively support configuration files.
.NET Framework currently has the WebRequest.DefaultWebProxy property to control this for all HttpWebRequest objects (and affects HttpClient) as well. On .NET Core, the WebRequest class is considered legacy. But we need a similar way to set global proxy config on .NET Core for HttpClient usage.
A static API on HttpClient (or other object) created for global proxy configuration for the application could be used by upstack components to create a similar configuration file (i.e. app.config) behavior.
The text was updated successfully, but these errors were encountered: