HTTPS Proxy support feature requests

[matteo-prosperi]

Hello,
As the support of HTTPS proxies is becoming more common, we need to execute HTTP and HTTPS requests through an HTTPS proxy.

To be clear, this is not about executing an HTTPS request through an HTTP proxy (which works well), it is about connecting to the proxy itself using HTTPS.

As an additional requirement, we are also supposed to authenticate the client using client certificates.

When I try to use a proxy created from an https URI

            HttpClientHandler handler = new HttpClientHandler();
            handler.Proxy = new WebProxy("https://127.0.0.1");
            HttpClient client = new HttpClient(handler);

I receive an Only the 'http' scheme is allowed for proxies. NotSupportedException.

I found some indication at https://github.com/dotnet/corefx/issues/20435 that this is not supported at all, so I am reopening this as a feature request.

Thanks

[richlander]

This looks like a good feature for .NET 5. I haven't heard enough requests about this scenario to warrant adding it earlier.

[scalablecory]

Can you give some use cases for HTTPS proxies?

[richlander]

Wouldn't it be the same as an HTTP proxy but with TLS? An HTTPS -> HTTP transition through an HTTP proxy would seem to undermine the value proposition of HTTPS. I cannot imagine that it is any more complicated than that.

[scalablecory]

An HTTPS -> HTTP transition through an HTTP proxy would seem to undermine the value proposition of HTTPS.

HTTPS requests have end-to-end encryption, regardless of if they go through a proxy, or if that proxy has encryption -- it is protected from the proxy as well.

A HTTPS proxy does not really add security, it merely shifts it: it makes it so that instead of your ISP being able to track what IPs you connect to, the proxy can.

[scalablecory]

Maybe @GrabYourPitchforks has some thoughts.

[matteo-prosperi]

We have been using https://github.com/fancy-owl/docker-mtls-https-proxy to test it.

CURL supports it by doing

curl --proxy-cert ./client.pem --proxy-cacert ./ca.crt --proxy https://some.uri.com https://i.want.to.see.this.com

We have customers who are setting up HTTPS proxies in their networks and requiring clients to authenticate against the proxy with client certificates.

[karelz]

Related to #17740 (Socks4/5 proxies)

[richlander]

@scalablecory -- that doesn't make sense to me. If you used an HTTP proxy, I assume any traffic after that proxy was "in the clear". That was my point, and avoiding that was my intended use case for an HTTPS proxy. If the proxy was one you trusted, then this model would seem to be fine. Right?

My mental model is this:

• There are scenarios for proxies.
• There are scenarios for HTTPS.
• This request targets the points where those scenarios overlap.

Again, is this any more complicated that that? I assume if you don't have an HTTPS proxy, then you violate the HTTPS-oriented requirements? I understand your point about proxy now being a trust point. It seems similar to using a third-party VPN. If the proxy isn't third party, then maybe no trust issue. Fair?

This seems to align with what @matteo-prosperi is saying.

[davidsh]

A HTTPS proxy does not really add security, it merely shifts it: it makes it so that instead of your ISP being able to track what IPs you connect to, the proxy can.

Besides consumer related scenarios (i.e. ISP's), we have heard from enterprise customers over the years wanting "TLS everywhere" which means that communicating with a proxy on the LAN should also use HTTPS. Currently even with communicating with a destination server over HTTPS, the initial 'CONNECT' verb to the proxy is unencrypted (HTTP right now). And that means any headers that get sent along (include 'Proxy-Authorization' headers) would be visible.

So, the scenario itself is reasonable. We would just have to prioritize this work and get a design/cost for it.

[richlander]

There has been talk about DNS with TLS for the same reason.

[davidsh]

There has been talk about DNS with TLS for the same reason.

There are significant disagreements about the usefulness of some of these related protocols (like DNS over HTTPS):
https://www.zdnet.com/article/dns-over-https-causes-more-problems-than-it-solves-experts-say/

But pure DNS over TLS (DoT) or using IPSEC would be the best for DNS solutions.

[matteo-prosperi]

I suppose in this case using client certificates also allows limiting which users or maybe even applications can interact with servers outside of a certain network boundary.

[scalablecory]

So lets draw out what a proxy connection looks like.

Client -> Client's ISP -> Proxy -> Proxy's ISP -> a series of tubes -> Server's ISP -> Server

For an HTTP proxy, the Client's ISP and the Proxy can see that you are talking to Server. Everyone else sees Proxy talking to Server.

For an HTTPS proxy, the Client's ISP can only see that you are talking to Proxy. This is the sole benefit of an HTTPS proxy -- if you can trust the proxy itself, or at least distrust your ISP, then you gain some small bit of anonymity from your ISP. Everything else remains the same.

Now, regardless of if your proxy is over TLS or not, an HTTPS request is encrypted all the way to Server: even if the Client's ISP can see that you're talking to Server, they still don't know what you're saying. You don't lose any of HTTPS benefits by going over an unencrypted proxy.

[scalablecory]

Note: not saying this is necessarily a bad idea, just that the real benefit is deceptively slim. I'd welcome some examples of when hiding from your ISP will be important under HttpClient usage scenarios.

If there are businesses that want to take a "feel good" TLS-everything approach, I'd hope they would use IPsec rather than relying on HTTP-level encryption inside their network, but I trust @davidsh knows the wants of that space better than me. That is a valid viewpoint to consider.

[richlander]

This is probably all true, but imagine that all the calls and all the end-points are running within a cloud environment. That's what this comment is making reference to: https://github.com/dotnet/corefx/issues/41658#issuecomment-539792003. There is no "ISP".

[richlander]

Let's also assume that the network traffic never leaves the cloud region and you are using an HTTPS version of Envoy.

[scalablecory]

The one significant benefit I'm seeing here is that HTTPS allows cert-based authentication. I'm not sure if we'd need a new API for this or if we could use the existing cert collections in HttpClientHandler.

Would love to know how common cert-based auth is for HTTPS proxies.

[richlander]

I assume there are two things going on:

• Use a proxy as part of a micro-services architecture where the network architecture is transparent to a given app.
• Use HTTPS as a defense-in-depth mechanism to prevent a class of attacks. It's also possible that this is required for regulated businesses.

[richlander]

This answers some of your questions: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/ssl

[davidsh]

If there are businesses that want to take a "feel good" TLS-everything approach, I'd hope they would use IPsec rather than relying on HTTP-level encryption inside their network, but I trust @davidsh knows the wants of that space better than me. That is a valid viewpoint to consider.

There are various initiatives in the regulatory space (government etc.) that are requiring encryption everywhere.

And despite its strong engineering appeal, IPsec hasn't really caught on as much as we would have liked with many enterprises because of the higher cost of setup, configuration, diagnostics, etc.

[opinionmachine]

So for instance, let's say you are using a service for ID&V which is well known and well funded but still unable to deploy normal web service auth. This world famous service provider instead asks its clients to authenticate to their proxy on their network and then proxy the web service calls through there. Obviously, talking HTTP over the entire internet whilst submitting PII in for validation purposes is not only undesireable but probably illegal, so this is not an option.

Today it isn't possible to call the proxy over HTTPS, while it was possible in .NET Core 2.x, presumably by mistake, and Microsoft is shouting that it's "Insane" to want to connect to a proxy over HTTPS, while I would argue it's "Insane" to make it impossible on purpose as there are no valid downsides.

[opinionmachine]

So lets draw out what a proxy connection looks like.

Client -> Client's ISP -> Proxy -> Proxy's ISP -> a series of tubes -> Server's ISP -> Server

For an HTTP proxy, the Client's ISP and the Proxy can see that you are talking to Server. Everyone else sees Proxy talking to Server.

For an HTTPS proxy, the Client's ISP can only see that you are talking to Proxy. This is the sole benefit of an HTTPS proxy -- if you can trust the proxy itself, or at least distrust your ISP, then you gain some small bit of anonymity from your ISP. Everything else remains the same.

Now, regardless of if your proxy is over TLS or not, an HTTPS request is encrypted all the way to Server: even if the Client's ISP can see that you're talking to Server, they still don't know what you're saying. You don't lose any of HTTPS benefits by going over an unencrypted proxy.

So, as I stated in my comment above, a scenario that is realistic for millions of people is the following:

1. Client network -> 2. Internet -> 3. Service Provider Proxy (because they don't know how to do normal web service auth, despite the billions of dollars of revenue) -> 4. Service

I e we need to connect to 3 without talking pure http over 2 as we're sending PII (which granted might be fake - that's what we're trying to validate, but still).

There is no valid reason not to have it, you had to break something for it not to be there by default. That's why people like this Identity Validation provider has no second thought requiring people to do this dodgy form of authentication, because they don't think it's totally mindblowing.

[karelz]

Triage: Original request -- support HTTPS proxies sounds reasonable, we should try to implement it in 5.0.
Some of the code already exists in the code base under if-statements.
With @davidsh's enterprise scenario testing, we should be able to test it easier.

[mottersheadt]

Any update on this feature? This will be very useful when it becomes available.

[mikekilburn]

Any plans to support this feature?

[richlander]

@karelz

[karelz]

Last time we looked into this, there were almost no scenarios for HTTPS proxies.
@normj can you please help me understand exactly the product (nginx?) that does support it? Is it mainstream these days?
I wonder why we hear about it from AWS and not from other sources.

@mottersheadt @mikekilburn which proxies do you use which support it? Are they GA, in Beta, or are you interested just from future-proofing perspective?

[opinionmachine]

Last time we looked into this, there were almost no scenarios for HTTPS proxies.

What is the reason to make it forbidden? That is extra effort once made just to make it impossible to connect to a proxy over HTTPS. There is no valid reason for that.

LexisNexis ID&V API cannot be the only one that requires using a remote proxy. This arbitrary unnecessary disabling of HTTPS reminds me of the gaslighting we got over the "crazy idea" to make Visual Studio 64-bit that "nobody wanted".

If AWS requiring it means we'll get in in .NET I'm happy.

[mottersheadt]

@karelz - I'm looking to support VGS's Forward Proxy, which is an HTTPS proxy requiring an additional cert to be added to the request. This product is GA and is often used to descope sensitive data from enterprise networks.

Here is an example of how the proxy can be used in a cURL request:

curl https://echo.apps.verygood.systems/post --cacert path/to/sandbox.pem \
  -x https://USERNAME:PASSWORD@tntqoslmkzi.SANDBOX.verygoodproxy.com:8443 \
  -H "Content-type: application/json" \
  -d '{"account_number": "ALIAS"}'

[normj]

@karelz My understanding from AWS user's point of view is this is most commonly wanted when making request from on-premise network to their AWS infrastructure through a VPN. They want complete end to end encryption with either TLS or MTLS through a proxy to AWS. Protecting themselves from bad actors within their own network.

When an enterprise has this situation they want all the languages that their application developers use to be able to support this configuration. AWS we have 12+ different languages and most all had this feature but .NET was one of the few outliers that didn't have this support.

[karelz]

@opinionmachine

What is the reason to make it forbidden? That is extra effort once made just to make it impossible to connect to a proxy over HTTPS. There is no valid reason for that.

It is not forbidden. It is not implemented. If you look at the code, you will see, it is not matter of single if-check. There is impact on connection pool, etc. There would need to be tests, E2E validation, etc. Almost every change is usually more involved than one would hope.

gaslighting we got over the "crazy idea" to make Visual Studio 64-bit that "nobody wanted"

I wasn't on VS team, but I know that 64-bit VS was a HUUUUUUGE investment. Easily hundreds or thousands of engineer-years (don't quote me, it is a wild guess). Let's not oversimplify things just because they seem to be simple from the outside. Let's judge the change based on its real cost.

LexisNexis ID&V API

Can you post a link? Is https proxy part of the API? (that seems weird)

... cannot be the only one that requires using a remote proxy

Well, if you read the thread here, you can notice that you are one of the first persons ever to mention a real product that needs it. 🤷

[karelz]

@mottersheadt great, thanks! That is helpful for our prioritization!

[karelz]

@normj thanks for info, few more questions:

making request from on-premise network to their AWS infrastructure through a VPN. They want complete end to end encryption with either TLS or MTLS through a proxy to AWS.

If they are under VPN, why is a proxy to AWS needed? I might be misunderstanding some common enterprise architecture here ... please bear with me.
Or is it defense in depth / paranoia to make everything HTTPS just to be sure? (I assume DNS will be next)
Naively, just secure connections should be good enough for communication ... that's why we didn't consider it important before.

When an enterprise has this situation they want all the languages that their application developers use to be able to support this configuration. AWS we have 12+ different languages and most all had this feature but .NET was one of the few outliers that didn't have this support.

Yeah, that totally makes sense from this point of view. Do you happen to have the list of languages that support it already handy? That would save us some research ...
Competitive disadvantage is a good argument ...

[opinionmachine]

It is not forbidden. It is not implemented. If you look at the code, you will see, it is not matter of single if-check. There is impact on connection pool, etc. There would need to be tests, E2E validation, etc. Almost every change is usually more involved than one would hope.

All I know is, it works in .NET Core 2.2 and was broken from 3.1 onwards. And making network connections over https works, so it can't be miles away.

I wasn't on VS team, but I know that 64-bit VS was a HUUUUUUGE investment. Easily hundreds or thousands of engineer-years (don't quote me, it is a wild guess). Let's not oversimplify things just because they seem to be simple from the outside. Let's judge the change based on its real cost.

The real reason of course was as you hint that the code was in such a state with hard assumptions of bitness that it would be a massive undertaking to change and on balance it wasn't worth it for a decade or so but instead of just saying that, we were
gaslighted with "nobody wants that, 64-bit makes no difference in performance".

Can you post a link? Is https proxy part of the API? (that seems weird)

I wish I could, but I left that employer a while ago so I don't have access to the docs anymore.

Well, if you read the thread here, you can notice that you are one of the first persons ever to mention a real product that needs it. 🤷

Yeah, I guess. I mean if Lexis Nexis just fixed their authentication method it wouldn't matter but the limitation is so arbitrary it doesn't make sense. I mean proxies are everywhere. Should nobody ever be able to use .NET to build a competitor to Istio because you can't connect to a reverse proxy over HTTPS?

[normj]

@karelz

If they are under VPN, why is a proxy to AWS needed? I might be misunderstanding some common enterprise architecture here ... please bear with me.

I make a poor sys admin so bare with me. I believe the concern is the traffic for the on-premise side. The proxy is on-premise which connects on-premise users to the VPN to AWS. So in our SDK's case developers using AWS SDK request will go workstation/on-premise-server -> proxy -> vpn -> AWS service but without HTTPS support that first hop is going over the on-premise network unencrypted.

Do you happen to have the list of languages that support it already handy?

I'll see if I can get you a complete list but this is my quick list.

• Java (https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html)
• Python (Add support for HTTPS connections to proxies. urllib3/urllib3#1679, Add support for TLS in TLS for HTTPS proxies. urllib3/urllib3#1806, Integrate TLS-in-TLS support into urllib3 urllib3/urllib3#1923)
• PHP (through third party package Guzzle https://docs.guzzlephp.org/en/stable/)
• Node.js (https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-proxies.html)
• Go (https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/custom-http/#configuring-a-proxy)
• Rust through community packages (https://docs.rs/hyper-proxy/latest/hyper_proxy/ https://docs.rs/hyper-socks/latest/hyper_socks/). Also here is an AWS Rust SDK issue discussing how to get it to work https_proxy support awslabs/aws-sdk-rust#169. The AWS Rust SDK is in preview.

[LiorBanai]

Hi,
it this planned for 8.X or for Future?

[karelz]

This will be part of .NET 8.0 Preview 7 (we missed the snap for Preview 6 by a few hours) -- dotnet/core#8438

[wfurt]

BTW it would be great if community can pick up daily build and give it try: https://github.com/dotnet/installer
My testing was very limited.

[LiorBanai]

Great. Thanks

[opinionmachine]

Thanks, folks, this is wonderful news. May you be blessed in every possible way! ❤️