HSTS support for HttpClient

[maartenba]

While the ASP.NET repository comes with all required infrastructure to support a web application to emit HTTP Strict Transport Security (HSTS) headers that browsers can use to adhere to the applicable RFC (6797), the client-side in .NET is lacking support for this.

It may be of interest to add (optional) support for this in HttpClient, probably through providing a DelegatingHandler implementation, that ensures that a server which requires a secure connection is always accessed using https://.

There are many scenarios for this, one example can be found in NuGet/Home#6940, where a suggestion is that the NuGet client should support it so that NuGet.org requires a secure connection at all times. Another example could be where application developers are accessing an HTTP-based API and a secure connection is required at all times. The main reason for that is in the RFC itself: making sure that certain headers (think authentication) are never sent over a non-secure connection.

I have been working on such handler (quick-and-dirty proof-of-concept) at https://github.com/maartenba/MaartenBalliauw.Extensions.Http.Hsts, and was wondering if there is interest in a proper PR to this repository (or corefxlab). And if it is, another question would be whether the handler requires support to load e.g. Chromium's preload list, much like web browsers do, to pre-populate a backing store with a list of hosts that always require a secure connection.

So in short: would this be of interest? If yes, what would be the best repo to contribute this to? (and if no, I'll just continue work on it and publish it as a NuGet package for those who need it)

[karelz]

@maartenba can you please summarize the RFC for client side? What does your handler do to help the users?
Naively, I assume server needs to start rejecting HTTP connections. And client has to use HTTPS connections. What else is there to do on client in the handler? (sorry the RFC is rather long and I didn't read it yet)

I don't think we should include list of known sites in CoreFX - that is mostly for browsers and not many apps use such endpoints. Maintaining such list would be also a challenge IMO. I think such a list (if useful) should be in separate nuget package, not maintained by Microsoft.

[maartenba]

Main idea is pretty much this:

• When host is known to require https, rewrite all requests to use https when attempting http
• When host says, on first response, that https is required we make future requests to that host using https

(with some peculiarities around subdomains, expiry of those rules, port numbers that should be preserved and sometimes rewritten)

The preload list is definitely not something I would see in corefx, yet the handler and base infrastructure to parse and respect HSTS headers could be (but could also be a separate package).

How does it help the users? By, for example, not sending basic auth headers to a host over https if that host says it requires https. By not allowing http requests if the host requires https. Much like the browser will know that GitHub requires https and will not even attempt http, even if you type it in the address bar. (Technically that's from the preload list but even if you don't have a preload list, GitHub will tell you in the first response that future responses need https)

So main question remains, would it be useful in corefx/corefxlab or do I make this a package for those who may need it.

[karelz]

Thanks! That is useful background.
I am not certain myself if it belongs to CoreFX vs. not. The safe path would be to make it standalone and eventually move it to CoreFX.

@dotnet/ncl @terrajobst any thoughts?

[maartenba]

Sounds good to me! Will leave the issue lingering here for a few days, and if not then I'll go the separate package route :-) Thanks Karel!

[karelz]

One interesting data point would be, how useful it is for apps. How many apps do we expect would leverage this functionality? (the benefit I assume would be perf - not wasting time (round-trips) with redirects on each request)

[maartenba]

Main benefit is security. Circling back to NuGet for example. Imagine I use NuGet.exe to push a package, and specify my API key, and address the http:// endpoint.

Even though NuGet.org has HSTS enabled, I would send my API key in a header, o er an unsecure http connection!

Now if NuGet.exe would use HSTS, and would preload just NuGet.org with an https requirement, that would never happen. Even if I would target http, the library would make that https, my credentials never transferred over http.

That would not need the full preload list, just that specific application adding its own known domain into an otherwise empty list. HSTS could be "core", a preload list populator could be a community thing (with regular, probably automated, updates of the preload list package)

As to how many libraries would use this, no idea to be honest :-) But if there is a user-entered URL in an app and e.g. credentials or other confidential data are sent, I can only hope they all already rewrite the request to https.

[blowdart]

I believe to be more useful the preload list needs to be there by default, and then updated every 3 months, like the one in Edge. We already use class generation in Unicode Encoding, so there's precedent for a tool to run as part of a build to generate a class.

There should also be a persistence mechanism, although that's more interesting, you want the file somewhere that could potentially be shared, or at least reloaded after a restart.

App transparency is best here, just drop in the new class and nothing else is needed.

[blowdart]

A couple of caveats for the preload, for myself as much as anyone else

The source at https://chromium.googlesource.com/chromium/src/net/+/master/http/transport_security_state_static.json?format=TEXT isn't the version in the browser. A pull from HEAD gets the latest, which may include people who added their site to preload, then changed their mind, but the removal hasn't taken affect.

The source is base64 encoded (ha, thanks google)

And there are comments in the JSON, pinned keys, and other mularky.

[karelz]

@blowdart are you going to push on including a list like that in .NET Core?
Given the impact on <1% of customers (wild guess from my side), I am not sure it is the right trade-off to sign ourselves up for regular maintenance of that list. I know it helps security, but ...

[blowdart]

I would if it were in the framework yes, both the preload and persistence.

[davidsh]

I would if it were in the framework yes, both the preload and persistence.

I don't think adding HSTS support to HttpClient is the right path.

Apps that need this type of feature should do so by writing a custom DelegatingHandler that can provide that support including doing the automatic redirection to the HTTPS endpoint.

[karelz]

@blowdart the persistence is IMO tricky - we would have to be able to write entries from any process into machine-wide info. That sounds like security nightmare. How would you imagine that to work?

[blowdart]

Tricky yes, but I have faith in all your abilities ;)

It is a bit of a nightmare, but the browsers do it already.

[maartenba]

So how about I try to mature this thing a bit in a separate repo?

[karelz]

@maartenba sounds like a good idea as it will likely take us a while to finalize a plan we all agree on :( (Sorry! And thanks for understanding!)

[maartenba]

All good, was my reason for opening this issue in the first place :-) Thanks all! Will report back in the next weeks.

[maartenba]

Here goes: https://github.com/maartenba/DotNetContrib.Net.Http.Hsts

Does not have the HSTS preload list from Chromium yet, but the README gives some examples on how (and why) to use it).

[karelz]

Triage: There are multiple opinions even on our side.
It still feels weird to put it into the platform itself.
However, to make it useful, we want larger by default exposure ... what about including it in HttpClientFactory? @davidfowl @rynowak

[rynowak]

We try not to put features in client factory - it's on opinionated way to configure things. If CoreFx shipped a message handler for this, then the factory will make it easy to wire up 😁

[fowl2]

Rather than a full HSTS implementation and all the complications that brings (persistence, preload list), what about a "HTTPS Everywhere"/Apple-style "App Transport Security" mode? Ideally unencrypted HTTP would be disabled by default, but let's take it step by step.