TLS: Provide ESNI support framework for curl and libcurl

[niallor]

TLS: Provide ESNI support framework for curl and libcurl

The proposed change provides a framework to facilitate work to
implement ESNI support in curl and libcurl. It is not intended
either to provide ESNI functionality or to favour any particular
TLS-providing backend. Specifically, the change reserves a
feature bit for ESNI support (symbol CURL_VERSION_ESNI),
implements setting and reporting of this bit, includes dummy
book-keeping for the symbol, adds a build-time configuration
option (--enable-esni), provides an extensible check for
resources available to provide ESNI support, and defines a
compiler pre-processor symbol (USE_ESNI) accordingly.

Proposed-by: @niallor (Niall O'Reilly)
Encouraged-by: @sftcd (Stephen Farrell)
See-also: this message
Limitations:

• Book-keeping (symbols-in-versions) needs real release number, not 'DUMMY'.
• Framework is incomplete, as it covers autoconf, but not cmake.
• Check for available resources, although extensible, refers only
  to specific work in progress (described here) to implement ESNI for OpenSSL,
  as this is the immediate motivation for the proposed change.

[bagder]

You could squash all your configure commits into a single one to reduce the number of commits and make the PR easier to review...

[niallor]

Yes, indeed. Sorry.

[niallor]

Do you need me to consolidate this, or is it tolerable as it is just this once?

[niallor]

Done. I began to feel that the jumble was hurting even my head. I felt it was useful to use two commits (rather than just one) in order to keep handling of the command-line token --enable-esni separate from applying the corresponding configuration.

[sftcd]

(In the hope that mail via GH gets there...;-)

On 11/06/2019 13:16, Daniel Stenberg wrote: This check ought to actually check that one of the necessary ESNI functions are available in OpenSSL. We've often seen users have a mismatch between headers and library, so just having the header file present is probably not a good enough signal. Something like this: AC_CHECK_FUNCS( SSL_ESNI_get_status )

Ah - my fault for a bad comment in esni.h, the one I think you mean is actually SSL_get_esni_status. It'd be fine to check for that. (I just fixed that comment in a commit a second ago:-)

(I don't know what function to actually check for, I just pulled this out of the air)

I also added a note to esni.h to not change that function name without checking back with curl. Cheers, S.

[bagder]

(The appveyor build issue is due to #3770 and not your fault)

[niallor]

Appveyor: I was pretty sure I hadn't introduced anything to do with SMB. Thanks for confirming.

[niallor]

I'm waiting to see what Travis reports before making any changes.

[niallor]

Codacy is reporting a violation of the "no-consecutive-blank-lines" rule. I can't see why.

[bagder]

no-consecutive-blank-lines

I think you can safely ignore that.

[niallor]

Unless I've missed something, components flagged in coverage check seem to be outside scope of this PR.

[bagder]

yeah, ignore coveralls. It goes totally crazy at times with no explanation available.

[niallor]

After taking time to learn how to drive libcurl from the curl tool, and OpenSSL from libcurl, achieved apparently successful interaction with ESNI-aware server today. Much tidying-up remains to be done.

[bagder]

As a fan of ESNI I'm curious, how's this going?

[sftcd]

Hiya,

On 16/08/2019 09:59, Daniel Stenberg wrote: As a fan of ESNI I'm curious, how's this going?

Niall's vacating still so may give a fuller answer when he's back, but we basically have a build of curl+openssl that works vs. cloudflare's deployment and the defo.ie demo site I've put up. It's not complete nor at all ready for upstreaming of course, but we plan to write a HOWTO in the next few weeks so others can play with it and we can get some feedback before we work to make it more complete. Cheers, S.

[niallor]

Hi.

On 16 Aug 2019, at 10:52, sftcd (Stephen Farrell) wrote: On 16/08/2019 09:59, Daniel Stenberg wrote: > As a fan of ESNI I'm curious, how's this going? Niall's vacating still so may give a fuller answer when he's back, but we basically have a build of curl+openssl that works vs. cloudflare's deployment and the defo.ie demo site I've put up.

So here's the fuller answer. The state of play as of before my vacation is frozen as branch 'freeze-20190731' of both niallor/openssl (which tracks sftcd/openssl) and niallor/curl. Building these is currently "an exercise for the reader." We've made it work by implementing options in libcurl and corresponding CLI options in curl so that we could push the desired ESNI parameters through to the OpenSSL backend and demonstrate interoperability in advance of working out how to construct these parameters from the target URL and related data published in the DNS. We're not sure yet whether to leave these options in place to allow over-riding the constructed values for testing, as work-arounds for errors in DNS data, or whatever. Next steps (in no particular order) include the following. - Write the HOWTO that sftcd mentions, so that building curl against his ESNI-supporting OpenSSL is less of an adventure. - Understand from (or agree with) bagder what is needed before PR 4011 (or a successor) can be pulled into curl/curl. - Understand from (or agree with) OpenSSL team what is needed to have sftcd's ESNI code pulled into OpenSSL. - Work out how to construct (in libcurl, and automatically from the target URL) the server name which is to be encrypted. - Work out how to have libcurl retrieve published ESNI data from the DNS. - Update code as needed to track IETF work on the ESNI draft. I think that's enough for now. /Niall

[bagder]

Cool, thanks for this write-up. (in my current poll for what users would like to see in our roadmap going forward, ESNI is one of the top rated choices)

I think I will get a better understanding of things once your HOWTO becomes real and we can play and see what it entails. Here's some initial comments in the mean time:

1. I would encourage you to work hard on first minimizing the necessary OpenSSL patching and then to take your proposed changes to them sooner rather than later. If you're asking for API changes, those take time.
2. Extracting the server name is easy. curl already parses the URL and keeps the host name around internally in an easily accessible struct.
3. We need to discuss how to make curl pull the required data off DNS for ESNI. Currently, the only DNS-oriented code we have use c-ares or DoH, and I would probably suggest that a first shot at this is done with a build and transfer that use one of those. We can then discuss later on how we can do this for regular transfers with a non-c-ares-built curl.
4. We should also check if this can be done in a QUIC friendly way too. I haven't checked how/if ESNI works the same way there.

[niallor]

Thanks for the helpful feedback.

[niallor]

I'm (currently) minded to explore DoH first, as this gives privacy in transit.

[bagder]

Makes perfect sense for a start, and for example Firefox still only supports ESNI when using DoH.

[niallor]

@bagder, as you put it, "We need to discuss how to make curl pull the required data off DNS for ESNI." Where is best: here, on curl-library list, offline in P2P email, or somewhere I haven't thought of?

[niallor]

@sftcd is following up on point 1 above. I've dealt with point 2, but will hold off on updating the branch that this PR depends on, likely at least until ESNI data is being fetched from DNS within libcurl.

[bagder]

The curl-library list is the best place for discussions, questions and brain storming.

[niallor]

This PR is just for reserving a code-point and implementing configuration- and build-time support for it. I don't expect to make further changes, unless to correct code-point collisions as in 5265d94.
If this PR were merged soon, other people minded to do ESNI work would have a code-point and configuration/build framework to work from. Is there more that I need to do so that this can happen?

[bagder]

1. It's an usual procedure and ask. I generally dislike merging code that only adds building blocks for a potentially future work since that future work can take a long time or even never happen and then we've only merged ourselves a lot of work. I typically prefer to wait until we can actually merge something that can be used, at least to some extent by some eager users. Are you saying that merging this PR "early" will actually help you work on ESNI better/easier/faster?

2. I propose you rename you docs file to plain and simple ESNI.md so that it can be the home of all ESNI related docs and in-progress notes.

3. if this is actually aimed at merging the [WIP] prefix should probably be removed as that suggests something else...

[niallor]

1. Yes (to final question). OTOH, if you prefer to have "something that can be used, at least to some extent by some eager users", then maybe I should close this and submit another PR which allows building something which can interoperate with existing servers, such as the targets which have been used in demo, Cloudflare and DEfO. It seems to me that you should call this according to whether you prefer multiple, compartmentalized, merges or rather a big one, kitchen-sink style.
2. Fair enough: happy to do, but depends on your call (see 1).
3. See 2.
   Thanks.

[bagder]

Thanks. Okay, I'm game. Let's work on getting this PR merged as a foundation to ease the ongoing work of getting the rest of the ESNI work going.

[niallor]

Great.

In docs/libcurl/symbols-in-versions, a release number is needed (instead of DUMMY) for when the symbol CURL_VERSION_ESNI will have been introduced. Should I update the PR with a real number, or is this something you would usually do afterwards? If the former, I'll need you to tell me the number.

You've dealt with item 3 (removing the [WIP] prefix.

I should look after item 2.

[bagder]

Go with 7.67.0 there and yes please update the PR accordingly. (We have 9 days before the window closes for that, but it seems that should be enough!)

[niallor]

AppVeyor failures don't look like they have to do with this PR. Trying rebase ...

[bagder]

Yes, the appveyor failure is unrelated.

[bagder]

Thanks. You'll notice that I did some minor edits before merge.

[niallor]

Great. Thanks for your patience with all the learning I had to do to get this far; next PR should be easier as a result. It will have enough function for an enthusiast to demonstrate interoperability against an ESNI-aware server.