Notes as I ESNI-enabled lighttpd-1.4.
-
I made a fork
$ ./autogen.sh ... stuff ... # I don't have bzip2 dev/headers and want my own openssl build so... # The below may also need --without-zlib $ ./configure --with-openssl=$HOME/code/openssl --with-openssl-libs=$HOME/code/openssl --without-bzip2 ... stuff ... $ make ... stuff ...
First idea is to have a minimal lighttpd config that can re-use the keys (TLS
and ESNI) otherwise used by testserver.sh - so we'll put things below
esnistuff in our openssl repo clone for now. I modified the
make-example-ca.sh script to produce the catenated private key +
certificate files that lighttpd needs to match our configuration.
That config is in lighttpdmin.conf
That basically has example.com and foo.example.com both listening on port 3443.
To ESNI-enable that I added three new lighttpd configuration settings:
- ssl.esnikeydir - the name of a directory we scan for ESNI key files (as
produced by
mk_esnikeys.c) - we load all key pairs where we find matching .priv and .pub files in that directory with the right content This allows for "outisde" key management as noted in our notes on web server integration. - ssl.esnirefresh - a time in seconds specifying how often the server should try re-load the keys (default: 1800)
- ssl.esnitrialdecrypt - set to "disable" (exactly) to turn off trial decryption, (it's is on by default).
Trial decryption here means if an ESNI extension received from a client has a digest that doesn't match any loaded ESNI key, then we go through all loaded ESNI keys and try use each to decrypt anyway, before we fail. For lighttpd, that seems to make sense as we're expecting servers to be small and not have many ESNI keys loaded.
The server will re-load all ESNI keys found inside the configured directory
once every refresh period using SSL_esni_server_enable which will reload
the keys if the relevant files are new or were modified. Before doing that it
flushes all the ESNI keys loaded more than a refresh period ago via
SSL_esni_server_flush_keys(). The upshot should be that the server
reflects the set of keys on disk.
The script testlighttpd.sh sets environment vars and
then runs lighttpd from the build, listening (for HTTPS only) on port 3443:
$ ./testlighttpd.sh
That starts the server in the foreground so you need to hit ^C to exit.
There's some temporary logging about ESNI that'll go away when we're more
done.
You can test that without ESNI with either of:
$ ./testclient.sh -p 3443 -s localhost -n -c example.com -d -f index.html
$ ./testclient.sh -p 3443 -s localhost -n -c foo.example.com -d -f index.html
Even before we changed any lighttpd code, ESNI greasing worked!
$ ./testclient.sh -p 3443 -s localhost -n -c example.com -d -f index.html -g
You can see the 0xffce extension value returned in the EncryptedExtensions with that, so it does seem to be using my ESNI code.
Based on the above new configuration settings (i.e. if esnikeydir is set) I added
the load_esnikeys() function to call SSL_esni_server_enable(), and that...
seems to just work, more-or-less first time. Who'da thunk! :-)
To try that out:
$ ./testlighttpd.sh
2019-09-28 16:37:12: (mod_openssl.c.862) load_esnikeys worked for /home/stephen/code/openssl/esnistuff/esnikeydir/ff01.pub
2019-09-28 16:37:12: (mod_openssl.c.862) load_esnikeys worked for /home/stephen/code/openssl/esnistuff/esnikeydir/e3.pub
2019-09-28 16:37:12: (mod_openssl.c.862) load_esnikeys worked for /home/stephen/code/openssl/esnistuff/esnikeydir/ff03.pub
2019-09-28 16:37:12: (mod_openssl.c.862) load_esnikeys worked for /home/stephen/code/openssl/esnistuff/esnikeydir/e2.pub
... then in another shell...
$ ./testclient.sh -p 3443 -s localhost -H foo.example.com -c example.com -d -f index.html -P esnikeydir/ff03.pub
...
OPENSSL: ESNI Nonce (16):
83:a5:0b:da:86:5a:f0:12:cd:28:e2:93:ea:56:f5:cb:
Nonce Back: <<< TLS 1.3, Handshake [length 001b], EncryptedExtensions
08 00 00 17 00 15 ff ce 00 11 00 83 a5 0b da 86
5a f0 12 cd 28 e2 93 ea 56 f5 cb
ESNI: success: cover: example.com, hidden: foo.example.com
And the esnistuff/lighttpd/log/access.log file for ligtthpd said:
...
127.0.0.1 foo.example.com - [28/Sep/2019:16:37:55 +0100] "GET /index.html HTTP/1.1" 200 458 "-" "-"
...
Yay!
Deployment on defo.ie
When I deployed that on defo.ie I noted fairly quickly that it didn't work:-)
Turned out the lighttpd mod_openssl.c:mod_openssl_client_hello_cb function
had two definitions - one that peeked into the TLS ClientHello octets to
extract the SNI, and another that called an OpenSSL to get the servername.
Using the latter with my fork results in the right thing happening for ESNI,
but of course the former would not. Easy enough fix to just force use of the
OpenSSL API. (Though presumably this may break wherever that peeking into
octets was really needed? Hopefully it was just legacy code or something.)
Anyway, good lesson that some applications might not be using all OpenSSL APIs
as designed and might be doing their own bits of TLS.
With that done, FF nightly and my test scripts both seem ok with things:-)
I'll go for having a bit of PHP script inside index.html displaying a check mark or cross, depending whether ESNI was used to access the page or not. I'm following the relevant bits of these instructions.
-
Install PHP if needed... (versions may be different, so 7.2 might be something else everywhere)
$ sudo apt-get -y install php7.2-fpm php7.2 -
Edit
/etc/php/7.2/fpm/php.inito uncommentcgi.fix_pathinfo=1 -
Edit your lighttpd config to include:
server.modules += ( "mod_fastcgi" ) fastcgi.server += ( ".php" => (( "socket" => "/var/run/php/php7.2-fpm.sock", "broken-scriptfilename" => "enable" )) )
I then further modified the lighttpd server (in mod_openssl.c:esni_status2env)
so that some ESNI related settings are placed into the environment. Those can be
used by e.g. PHP scripts.
Those are:
SSL_ESNI_STATUS: values can be:- "not attempted" - if the client didn't include the TLS ClientHello extension at all
- "success" - if it all worked (succesful ESNI decrypt)
- "tried but failed" - something went wrong during attempted decryption
- "worked but bad name" - this is a client-side error, if the TLS server cert didn't match the ESNI
- "error getting ESNI status" - if the call to
SSL_esni_get_statusfailed
SSL_ESNI_HIDDEN: will contain the actual ESNI used or "EMPTY"SSL_ESNI_COVER: will contain the cleartext SNI seen or "EMPTY"
Here's a PHP snippet that will display those:
<?php
function getRequestHeaders() {
$headers = array();
foreach($_SERVER as $key => $value) {
if (substr($key, 0, 9) <> 'SSL_ESNI_') {
continue;
}
$headers[$key] = $value;
}
return $headers;
}
$headers = getRequestHeaders();
foreach ($headers as $header => $value) {
echo "$header: $value <br />\n";
}
?>
For now, similar information is also written to the lighttpd error.log for every request if logging is enabled. That has the result, the cover (if any) and the hidden (if any) and looks like:
2019-09-30 16:18:02: (mod_openssl.c.462) esni_status: success cover.defo.ie only.esni.defo.ie
2019-09-30 16:29:18: (mod_openssl.c.462) esni_status: not attempted NULL NULL
2019-09-30 16:29:38: (mod_openssl.c.462) esni_status: success NULL canbe.esni.defo.ie
The basic idea here is to explore whether or not it's useful to mark a VirtualHost as "ESNI only", i.e. to try deny it's existence if it's asked for via cleartext SNI. I'm very unsure if this is worthwhile but since it could be done, it may be fun to play and see if it turns out to be useful.
To that end we've added an "ssl.esnionly" label that can be in a lighttpd configuration for a TLS listener. If that is present and if the relevant server.name is used in the cleartext SNI (with or without ESNI) then the TLS connection will fail. For example, in my localhost test setup baz.example.com is now maked "ESNI only" as is only.esni.defo.ie in our test deployment.
Failing this check is logged in the error log, e.g.:
2019-10-07 21:33:33: (mod_openssl.c.531) esni_status: not attempted cover: NULL hidden: NULL
2019-10-07 21:33:33: (mod_openssl.c.644) esnionly abuse for only.esni.defo.ie from 2001:DB8::bad
2019-10-07 21:33:33: (mod_openssl.c.2130) SSL: 1 error:140000EA:SSL routines::callback failed
That log line includes the requesting IP address for now.
-
Test whether or not a client could assert the
SSL_ESNI_STATUSby sending an HTTP header field, e.g.:$ curl -H "SSL_ESNI_STATUS:brat" --connect-to foo.example.com:443:localhost:3443 https://foo.example.com/index.html --cacert cadir/oe.csr -
The check as to whether or not ESNI keys need to be re-loaded happens with each new TLS connection. (Actually loading keys only happens when the refresh period has gone by.) There may well be a better way to trigger that check, e.g. there is some timing-based code in
server.cbut putting OpenSSL-specific code in there would seem wrong, so maybe come back to this later. -
The interaction between "outside" key management and re-publication, coupled with the way I'm reloading keys caused a problem - initially keys were being reloaded every 1200 seconds, but there was only 3 minutes between the time when the "outside" key manager job generated new keys and the time when the zonefactory (re-)publisher tested to see if they worked. So that test was failing, resulting in the new keys not being published and things getting out of whack. As a quick, temporary, fix, I'm reloading keys every 2 mins now, but this just highlights the need for a different interface, e.g. sending a signal that a reload is needed or something.