Playing about with Apache2
State of play: ESNI seems to work ok. Not much tested of course;-) Currently deployed on https://defo.ie:9443.
Getting ESNI working was a bit harder than with nginx as
mod_ssl sniffed the ClientHello as soon as one is seen (i.e., before ESNI
processing) and then set the key pair to use based on the cleartext SNI. I
used the ESNI callback instead (if ESNI configured) so the
call (you can guess what that does:-) happens after successful server-side
Clone and Build
I started by forking httpd from https://github.com/apache/httpd, just because it's familiar. That's apache 2.5 - whereas 2.4 is probably what's widely used. Might want to revert to that later, but we'll see (also later:-).
Turns out that needs the Apache Portable Runtime (APR) to build. (That name
rings a bell from the distant past;-) As recommended, my httpd build has the
APR stuff in a
srclib sub-directory of the httpd source directory.
$ cd $HOME/code $ git clone https://github.com/sftcd/httpd $ cd httpd $ cd srclib $ git clone https://github.com/apache/apr.git $ cd .. $ ./buildconf ... stuff ...
Before running configure, this build seems to assume that OpenSSL shared
objects will be in a
lib subdirectory of the one we specify, and similarly
include directory. The latter is true of OpenSSL builds, but the
former is not (in my case anyway). We'll work around that with a link:
$ ln -s $HOME/code/openssl $HOME/code/openssl/lib
If you re-configure your OpenSSL build (e.g. re-running
$HOME/code/openssl/config) then you may need to re-do the above step.
And off we go with configure and make ...
$ ./configure --enable-ssl --with-ssl=$HOME/code/openssl ... loads of stuff ... # I got an error on the 2nd last ouput line there: # rm: cannot remove 'libtoolT': No such file or directory # but it seems to work out ok so far $ make -j8 ... lotsa lotsa stuff ... - The above was on a laptop running Ununtu 19.10 on which lots of other s/w has been built. - An Ubuntu 18.04 server required an additional ``sudo apt install libxml2-dev`` and adding ``--with-libxml2`` to the configure command line above to get that to work.
After running configure, I see mention of
modules/ssl/modules.mk that seems to the right things with
includes and shared objects.
Other configure options I may want (later): --enable-debugger-mode --enable-log-debug
Generate TLS and ESNI keys
This should be the same as for nginx
At least, I'm using the same keys for now and that seems ok.
ESNI Configuration in Apache
I added a server-wide
SSLESNIKeyDir setting (as with
lighttpd that ought have the directory where ESNI key pair
files are stored, and we then load those keys as before using a
load_esnikeys() function in
ssl_module_init.c. That seems to load keys
ok. There's an example in apachemin.conf.
I created a testapache.sh script to start a local instance of apache for example.com and baz.example.com on port 9443. That uses (what I hope is) a pretty minimal configuration that can be found in apachemin.conf. That starts an instance of httpd listening on port 9443 with VirtualServers for example.com (default) and baz.example.com.
When that's running then you can use curl to access web pages:
$ cd $HOME/code/openssl/esnistuff $ ./testapache.sh Killing old httpd in process 17365 Executing: httpd -f apachemin.conf $ $ curl --connect-to example.com:9443:localhost:9443 https://example.com:9443/index.html --cacert cadir/oe.csr ... you should see HTML now ... $ curl --connect-to baz.example.com:9443:localhost:9443 https://baz.example.com:9443/index.html --cacert cadir/oe.csr ... you should see slightly different HTML now ...
If I try my testclient against an apache server with no ESNI configured I get the expected behaviour, which is for the server to return a GREASE ESNI value, when it gets sent one.
$ ./testclient.sh -p 9443 -s localhost -H baz.example.com -c example.com -P esnikeydir/ff03.pub -d ... loadsa stuff... ESNI Nonce (16): 96:52:2d:18:f9:bc:09:7e:8e:70:cb:1d:bf:db:25:50: Nonce Back: <<< TLS 1.3, Handshake [length 006c], EncryptedExtensions 08 00 00 68 00 66 00 00 00 00 ff ce 00 5e 01 55 8c 49 42 e3 30 d0 9d b7 3c ce fe 14 ad 13 ea 1d 2b 27 97 63 eb e8 79 42 e3 9f b8 15 b4 76 7a 19 85 d8 ab 8c 9c 59 82 eb 2d 05 83 16 75 18 80 1f b6 24 2c ab c0 c6 a7 6d 03 28 ab 53 b1 44 8c e7 ESNI: tried but failed
The "ff ce" just after the "Nonce Back" line there is the extension type for the GREASEd value - in that case it's 0x5e long. (The extract above doesn't have the entire value in case you're wondering.)
Trying after ESNI is configured now works and (with OpenSSL tracing on) looks like:
$ ./testclient.sh -p 9443 -s localhost -H baz.example.com -c whatever -P esnikeydir/ff03.pub -d -f index.html ... lotsa stuff ... ./testclient.sh Summary: Nonce sent: ESNI Nonce: buf is NULL ESNI H/S Client Random: buf is NULL -- ESNI Nonce (16): 8f:90:5c:63:d9:83:4c:ae:83:3b:75:0b:0a:39:89:1a: Nonce Back: EncryptedExtensions, Length=23 extensions, length = 21 extension_type=encrypted_server_name(65486), length=17 Got an esni of length (17) ESNI (len=17): 008F905C63D9834CAE833B750B0A39891A ESNI: success: clear sni: 'whatever', hidden: 'baz.example.com'
Without OpenSSL tracing you'll see fewer lines but it's the last one that counts.
In the apache server error log (with "info" log level) we also see:
[Sat Nov 16 07:30:46.717225 2019] [ssl:info] [pid 7769:tid 139779161855744] [client 127.0.0.1:52010] AH01964: Connection to child 129 established (server example.com:443) [Sat Nov 16 07:30:46.718464 2019] [ssl:info] [pid 7769:tid 139779161855744] [client 127.0.0.1:52010] AH10246: later call to get server nane of |baz.example.com| [Sat Nov 16 07:30:46.718519 2019] [ssl:info] [pid 7769:tid 139779161855744] [client 127.0.0.1:52010] AH10248: init_vhost worked for baz.example.com
Code changes in httpd
Quick notes on code changes I've made so far:
All changes are within
I've bracketed my changes with
#ifdef HAVE_OPENSSL_ESNI. That's defined in
ssl_private.hif the included
The build generated a few warnings of deprecated OpenSSL functions, but seems ok otherwise. (This is similar to what I saw in nginx and lighttpd.) I modified calls to these as I did for lighttpd but the changes may be dodgier in this case and I likely won't be testing them (soon) as they seem related to client auth and CRLs. The deprecated functions are listed below
ap_log_error()liberally for now, mostly with
APLOG_INFOlevel (or higher). There's a semi-automated log numbering scheme - the idea is to start with code that uses the
APLOGNO()macro with nothing in the brackets, then to run a perl script (from $HOME/code/httpd) that'll generate the next unique log number to use, and modify the code accordingly. (I guess that would need re-doing when a PR is eventually submitted but can cross that hurdle when I get there.) As I'll forget what to do, the first time I used this the command I ran was:
$ cd $HOME/code/httpd $ perl docs/log-message-tags/update-log-msg-tags modules/ssl/ssl_engine_config.c
Adding the SSLESNIKeyDir config item required changes to:
I added a
load_esnikeys()function as with other servers, (in
ssl_engine_init.c) but as that is called more than once (not sure how to avoid that yet) I needed it to not fail if all the keys we attempt to load in one call are there already. That's a change from what I did with other servers. That seems to be called more than once for each VirtualHost at the moment, which could do with being fixed (but doesn't break).
There are various changes in
ssl_engine_kernel.cto handle ESNI.
With a bit of arm-wrestling I figured out how to run apache in the debugger loading all the various shared libraries needed with one process. Since that's too much to type each time, I made an apachegdb.sh script to do that. If you give it a function name as a command line argument it'll start the server with a breakpoint set there. With no command line argument it just starts the server.
Reloading ESNI keys
Apparently giving apache a command line argument of "-k graceful" causes a graceful reload of the configuration, without dropping existing connections. (Not sure how well I can test that proposition.) In any case, "-k graceful" does seem to have the required effect, so we'll try that whenever we deploy in a context with regular key updates. For the present that can be done via the testapache.sh script by providing a "graceful" parameter to the script:
$ ./testapache.sh graceful Telling apache to do the graceful thing
In the error.log file we see:
[Sat Nov 16 15:21:14.405010 2019] [mpm_event:notice] [pid 26579:tid 140622617413440] AH00493: SIGUSR1 received. Doing graceful restart followed by some (but fewer) of the usual startup lines.
If we check the process IDs, that seems to be behaving as desired:
$ ps -ef | grep httpd stephen 26579 1882 0 15:20 ? 00:00:00 httpd -d ./esnistuff -f ./esnistuff/apachemin.conf stephen 26580 26579 0 15:20 ? 00:00:00 httpd -d ./esnistuff -f ./esnistuff/apachemin.conf stephen 26581 26579 0 15:20 ? 00:00:00 httpd -d ./esnistuff -f ./esnistuff/apachemin.conf stephen 26582 26579 0 15:20 ? 00:00:00 httpd -d ./esnistuff -f ./esnistuff/apachemin.conf stephen 26665 24624 0 15:20 pts/2 00:00:00 grep --color=auto httpd $ ./testapache.sh graceful Telling apache to do the graceful thing $ ps -ef | grep httpd stephen 26579 1882 0 15:20 ? 00:00:00 httpd -d ./esnistuff -f ./esnistuff/apachemin.conf stephen 26709 26579 0 15:21 ? 00:00:00 httpd -d ./esnistuff -f ./esnistuff/apachemin.conf stephen 26710 26579 0 15:21 ? 00:00:00 httpd -d ./esnistuff -f ./esnistuff/apachemin.conf stephen 26711 26579 0 15:21 ? 00:00:00 httpd -d ./esnistuff -f ./esnistuff/apachemin.conf stephen 26810 24624 0 15:21 pts/2 00:00:00 grep --color=auto httpd
(Note that the above output from
ps has been edited for clarity, in reality you'd see
longer absolute path names there due to how testapache.sh starts the
server. I'm not sure if that's really needed or not though.)
As with lighttpd and nginx I added the following variables that are now visible to PHP code:
successmeans that others also mean what they say
SSL_ESNI_HIDDEN- has value that was encrypted in ESNI (or
SSL_ESNI_COVER- has value that was seen in plaintext SNI (or
I setup PHP for my apache deployment on https://defo.ie:9443. That's not part of the localhost test setup, and there were a couple of things to do:
- If needed, install fast-cgi: $ sudo apt install php7.2-cgi - I edited ``/etc/php/7.2/fpm/pool.d/www.conf`` to use localhost:9000, added ``proxy_module`` and ``proxy_fcgi_module`` to the global apache config and turn on PHP and added the following to the apache config for the VirtualHost using ESNI: <FilesMatch "\.php$"> SetHandler "proxy:fcgi://127.0.0.1:9000" </FilesMatch> Options +ExecCGI
- Fix up 1st error.log line that says e.g. "Connection to child 128 established (server example.com:443)" since we're not using port 443 at all
- Check how ESNI key configuration plays with VirtualHost and other stanzas.
load_esnikeys()is still being called a lot of times.)
- Add other ESNI key configuration options (i.e. SSLESNIKeyFile) - maybe solicit feedback from some apache maintainer first.
- Check if changes for deprecated functions break anything