I have a first version of Nginx with ESNI enabled working. Not much tested and but it was pretty easy and seems to work.
Clone and Build
First, you need our OpenSSL build:
$ cd $HOME/code $ git clone https://github.com/sftcd/openssl.git openssl-for-nginx $ cd openssl-for-nginx $ ./config --debug ...stuff... $ make ...go for coffee...
Then you need nginx:
$ cd $HOME/code $ git clone https://github.com/sftcd/nginx.git $ cd nginx $ ./auto/configure --with-debug --prefix=nginx --with-http_ssl_module --with-openssl=$HOME/code/openssl-for-nginx --with-openssl-opt="--debug" $ make ... go for coffee ...
- That builds openssl afresh (incl. a
make config; make clean) and then links static libraries from that build. Hence cloning the OpenSSL fork into
openssl-for-nginx- if you've another OpenSSL build (say for lighttpd this build would mess that up.
- The static libraries for OpenSSL end up below
makein the nginx directory doesn't detect code changes within OpenSSL. Bit brute-force but deleting that new
.openssldirectory gets you a re-build.
- End result is you need two clones of openssl if you want to build openssl shared objects (e.g. for lighttpd) and staticly for nginx. I mucked up a few times when using the same source tree for both. I'm sure that can be improved, but I've not figured out how yet.
Generate TLS and ESNI keys
We have a couple of key generation scripts:
- make-example-ca.sh that generates a fake CA and TLS server certs for example.com, foo.example.com and baz.example.com - that can be used for testing on localhost.
- make-esnikeys.sh generates ESNI keys for local testing
(Note that I've not recently re-tested those, but bug me if there's a problem and I'll check/fix.)
Run nginx for localhost testing
$ cd $HOME/code/openssl/esnistuff $ ./testnginx.sh ... prints stuff, spawns server and exits ... $ curl --connect-to baz.example.com:443:localhost:5443 https://baz.example.com/index.html --cacert cadir/oe.csr <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org...
If you'd prefer the server to not daemoise, there's a "daemon off;" line in the config file you can uncomment. That's useful with valgrind or gdb.
Valgrind seems to be ok wrt leaks in various tests, though it's a little harder to tell given the master/worker process model. Nothing definitely leaked though. (And our tests are pretty basic so far.)
ESNI configuration in Nginx
For now, we've got two different ways to turn on ESNI - by configuring
a directory that contains key files, or by configuring the names of
key files. I did the directory based approach first, as I'd used that
openssl s_server and lighttpd, but one of
the upstream maintainers wasn't keen on that so I added the file name
based approach as well.
At the moment, I'm still in the process of testing the 2nd option there.
I added an ESNI key directory configuration setting that can be within the
http stanza (and maybe elsewhere too, I don't fully understand all that
yet;-) in the nginx config file. Then, with a bit of generic parameter handling
and the addition of a
load_esnikeys() function that's pretty much as done
for lighttpd, ESNI... just worked!
load_esnikeys() function expects ENSI key files to be in the configured
directory. It attempts to load all pairs of files with matching
<foo>.pub file names. It should nicely skip any files that don't parse
correctly. I think that may be implemented portably (I use
now instead of
readdir but more may be needed for it to work ok on win32,
You can see that configuration setting, called
ssl_esnikeydir in our
$ ./testnginx.sh ... stuff ... $ /testclient.sh -p 5443 -s localhost -H baz.example.com -c example.net -P esnikeydir/ff03.pub Running ./testclient.sh at 20191012-125357 ./testclient.sh Summary: Looks like 1 ok's and 0 bad's.
We log when keys are loaded or re-loaded. That's in the error log and looks like:
2019/10/12 14:32:13 [notice] 16953#0: load_esnikeys, worked for: /home/stephen/code/openssl/esnistuff/esnikeydir/ff01.pub 2019/10/12 14:32:13 [notice] 16953#0: load_esnikeys, worked for: /home/stephen/code/openssl/esnistuff/esnikeydir/e3.pub 2019/10/12 14:32:13 [notice] 16953#0: load_esnikeys, worked for: /home/stephen/code/openssl/esnistuff/esnikeydir/ff03.pub 2019/10/12 14:32:13 [notice] 16953#0: load_esnikeys, worked for: /home/stephen/code/openssl/esnistuff/esnikeydir/e2.pub 2019/10/12 14:32:13 [notice] 16953#0: load_esnikeys, total keys loaded: 4
Note that even though I see 3 occurrences of those log lines, we only end up with 4 keys loaded as the library function checks whether files have already been loaded. (Based on name and modification time, only - not the file content.)
We log when ESNI is attempted, and works or fails, or if it's not tried. The success case is at the NOTICE log level, whereas other events are just logged at the INFO level. That looks like:
2019/10/13 14:50:29 [notice] 9891#0: *10 ESNI success cover: example.net hidden: foo.example.com while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:5443
The second option for loading ESNI keys is to have both public and private key
in one file and to load a bunch of those. This config setting can be
in the same places as
ssl_esnikeydir, that is, within the http settings
or below. (And I still don't grok all that stuff:-)
ssl_esnikeyfile esnikeydir/ff01.key; ssl_esnikeyfile esnikeydir/ff03.key;
Since the files here are a mixture of private and public keys, we need both to be PEM encoded. For no particularly good reason, the priavte key must be first in the file. An example of such a file might be:
-----BEGIN PRIVATE KEY----- MC4CAQAwBQYDK2VuBCIEIEDyEDpfvLoFYQi4rNjAxAz7F/Dqydv5IFmcPpIyGNd8 -----END PRIVATE KEY----- -----BEGIN ESNIKEY----- /wG+49mkACQAHQAgB8SUB952QOphcyUR1sAvnRhY9NSSETVDuon9/CvoDVYAAhMBAQQAAAAAXYZC TwAAAABdlBoPAAA= -----END ESNIKEY-----
Reloading ESNI keys
Nginx will reload its configuration if you send it a SIGHUP signal. That's easier to use than we saw with lighttp, so if you change the set of keys in the ESNI key directory then you can:
$ kill -SIGHUP `cat nginx/logs/nginx.pid`
...and that does cause the ESNI key files to be reloaded nicely. If you add and remove key files, that all seems ok, I guess because nginx cleans up (worker) processses that have the keys in memory. (That's nicely a lot easier than with lighttpd:-)
As with lighttpd I added the following variables that are now visible to PHP code:
- ``SSL_ENSI_STATUS`` - ``success`` means that others also mean what they say - ``SSL_ESNI_HIDDEN`` - has value that was encrypted in ESNI (or ``NONE``) - ``SSL_ESNI_COVER`` - has value that was seen in plaintext SNI (or ``NONE``)
To see those using fastcgi you need to include the following in the relevant bits of nginx config:
fastcgi_param SSL_ESNI_STATUS $ssl_esni_status; fastcgi_param SSL_ESNI_HIDDEN $ssl_esni_hidden; fastcgi_param SSL_ESNI_COVER $ssl_esni_cover;
Some OpenSSL deprecations
On 20191109 I re-merged my nginx fork with upstream, and then built against the
latest OpenSSL. I had to fix up a couple of calls to now-deprecated OpenSSL
functions. I think I found non-deprecated alternatives for both. Those were:
- Figure out how to get nginx to use openssl as a shared object.
- It'd be better if the
ssl_esnikeydirwere a "global" setting probably (like
error_log) but I need to figure out how to get that to work still. For now it seems it has to be inside the
httpstanza, and one occurrence of the setting causes
load_esnikeys()to be called three times in our test setup which seems a little off. (It's ok though as we only really store keys from different files.)