[ssl-cert-by-lua] Cipher mismatch

[greggilbert]

Hey. Apologies in advance for creating this ticket, but I've been racking my brain.

I'm trying to serve a self-signed certificate through nginx, but I keep getting an ssl_error_no_cypher_overlap error in Firefox (ERR_SSL_VERSION_OR_CIPHER_MISMATCH in Chrome).

Here's some of what I've got in nginx.conf:

lua_package_path "/tmp/lua-resty-core-0.1.1/lib/?.lua;/tmp/ngx_lua/lua/?.lua;lua/?.lua;../lua-resty-core/lib/?.lua;;";
lua_ssl_trusted_certificate "/vagrant/cert-v2/rootCA.pem";

server {
    listen       443 ssl;
    server_name  _;

    ssl on;

    ssl_certificate cert/nginx.crt;
    ssl_certificate_key cert/nginx.key;

    ssl_protocols        TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    ssl_prefer_server_ciphers  on;

    ssl_certificate_by_lua '
      local ssl = require "ngx.ssl";
      ssl.clear_certs();

      local common_name = ssl.server_name()
      if common_name == nil then
          common_name = "unknown"
      end
      local key_data = nil;


      local f = io.open("/vagrant/cert-v2/device.crt", "r")
      local cert_data = f:read("*a")
      f:close()

      local ok, err = ssl.cert_pem_to_der(cert_data);
      if not ok then
          ngx.log(ngx.ERR, "failed to set DER cert: ", err)
          return
      end

      local f = io.open("/vagrant/cert-v2/device.der", "r")
      local pkey_data = f:read("*a")
      f:close()

      local ok, err = ssl.set_der_priv_key(pkey_data);
      if not ok then
          ngx.log(ngx.ERR, "failed to set DER priv: ", err)
          return
      end

      ngx.log(ngx.ERR, "HELLO THIS ACTUALLY WORKED!")
    ';
}

Certs were generated by doing the following:

openssl genrsa -out rootCA.key 2048
openssl req -x509 -new -nodes -key rootCA.key -days 1024 -out rootCA.pem
openssl genrsa -out device.key 2048
openssl req -new -key device.key -out device.csr
openssl x509 -req -in device.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out device.crt -days 500
openssl rsa -in device.key -outform DER -out device.der

If I comment out the whole Lua block and link directly to /vagrant/cert-v2/device.crt and /vagrant/cert-v2/device.key, it works fine.

Within Vagrant, I get this:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ openssl s_client -connect 127.0.0.1:443
CONNECTED(00000003)
140157124298400:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:

---
no peer certificate available

---
No client certificate CA names sent

---
SSL handshake has read 7 bytes and written 317 bytes

---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated

---

Any ideas? Thanks!

[agentzh]

@greggilbert It does not seem to be related to ngx_lua nor ssl_certificate_by_lua. So it's off-topic I'm afraid. Try commenting out your own ssl_ciphers configurations and see if it works.

[greggilbert]

@agentzh Hmm, okay. At a glance, though, is the approach correct and in line with how the module is supposed to function? Like, wiping the ssl object, adding data via cert_pem_to_der and set_der_priv_key, and then that's it? Or is there some other part to it?

[agentzh]

@greggilbert I believe it's not related. The cipher settings are a separate thing. If it makes a difference, then it's a bug in OpenSSL.

[agentzh]

@greggilbert My hunch is that you just use the wrong values in the ssl_ciphers directive. But I could be wrong :)

[greggilbert]

@agentzh So I went back and forth on this a thousand times, and eventually came up with something that works. Turns out I was converting the cert from PEM to DER, but not setting it:

cert_data, err = ssl.cert_pem_to_der(cert_data)
if not cert_data then
    ngx.log(ngx.ERR, "failed to convert pem cert to der cert: ", err)
    return
end
local ok, err = ssl.set_der_cert(cert_data)
if not ok then
    ngx.log(ngx.ERR, "failed to set DER cert: ", err)
    return
end

So with that in, we're good. I'm gonna go rest my eyes now...

[agentzh]

@greggilbert Glad to hear that you've figured it out :) Yeah, such things happen :)