Build fails with OpenSSL 1.1

[neilstuartcraig]

Hi

I have a project in which I am doing a custom NGINX build and want to use this module. The build is an RPM build (currently on CentOS 7) and it's failing with NGINX (1.10.0) with OpenSSL 1.1-pre4 (pre5 doesn't compile with NGINX):

In file included from /tmp/tmp44TVHu/BUILD/nginx-1.10.0/openssl-1.1.0-pre4/.openssl/include/openssl/ssl.h:912:0,
                 from src/event/ngx_event_openssl.h:15,
                 from src/core/ngx_core.h:83,
                 from /tmp/tmp44TVHu/BUILD/nginx-1.10.0/lua-nginx-module/src/ddebug.h:13,
                 from /tmp/tmp44TVHu/BUILD/nginx-1.10.0/lua-nginx-module/src/ngx_http_lua_ssl_certby.c:10:
/tmp/tmp44TVHu/BUILD/nginx-1.10.0/lua-nginx-module/src/ngx_http_lua_ssl_certby.c: In function 'ngx_http_lua_ffi_ssl_get_tls1_version':
/tmp/tmp44TVHu/BUILD/nginx-1.10.0/openssl-1.1.0-pre4/.openssl/include/openssl/tls1.h:182:20: error: dereferencing pointer to incomplete type
                 ((s->version >> 8) == TLS1_VERSION_MAJOR ? s->version : 0)
                    ^
/tmp/tmp44TVHu/BUILD/nginx-1.10.0/lua-nginx-module/src/ngx_http_lua_ssl_certby.c:524:18: note: in expansion of macro 'TLS1_get_version'
     return (int) TLS1_get_version(ssl_conn);
                  ^
/tmp/tmp44TVHu/BUILD/nginx-1.10.0/openssl-1.1.0-pre4/.openssl/include/openssl/tls1.h:182:61: error: dereferencing pointer to incomplete type
                 ((s->version >> 8) == TLS1_VERSION_MAJOR ? s->version : 0)
                                                             ^
/tmp/tmp44TVHu/BUILD/nginx-1.10.0/lua-nginx-module/src/ngx_http_lua_ssl_certby.c:524:18: note: in expansion of macro 'TLS1_get_version'
     return (int) TLS1_get_version(ssl_conn);
                  ^
/tmp/tmp44TVHu/BUILD/nginx-1.10.0/lua-nginx-module/src/ngx_http_lua_ssl_certby.c:527:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
make[1]: *** [objs/addon/src/ngx_http_lua_ssl_certby.o] Error 1
make[1]: *** Waiting for unfinished jobs....
/tmp/tmp44TVHu/BUILD/nginx-1.10.0/lua-nginx-module/src/ngx_http_lua_ssl_ocsp.c: In function 'ngx_http_lua_ffi_ssl_set_ocsp_status_resp':
/tmp/tmp44TVHu/BUILD/nginx-1.10.0/lua-nginx-module/src/ngx_http_lua_ssl_ocsp.c:471:17: error: dereferencing pointer to incomplete type
     if (ssl_conn->tlsext_status_type == -1) {
                 ^
/tmp/tmp44TVHu/BUILD/nginx-1.10.0/lua-nginx-module/src/ngx_http_lua_ssl_ocsp.c:493:13: error: dereferencing pointer to incomplete type
     ssl_conn->tlsext_status_expected = 1;

It would be great to get OpenSSL 1.1 compatibility (especially as it'll be stable in a few weeks) but I absolutely understand it's a pre-release so it's 100% understandable that the build fails right now. This ticket is for 2 reasons:

1. See what the timeline is likely to be for OpenSSL 1.1 support
2. Record the above for anyone else who finds the same issue

Happy to provide any info and/or run tests builds if needed.

Thanks for the great work!

Cheers

[agentzh]

@neilstuartcraig Right, we haven't tried that yet ourselves. Thanks for the report. Patches welcome! :)

[neilstuartcraig]

Sorry for the slow reply @agentzh. I'd love to help but my C was never great and has got worse in the 15 years since I really did any :-).

Can I perhaps do some digging to help find the issue?

Also, do you have a rough idea when the module is likely to compile against openssl 1.1 - i.e. how high it is on your priority scale. I believe openssl 1.1 is going to be stable this month and should therefore be shipped with OS's and updates within a few weeks.

Cheers

[ghedo]

All those errors are due to the fact that the SSL struct in OpenSSL 1.1 is going to be opaque.

The TLS1_get_version() errors are caused by an OpenSSL bug, I opened openssl/openssl#1038 to fix it.

The other ones I'm not sure how to fix. A new API is probably required to get/set those fields.

[neilstuartcraig]

Thanks @ghedo - i see the tests passed on your PR.
If i can be any help, please let me know.

[ghedo]

So, the other problem is that the OCSP stapling support in ngx_lua is kind of hacky. I made openssl/openssl#1054 but it's not enough to make this work and I think we might want to find a better solution (e.g. instead of calling SSL_set_tlsext_status_ocsp_resp in ngx_http_lua_ffi_ssl_set_ocsp_status_resp directly, we might want to save the OCSP response somewhere and define a SSL_CTX_set_tlsext_status_cb callback that retrieves that response and sets it properly).

There are also other changes needed to make this build with OpenSSL master, I'll open a PR for that soon. I also have a patch to make NGINX build with OpenSSL master (but I haven't sent it to nginx-devel yet).

[ghedo]

@neilstuartcraig could you post the NGINX build errors you got with OpenSSL 1.1.0-pre5? (just to confirm the errors I fixed are the same).

[ghedo]

Actually, I don't think we ven need the ssl_conn->tlsext_status_expected = 1 part at all, since the ngx_http_lua_ssl_empty_status_callback callback returns SSL_TLSEXT_ERR_OK which makes OpenSSL set the flag by itself. So the two OpenSSL PRs I made should be enough to make OpenResty OCSP code build and work (but it will need a change to call the new OpenSSL function instead of accessing the SSL tlsext_status_type field directly, assuming my PR is merged).

[ghedo]

Not tested though...

[neilstuartcraig]

@ghedo - thanks so much for picking this up and doing it so quickly, we very much appreciate it.

The build errors I see (RPM build via spec file on CentOS7) are:

src/event/ngx_event_openssl.c:954:11: error: dereferencing pointer to incomplete type
         dh->p = BN_bin2bn(dh1024_p, sizeof(dh1024_p), NULL);
           ^
src/event/ngx_event_openssl.c:955:11: error: dereferencing pointer to incomplete type
         dh->g = BN_bin2bn(dh1024_g, sizeof(dh1024_g), NULL);
           ^
src/event/ngx_event_openssl.c:957:15: error: dereferencing pointer to incomplete type
         if (dh->p == NULL || dh->g == NULL) {
               ^
src/event/ngx_event_openssl.c:957:32: error: dereferencing pointer to incomplete type
         if (dh->p == NULL || dh->g == NULL) {

Is that enough info? ICIH, openssl-pre4 builds just fine for nginx (without the openresty lua module).

Let me know if I can be useful in any way - testing, raising tickets, getting debug info etc. and thanks again!

BTW, not sure if you know but we're using openresty in the A/V chain for BBC online now - there are a couple of blog posts e.g. this one and this one (for 'nginx' read 'openresty') - not my team (thought they're about 15 metres away from me) but i'll extend my thanks and appreciation from them too!

[ghedo]

Yes, those errors are the same I see. The NGINX developers said they are going to remove that whole DH-related code in 1.11.0, so it should get fixed in a few days (so my patch that fixed that code was rejected).

[neilstuartcraig]

Excellent! Thanks for the update @ghedo

[ghedo]

I opened #761 to fix some of the build errors, but since changes to OpenSSL itself are also required, it doesn't fix all the errors yet.

[neilstuartcraig]

Thanks for the update @ghedo :-)

[centminmod]

OpenSSL 1.1.0 final is out :)

[neilstuartcraig]

wOOt! Hopefully no repeat work needed then :-)

Sent from my iPhone

On 25 Aug 2016, at 18:45, George Liu (eva2000) notifications@github.com wrote:

OpenSSL 1.1.0 final is out :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[ctrochalakis]

Since v0.10.6 (building as a dyn module for 1.11.4) I am getting the following failure when building against openssl 1.1.0:

/<<PKGBUILDDIR>>/debian/modules/nginx-lua/src/ngx_http_lua_ssl_session_storeby.c:252:33: error: dereferencing pointer to incomplete type 'SSL_SESSION {aka struct ssl_session_st}'
     cctx->session_id.data = sess->session_id;
                                 ^~

[centminmod]

any progress on openssl 1.1.0+ support ? only thing missing for me to switch from libressl 2.4/openssl 1.0.2j to openssl 1.1.0

[agentzh]

@centminmod OpenSSL 1.1 has both big architectural changes and also some API changes. So it will take a relatively long time on our side to port and test.

[centminmod]

@agentzh ok understood :)

[neilstuartcraig]

Hi @agentzh
Just wondered whether there's any update on progress or plans WRT openssl 1.1 yet? i'm keen to be able to adopt TLS1.3 once it's rolled in to openssl (i'm assuming openssl 1.1 contains the majority of architectural changes to the API).

Cheers :-)

[agentzh]

@neilstuartcraig OpenSSL 1.1.0 has many more quirks than we originally expected. You can watch and help in #761 and #922.

[agentzh]

@neilstuartcraig BTW, our current focus is on OpenSSL 1.1.0 while the closest OpenSSL release that may get TLS 1.3 looks like to be 1.1.1, which is a different series than 1.1.0, so it may have its own even newer incompatibles and quirks :)

[neilstuartcraig]

Thanks @agentzh - i'll check them out, i am not sure i can help much in development as my C is rustier than anyone could imagine and wasn't good to start with! Very happy to help testing and/or with docs etc. though. i'll subscribe to the issues and keep track as best i can.

[challengeof]

hi @agentzh ,I use version is openresty-1.11.2.2 , openssl 1.0.2 build still is error :

command ：
1、./configure --prefix=/usr/local/openresty \
            --with-luajit \
            --without-http_redis2_module \
            --with-http_iconv_module
2、make -j4
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f objs/Makefile
cc -c -I/Users/challenge/source/openresty-1.11.2.2/build/luajit-root/usr/local/openresty/luajit/include/luajit-2.1  -pipe  -O -Wall -Wextra -Wpointer-arith -Wconditional-uninitialized -Wno-unused-parameter -Wno-deprecated-declarations -g -O2 -DNDK_SET_VAR -DNDK_SET_VAR -DNDK_UPSTREAM_LIST -DNDK_SET_VAR -DNDK_SET_VAR -DNDK_SET_VAR -DLUA_DEFAULT_PATH='"/usr/local/openresty/site/lualib/?.lua;/usr/local/openresty/site/lualib/?/init.lua;/usr/local/openresty/lualib/?.lua;/usr/local/openresty/lualib/?/init.lua"' -DLUA_DEFAULT_CPATH='"/usr/local/openresty/site/lualib/?.so;/usr/local/openresty/lualib/?.so"' -DNDK_SET_VAR -I src/core -I src/event -I src/event/modules -I src/os/unix -I ../ngx_devel_kit-0.3.0/objs -I objs/addon/ndk -I ../ngx_lua-0.10.7/src/api -I /usr/local/include -I objs \
		-o objs/src/event/ngx_event_openssl.o \
		src/event/ngx_event_openssl.c
src/event/ngx_event_openssl.c:2048:21: error: use of undeclared identifier 'SSL_R_NO_CIPHERS_PASSED'
            || n == SSL_R_NO_CIPHERS_PASSED                          /*  182 */
                    ^
1 error generated.
make[2]: *** [objs/src/event/ngx_event_openssl.o] Error 1
make[1]: *** [build] Error 2
make: *** [all] Error 2
command:
openssl version
OpenSSL 1.0.2k  26 Jan 2017

[agentzh]

@challengeof OpenSSL 1.0.2 is proven to work for years. The error may be an indicator you may not really use the correct versions of header files of OpenSSL. You should ensure that you are indeed using OpenSSL 1.0.2's headers.

[neilstuartcraig]

Hi again
Time for my irregular checking of the status...how are things looking for 1.1 support? Any progress?

JFYI (apologies if you know this) but Debian 9 will be shipping very soon and comes with 1.1.
Also, i had a quick catch up with a colleague who's on various working groups which use and feed into openssl - 1.1 is intended to be the major change, 1.1.1 (i am told this will include TLS1.3) is planned to be API-compatible with 1.1.

Hope that helps. Any news would be gratefully received (i wish i could contribute code) :-)

[agentzh]

@thehowl Mind to provide the C and Lua backtraces for that error? It should be an easy fix.

[thehowl]

Sure, what's needed to do that? An environment flag? Running openresty by itself doesn't print any stacktrace, btw, it only prints the error message I posted, and quits. Also, do you want me to create a separate issue?

@spacewander might give it a shot later, thanks!

[agentzh]

@thehowl You just need debug symbols and gdb for that. If you compile openresty from source then you already have debug symbols by default (unless you strip the binaries). And in gdb, you can use bt full to get the C land backtrace and use lbt full to get the Lua land backtrace. See

https://github.com/openresty/openresty-gdb-utils#lbt

Just do the core dump or process inspection in gdb when the worker process crashes.

[thehowl]

After a lot of trying to understand how to get the backtrace because I never barely even touched gdb before, I managed to get a C backtrace (using a catchpoint on exit)

#0  0x0000ffff80316aac in __GI__exit (status=status@entry=1) at ../sysdeps/unix/sysv/linux/_exit.c:31
#1  0x0000ffff802ad300 in __run_exit_handlers (status=status@entry=1, listp=0xffff803be4c8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:98
#2  0x0000ffff802ad384 in __GI_exit (status=status@entry=1) at exit.c:105
#3  0x0000ffff8055eeb4 in err_unwind (stopcf=0x0, errcode=2, L=0x41a2bd468378) at lj_err.c:179
#4  lj_err_throw (L=L@entry=0x41a2bd468378, errcode=errcode@entry=2) at lj_err.c:529
#5  0x0000ffff8055f1bc in lj_err_run (L=L@entry=0x41a2bd468378) at lj_err.c:623
#6  0x0000ffff8055f2d8 in err_msgv (L=0x41a2bd468378, em=em@entry=LJ_ERR_BADLU) at lj_err.c:636
#7  0x0000ffff8055f2e0 in lj_err_msg (L=<optimized out>, em=em@entry=LJ_ERR_BADLU) at lj_err.c:642
#8  0x0000ffff8056b958 in lua_pushlightuserdata (L=L@entry=0x41a2bd468378, p=<optimized out>) at lj_api.c:699
#9  0x0000aaaad4563e64 in ngx_http_lua_init_registry (log=0xaaaad4797d18 <ngx_log>, L=0x41a2bd468378) at ../ngx_lua-0.10.10/src/ngx_http_lua_util.c:660
#10 ngx_http_lua_new_state (log=0xaaaad4797d18 <ngx_log>, lmcf=0xaaaae088cb00, cycle=0xaaaae08827f0, parent_vm=0x0) at ../ngx_lua-0.10.10/src/ngx_http_lua_util.c:298
#11 ngx_http_lua_init_vm (parent_vm=parent_vm@entry=0x0, cycle=0xaaaae08827f0, pool=<optimized out>, lmcf=lmcf@entry=0xaaaae088cb00, log=0xaaaad4797d18 <ngx_log>, pcln=0x464e4f43, pcln@entry=0x0)
    at ../ngx_lua-0.10.10/src/ngx_http_lua_util.c:3749
#12 0x0000aaaad455adf0 in ngx_http_lua_init (cf=0xffffe044f5a8) at ../ngx_lua-0.10.10/src/ngx_http_lua_module.c:744
#13 0x0000aaaad44c7e90 in ngx_http_block (cf=0xffffe044f5a8, cmd=<optimized out>, conf=<optimized out>) at src/http/ngx_http.c:312
#14 0x0000aaaad44a6124 in ngx_conf_handler (last=1, cf=0xffffe044f5a8) at src/core/ngx_conf_file.c:427
#15 ngx_conf_parse (cf=cf@entry=0xffffe044f5a8, filename=0x3e9000003e9, filename@entry=0xaaaae08829b8) at src/core/ngx_conf_file.c:283
#16 0x0000aaaad44a3f94 in ngx_init_cycle (old_cycle=0xffffe044f7b8) at src/core/ngx_cycle.c:274
#17 0x0000aaaad4491b24 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:276

Trying to use lbt gives this

Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x48: 
Error occurred in Python command: Cannot access memory at address 0x48

[agentzh]

@thehowl Please use the gdb command bt full insteadof bt in that context. Thanks!

[agentzh]

@thehowl Also, will you re-compile your openresty with the ./configure flag --with-cc-opt="-O0"? So that we can see all the parameter and local variable values. Right now many of them are optimized away (i.e., those <optimized out> placeholders in your gdb output). Thanks!

[agentzh]

@thehowl BTW, your crash is due to LuaJIT's GC64 mode's limitations on lightuserdata value ranges. It has nothing to do with OpenSSL at any rate.

[thehowl]

@agentzh Yep, I was aware that OpenSSL was unrelated (guess I should have pointed that out!). What I said was mostly a tip for googlers finding this thread in hope to managing to compile from source (either for leisure or, like me, necessity), pointing out that even after managing to compile with openssl, the program wouldn't work because of the issue I pointed out.

bt full from an -O0 compile

[agentzh]

@thehowl Then you shouldn't hijack this github issue (the title is "Build fails with OpenSSL 1.1").

[agentzh]

@thehowl Anyway, will you try the following gdb command in that gdb session?

p &ngx_http_lua_coroutines_key

[agentzh]

@thehowl Next time please create a separate github issue to avoid hijacking existing unrelated issues. Thank you for your cooperation.

[agentzh]

@thehowl Seems like on ARM64, the C global variables are stored in the high address space like stacks (but we still need the output of the gdb command p &ngx_http_lua_coroutines_key on your side to be sure, as requested above).

In that case, the following patch should fix the problem:

https://github.com/openresty/lua-nginx-module/pull/1151/files

Will you try it out on your side? It should be something like this:

cd /path/to/openresty-1.11.2.5/
cd bundle/ngx_lua-0.10.10/
wget https://github.com/openresty/lua-nginx-module/pull/1151.diff
patch -p1 < 1151.diff
cd ../..
./configure -j9 ...
make -j9
sudo make install

Thanks for the report!

[agentzh]

@thehowl BTW, it's recommended to build OpenSSL as a shared library otherwise the linker would strip away OpenSSL symbols that are not directly referenced in the nginx binary, which may cause errors when you try to use dynamic Lua modules referencing new symbols in OpenSSL.

[thehowl]

Sorry for the late reply, and I understand I should have made a new issue (as I've even asked you if I should do some replies ago - though I'm at fault for not properly pointing out that the issue was unrelated in my first post, sorry about that!). Either way, getting to the issue at hand.

The print on gdb returned the following:

$1 = 0xaaaab492fc04 <ngx_http_lua_coroutines_key> ""

I will now apply the patch you linked and try doing as you said. Thanks for the advice for openssl!

[agentzh]

@neilstuartcraig OK, the C global variables are indeed stored in high address space on ARM64, going beyond the 47-bit address space. So it is confirmed. My patch above should fix the problem on your side. Let me know how it works.

[dndx]

@thehowl Yep it seems to be a known issue due to LuaJIT's implementation. Attempting to address addresses higher than 0x7fffffffffff used to cause LuaJIT to crash, therefore a check was added to throw an error instead.

If you have time, mind sharing the output of:

cat /proc/[PID]/maps

on your architecture? You can use the NGINX process as the [PID] if you can freeze it with GDB while executing the command, but any other process on your system should offer same insight into the virtual memory layout on your system for us.

[thehowl]

@agentzh Still winds up in the same error, but this time with a different line to cause it. As you can see from the backtrace, frame #9, there is still one call not wrapped in push_47bit_lightud, which is to say this

@dndx Sure! I'm not sure if by "nginx" you actually meant openresty or the actual nginx I'm running right now, so just to be sure I included both. https://gist.github.com/thehowl/0b25712130ff0691cfec50e71c66bcef

[thehowl]

I went ahead and replaced the function call specified in the previous comment, and at last it, at the very least, starts listening to 0.0.0.0:80 with the default config and it gives the "Welcome to OpenResty" page. (although ack lua_pushlightuserdata | wc -l gives 81, so I'm assuming there are more references to lua_pushlightuserdata to be replaced and that it's not ready to run scripts yet)

[dndx]

@thehowl Thanks for the info. Looks like ARM64 Linux indeed maps program image to a relatively high starting address (0xaaaaaaaaa000).

Since the entire program image resides inside 0xaaaaaaaaa000 - 0xaaaaaaf0b000, masking with 0x7fffffffffff as @agentzh proposed is completely safe as the 48th bit is always 1 and only the lower 47 bits changes.

[agentzh]

Ah, this is problematic. @thehowl Can you change the starting point of the address mapping? We cannot mask off normal pointers like ngx_cycle, that would lead to bad pointers when we need them (and we have to amend the pointer values every time we use them).

[agentzh]

If this is the mapping used for all Arm64 systems, then we could amend for the pointer values in an ad-hoc way. But still, it's very hacky. The easiest way is build OpenResty as 32-bit binaries, as @spacewander suggested earlier. Or we should simply patch LuaJIT for this specific address mapping in this particular system/architecture.

[agentzh]

Let's move this OT discussion to #1152. It's deeper than expected.

[neilstuartcraig]

@agentzh - My apologies, i missed your message above. I have just tried the patch above (1151.diff) and it definitely fixes some but not quite all issues. The remaining thing i can see is an error during build of:

/root/modules/lua-nginx-module-0.10.11/src/ngx_http_lua_module.c: In function 'ngx_http_lua_merge_srv_conf':
/root/modules/lua-nginx-module-0.10.11/src/ngx_http_lua_module.c:1022:37: warning: passing argument 2 of 'SSL_CTX_sess_set_get_cb' from incompatible pointer type [enabled by default]
                                     ngx_http_lua_ssl_sess_fetch_handler);
                                     ^
In file included from src/event/ngx_event_openssl.h:15:0,
                 from src/core/ngx_core.h:82,
                 from /root/modules/lua-nginx-module-0.10.11/src/ddebug.h:13,
                 from /root/modules/lua-nginx-module-0.10.11/src/ngx_http_lua_module.c:11:
/root/modules/openssl-1.1.0g/.openssl/include/openssl/ssl.h:639:6: note: expected 'struct SSL_SESSION * (*)(struct ssl_st *, const unsigned char *, int,  int *)' but argument is of type 'struct SSL_SESSION * (*)(struct SSL *, u_char *, int,  int *)'
 void SSL_CTX_sess_set_get_cb(SSL_CTX *ctx,
      ^

This is with:

nginx -V

nginx version: nginx/1.13.7
built with OpenSSL 1.1.0g
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/default-error.log --http-log-path=/var/log/nginx/default-access.log --lock-path=/var/run/nginx.lock --http-proxy-temp-path=/var/cache/nginx/proxy_temp --user=nobody --group=nobody --with-http_realip_module --with-http_v2_module --with-http_ssl_module --with-openssl=/root/modules/openssl-1.0.2m --with-http_geoip_module --with-pcre-jit --with-file-aio --with-threads --add-module=/root/modules/headers-more-nginx-module-0.33 --add-module=/root/modules/nginx-module-vts-0.1.15 --add-module=/root/modules/ngx_devel_kit-0.3.0 --add-module=/root/modules/naxsi-0.55.3/naxsi_src --add-module=/root/modules/lua-nginx-module-0.10.11 --with-cc-opt=

Any thoughts would be very welcome and I can easily try different builds now as i have a parameterised, automated build.

Cheers

[agentzh]

@neilstuartcraig I don't think OpenSSL 1.1.x is supported by this module yet. See #761 and #922.

[agentzh]

Closing this since it's duplicate with #761 and #922.