Add OPENSSL_NO_DLL_ATEXIT_HANDLERS to control whether to use atexit().

This fixes an issue with a mix of atexit() usage in DLL and statically linked
libcrypto that came out in the test suite on NonStop, which has slightly
different DLL unload processing semantics compared to Linux. The change
allows a build configuration to select whether to register OPENSSL_cleanup()
with atexit() or not, so avoid situations where atexit() registration causes
SIGSEGV.

Fixes: #23135

Signed-of-by: Randall S. Becker <randall.becker@nexbridge.ca>

Configurations/50-nonstop.conf

@@ -15,7 +15,8 @@
                                 '_XOPEN_SOURCE_EXTENDED=1',
                                 '_TANDEM_SOURCE',
                                 '__NSK_OPTIONAL_TYPES__',
-                                'B_ENDIAN'),
+                                'B_ENDIAN',
+                                'OPENSSL_NO_DLL_ATEXIT_HANDLERS=1'),
         perl             => '/usr/bin/perl',
         shared_target    => 'nonstop-shared',
         shared_extension => ".so",

NOTES-NONSTOP.md

@@ -57,7 +57,12 @@ relating to `atexit()` processing when a shared library is unloaded and when
 the program terminates. This limitation applies to all OpenSSL shared library
 components.
 
-A resolution to this situation is under investigation.
+A control has been added as of 3.3.x to disable, by default, calls to `atexit()`
+within the `libcrypto` builds (specifically in `crypto/init.c`). This variable,
+`OPENSSL_NO_DLL_ATEXIT_HANDLERS=1` is set by default for NonStop builds. If you
+need to have `atexit()` functionality, set `OPENSSL_NO_DLL_ATEXIT_HANDLERS=0` to
+enable the `atexit()` call to register `OPENSSL_cleanup()` automatically.
+Preferably, can explicitly call `OPENSSL_cleanup()` from your application.
 
 About Prefix and OpenSSLDir
 ---------------------------

crypto/init.c

@@ -97,17 +97,19 @@ static int win32atexit(void)
 
 DEFINE_RUN_ONCE_STATIC(ossl_init_register_atexit)
 {
-#ifdef OPENSSL_INIT_DEBUG
+#if ! defined(OPENSSL_NO_DLL_ATEXIT_HANDLERS) || !OPENSSL_NO_DLL_ATEXIT_HANDLERS
+# ifdef OPENSSL_INIT_DEBUG
     fprintf(stderr, "OPENSSL_INIT: ossl_init_register_atexit()\n");
-#endif
-#ifndef OPENSSL_SYS_UEFI
-# if defined(_WIN32) && !defined(__BORLANDC__)
+# endif
+# ifndef OPENSSL_SYS_UEFI
+#  if defined(_WIN32) && !defined(__BORLANDC__)
     /* We use _onexit() in preference because it gets called on DLL unload */
     if (_onexit(win32atexit) == NULL)
         return 0;
-# else
+#  else
     if (atexit(OPENSSL_cleanup) != 0)
         return 0;
+#  endif
 # endif
 #endif