Segfault when trying to call ExDataFreeOcspResponse after .NET module has been unloaded

[ulrichb]

Description

TL;DR: In EnsureOpenSsl11Initialized(), free_func pointers are registered which get dangling after the .NET module has been unloaded. When OpenSSL later tries to call this free_func, we get a segfault. It seems that CRYPTO_free_ex_index() calls are missing.

We have the following process/module scenario:

• OpenSSL is the entrypoint (openssl cms ... command with -engine pkcs11).
• => The libp11 OpenSSL engine (pkcs11.so) gets dynamically loaded by OpenSSL.
• By configuration, libp11 dynamically loads our PKCS#11 module (libSignPath.Cryptoki.so)
  • This is a Native AOT .NET 9 library => CLR gets initialized and eventually runs EnsureOpenSsl11Initialized().
• (Note that the following segmentation fault can only reproduced with recent OpenSSL and libp11 versions, e.g. on Debian 13 with OpenSSL 3.5.4 and libp11 0.4.13.)

The segmentation fault happens during unloading (when OpenSSL's cms_main calls CMS_ContentInfo_free()).

Full stack trace (linked OpenSSL is 3.5.4)

func                        file                            addr
??                                                          0x7acfe4f0dc70   
CRYPTO_free_ex_data         crypto/ex_data.c:406            0x7acfe6695813  
x509_cb                     crypto/x509/x_x509.c:85         0x7acfe6858665  
ossl_asn1_item_embed_free   crypto/asn1/tasn_fre.c:117      0x7acfe6520842  
ossl_asn1_template_free     crypto/asn1/tasn_fre.c:146      0x7acfe6520980  
ossl_asn1_item_embed_free   crypto/asn1/tasn_fre.c:70       0x7acfe652064f  
ossl_asn1_template_free     crypto/asn1/tasn_fre.c:141      0x7acfe6520923  
ossl_asn1_item_embed_free   crypto/asn1/tasn_fre.c:114      0x7acfe6520805  
ossl_asn1_template_free     crypto/asn1/tasn_fre.c:146      0x7acfe6520980  
ossl_asn1_item_embed_free   crypto/asn1/tasn_fre.c:114      0x7acfe6520805  
ASN1_item_free              crypto/asn1/tasn_fre.c:20       0x7acfe652046a  
CMS_ContentInfo_free        crypto/cms/cms_lib.c:27         0x7acfe659d29b  
cms_main                    apps/cms.c:1320                 0x5fb2c34d66ea  
do_cmd                      apps/openssl.c:428              0x5fb2c34f0c50  
main                        apps/openssl.c:309              0x5fb2c34f0812  

When the segfault happens, the mentioned .NET module libSignPath.Cryptoki.so already got unloaded (confirmed via LD_DEBUG=libs tracing).

On the stack trace, OpenSSL is in the process of freeing X.509 objects, which it used in the openssl cms command. On these X.509 objects, .NET attached OpenSSL "exdata" (registered at EnsureOpenSsl11Initialized() (using class index CRYPTO_EX_INDEX_X509). .NET registers ExDataFreeOcspResponse() as free_func.

Now OpenSSL (in the process of freeing X.509 objects) tries to call the registered free_func, which got unloaded. => Segfault.

We confirmed this by remembering the free_func pointer in ossl_crypto_get_ex_new_index_ex() for ExDataFreeOcspResponse() and seeing that the segfault happens due to accessing exactly this address.

So the problem is that .NET does not unregister the registered free_funcs. That is possible via CRYPTO_free_ex_index(), which is also recommended in the OpenSSL docs "if a dynamic library can be unloaded":

If a dynamic library can be unloaded, it should call CRYPTO_free_ex_index() when this is done. This will replace the callbacks with no-ops so that applications don't crash. Any existing exdata will be leaked.

Reproduction Steps

See above for a description of the necessary steps. Unfortunately we don't have a minimal reproduction sample.

Expected behavior

No segfault 😆

Actual behavior

See above.

Regression?

No response

Known Workarounds

A (very) hacky workaround is to run CRYPTO_free_ex_index(3, 1 /* index may vary! */); CRYPTO_free_ex_index(2, 1 /* index may vary! */); from the .NET process before module unloading.

Configuration

.NET 9 (AOT) in Debian 13 Linux container, AMD64.

Other information

No response

[jkotas]

So the problem is that .NET does not unregister the registered free_funcs. That is possible via CRYPTO_free_ex_index(), which is also recommended in the OpenSSL docs "if a dynamic library can be unloaded":

.NET runtime does not support OpenSSL being unloaded.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[ulrichb]

.NET runtime does not support OpenSSL being unloaded.

That's totally understandable. Here it's the other way round: the (AOT compiled) .NET module gets unloaded (OpenSSL still loaded and later running the dangling free_func).

[jkotas]

.NET runtime (including binaries published with NativeAOT) does not support unloading. #64629 and linked issues have discussion about it.

[ulrichb]

Okay, I understand that CoreCLR doesn't fully/officially support (shared library) unloading (i.e. dlclose()).

In our case -- where a .NET AOT library is unloaded indirectly -- we unfortunately have no control about the unloading behavior.

Unloading works in all our integration test cases, except the openssl cms scenario described above b/c of order of unloading, producing this segfault.

So I'd ask whether we could add "partial unloading support" by addressing this very specific issue by calling CRYPTO_free_ex_index() before unloading as suggested in the OpenSSL docs.

The reason I'm asking is that it's very difficult to fix this issue "from the outside" b/c it's hard to get the g_x509_ocsp_index value.

[jkotas]

Unloading works in all our integration test cases, except the openssl cms scenario described above b/c of order of unloading,

I expect that you are going to see other types of crashes once you get this one out of the way. Fixing them all is very difficult.

[ulrichb]

Indeed this is the only issue (in our scenario). We proofed this by applying the CRYPTO_free_ex_index()-workaround mentioned above. (When applying the workaround shutdown runs w/o any failures).