feature request: add method to dlclose without unloading the library

[Pranav2612000]

Golang has issues with unloading of libraries ( golang/go#11100 ). This results in app crashes, ungraceful shutdowns and applications hanging during shutdown when go created libraries are dlclose d. As suggested here, the solution is to not unload the library during dlclose, but to keep it open. This is done using RTLD_NODELETE on unix and GET_MODULE_HANDLE_EX_FLAG_PIN for windows.

Due to the semantic differences in the way these flags are to be used, and libloading not providing functions to directly access the os-specific handle, the current solution to do this is something like this

        // By default, go builds the libraries with '-Wl -z nodelete' which does not
        // unload the go runtime. This isn't respected on mac ( https://github.com/golang/go/issues/11100#issuecomment-932638093 )
        // so we need to explicitly load the library with RTLD_NODELETE( which prevents unloading )
        #[cfg(unix)]
        let library: libloading::Library = unsafe {
            const RTLD_NODELETE: i32 = 0x80;

            libloading::os::unix::Library::open(
                Some(filename.as_ref()),
                libloading::os::unix::RTLD_LAZY | libloading::os::unix::RTLD_LOCAL | RTLD_NODELETE,
            )
            .map(Into::into)
            .map_err(libloading_error_to_adbc_error)?
        };
        // on windows, we emulate the same behaviour by using the GET_MODULE_HANDLE_EX_FLAG_PIN. The `.pin()`
        // function implements this.
        #[cfg(windows)]
        let library: libloading::Library = unsafe {
            let library: libloading::os::windows::Library =
                libloading::os::windows::Library::new(filename.as_ref())
                    .map_err(libloading_error_to_adbc_error)?;
            library.pin().map_err(libloading_error_to_adbc_error)?;
            library.into()
        };

PR for reference: apache/arrow-adbc#3844

We were hoping to get these changes merged here to help others who may also face the same issue. If the approach looks good, I'll be happy to do the changes and submit a PR. If you have other ideas on how to implement this, let's discuss those as well.

[nagisa]

What you're doing right now is the current way to go about this, yes.

It would be nice to have this present in the top-level API somehow but I'm reluctant to a plain constructor for this. Primarily because there are many other flags and options that people want to have (see e.g. #36 and #190 for hard to reconcile examples) and we can't keep adding constructors and getters for each possible combination. Exposing flags in the os level API has been a compromise that would allow most people to achieve what they need to do (even if the code turned out somewhat involved.)

In my mind std::fs::OpenOptions-type builder might be the most plausible way forward to configure how a library is opened, but I'm still not sure how to reconcile the problem of the fact that each target has its own mildly different options, flags and features; and the common subset of the features is clearly not sufficient for all the different use-cases where libloading is applied at.

[Pranav2612000]

Thank you for your response.

It would be nice to have this present in the top-level API somehow but I'm reluctant to a plain constructor for this. Primarily because there are many other flags and options that people want to have

Yes make sense. And I agree with you, it doesn't make sense to add constructors for each possible combination. I'm not sure what the best solution is ( or if there is one ) but here were some ideas I could think of to make this experience better

Approach I) Supporting something like fs::OpenOptions with custom OpenOptions ( not the OS specific ones ). For each custom OpenOption we then call the appropriate functions which provide the desired behaviour. For this case, the OpenOption could be NO_UNLOAD which sets the RTLD_NO_DELETE for linux and calls pin for windows

Approach II) Provide methods to directly access and modify the internal os specific Library ( Right now even though there's a pin function for windows, I can't call it if I load my library using Libloading directly.

What are your thoughts?

[nagisa]

Provide methods to directly access and modify the internal os specific Library. Right now even though there's a pin function for windows, I can't call it if I load my library using Libloading directly

But the snippet in the first comment is already doing exactly that, isn't it? You can convert between OS and top-level Library with the From implementations.

Regardless this wouldn't help unix much as pin-equivalent is quite a bit harder to implement. We'd have to track the drop flag on our own.

---

What are your thoughts?

I said in previous comment that I could see OpenOptions-style approach with extension traits but it needs design and consideration (what happens for conflicting options, how to handle options that are OS-specific, how to handle dlmopen question, what to do about dlvsym)