runtime: dlopen/dlsym without CGo #18296
Would it be possible to implement dlopen/dlsym functionality to match the Windows DLL loading functionality, for Linux/*nix platforms?
It has been discussed that this is possible with CGo, but that doesn't play as well with a simple build and cross-compile environment as your pure-Go tool-chain does.
Discussed in golang-nuts: https://groups.google.com/forum/#!topic/golang-nuts/QDEczMhlQBU
They suggested following the code path from one of the libc implementations and re-writing it into Go.
@cznic please excuse my ignorance, but could you elaborate on that for me?
Using the Windows "syscall.LazyDLL" and "syscall.LazyProc" constructs, I can attach and call shared libraries that are written in C, using pointers and the "unsafe" package, while manually translating the type differences.
Is the way shared library loading works on Linux and other non-Windows platforms fundamentally different and incompatible with this method?
So I guess the question is, the stack switch you mentioned, is that already something that is required & has been overcome to allow it to work on Windows, or does the OS handle all that for us?
Please excuse me if I'm wrong, it's quite foreign to me, but I've tried to follow the Windows code path and it looks like this:
https://github.com/golang/sys/blob/master/windows/dll_windows.go (LoadDLL -> loadlibrary)
I don't know enough to say if that is the required stack switch taking place, or if the example we have here is anything like what is required for a Linux/Unix implementation.
Although, it does look like they are using the same "cgocall" on Solaris too, so perhaps it's not so different?
The stack switch on Windows and Solaris takes place in the call to
A bigger problem is that the dynamic linker, which implements the usual
On the plus side, this code does not actually have to live in the Go runtime. It needs some runtime support, but that is available using the unsafe package and the
Thanks @ianlancetaylor, that is very helpful!
I'll do some more research and see if it is something I can work on, but what you've said about the dynamic linker sounds like it would be a lot of work.
Is it possible that when you load the library, it could call the dynamic linker on it's own for it's dependancies? I was reading that the ELF header links to the dynamic linker the binary requires and if the requirement is to set up it's own environment, we wouldn't care how it's specific implementation handles it, would we?
The ELF header of a dynamically linked executable refers to the dynamic linker, yes. But that is not true of a shared library. And since your driving desire is to be able to build programs without a C cross-compiler, you must be creating statically linked executables. So there isn't any place for you to look to find the dynamic linker.
But let's say you could find the real dynamic linker. That still wouldn't help, because the dynamic linker is designed to start the program, and in your case the program is already started. The dynamic linker doesn't have a way to run within a program that is already started. And since the dynamic linker is highly optimized for what it does, it doesn't have anything like the hooks you would need to make it act differently.
Realistically, it's impossible to support this on Solaris. On Solaris, the dynamic linker must be used; it's the only one that's sufficiently aware of system configuration and that supports the many types of relocations that might be needed. Any sort of attempt to workaround this is likely to end in tears, especially since on Solaris, libc must be used for "system calls".
After implementing a pcap version for windows in gopacket (=call into libpcap) that doesn't require cgo, I thought it would be nice to have that in linux too (gopacket often gets crosscompilation questions...).
Well I landed here and thought: This can't be that complicated - or can it?
Since it was mentioned above as a starting point, I had a look at the dynamic linker and also came to the conclusion, that reimplementing that one is a no-starter. But we actually we want to get rid of cgo - not the dynamic linker.
go can already help us there:
Next up calling C functions (including
Ok so I found out one can convince the go runtime not touching TLS by providing something in
To solve this chicken and egg problem I created trampoline functions in go assembly that convert the C calling conventions to go calling conventions (and vice versa), loaded all the necessary C functions via dynamic symbols and also wrote wrappers for those (so we can call C from go), like in the solaris and darwin implementation in the runtime. => I could reimplement these functions in golang.
So now the only thing that's missing is to provide some kind of
Since writing everything several times is cumbersome, I created a proof of concept library including full description and everything under https://github.com/notti/nocgo/
Everything is carefully split into parts that should make implementing further architectures and OSes simple with sharing most of the code.
Stuff I don't like
I created a proof of concept at: https://github.com/notti/nocgo/
What do you guys think about this solution? Is this viable? Suggestions? Could have something like this or parts of it chances of getting into the runtime?