runtime: dlopen/dlsym without CGo

[iamacarpet]

Hello,

Would it be possible to implement dlopen/dlsym functionality to match the Windows DLL loading functionality, for Linux/*nix platforms?

It has been discussed that this is possible with CGo, but that doesn't play as well with a simple build and cross-compile environment as your pure-Go tool-chain does.

Discussed in golang-nuts: https://groups.google.com/forum/#!topic/golang-nuts/QDEczMhlQBU
As I'd like to make this library for SQLite without CGo (https://github.com/iamacarpet/go-sqlite3-win64) cross platform.

They suggested following the code path from one of the libc implementations and re-writing it into Go.
This for the moment is quite a way beyond my capabilities, so I'd really love some help.

Regards,
iamacarpet

[cznic]

It wouldn't buy you much. It's not possible in the general case to call C from Go without CGO (or equivalent mechanism). IOW, the problem is not in dlopen/dlsym.

[iamacarpet]

@cznic please excuse my ignorance, but could you elaborate on that for me?

Using the Windows "syscall.LazyDLL" and "syscall.LazyProc" constructs, I can attach and call shared libraries that are written in C, using pointers and the "unsafe" package, while manually translating the type differences.

Is the way shared library loading works on Linux and other non-Windows platforms fundamentally different and incompatible with this method?

[cznic]

Heh, I am the ignorant one. I know next to nothing about Windows. On unix, one of the problems is the required stack switch. Without that the C routine will/can crash.

[iamacarpet]

Thanks @cznic,

So I guess the question is, the stack switch you mentioned, is that already something that is required & has been overcome to allow it to work on Windows, or does the OS handle all that for us?

Please excuse me if I'm wrong, it's quite foreign to me, but I've tried to follow the Windows code path and it looks like this:

https://github.com/golang/sys/blob/master/windows/dll_windows.go (LoadDLL -> loadlibrary)
https://github.com/golang/sys/blob/master/windows/asm_windows_amd64.s (loadlibrary -> syscall.loadlibrary)
https://github.com/golang/go/blob/master/src/runtime/syscall_windows.go (syscall_loadlibrary -> cgocall)
https://github.com/golang/go/blob/master/src/runtime/cgocall.go (cgocall -> asmcgocall)
https://github.com/golang/go/blob/master/src/runtime/asm_amd64.s (asmcgocall).

I don't know enough to say if that is the required stack switch taking place, or if the example we have here is anything like what is required for a Linux/Unix implementation.

Although, it does look like they are using the same "cgocall" on Solaris too, so perhaps it's not so different?

[ianlancetaylor]

The stack switch on Windows and Solaris takes place in the call to asmcgocall. It's done that way on those systems because there is no alternative. That approach is much slower than the syscall approach used on other Unix systems. Still, in theory it could be used with any dlopen implementation because we would force people to use a Call method along the lines of syscall.Proc.Call on Windows.

A bigger problem is that the dynamic linker, which implements the usual dlopen, is a complex program that is closely tied to the system C library. Making dlopen work without invoking the dynamic linker would require implementing precisely what the dynamic linker does, and that is highly system and version dependent. I don't see that as feasible.

On the plus side, this code does not actually have to live in the Go runtime. It needs some runtime support, but that is available using the unsafe package and the go:linkname magic comment. So the first step for anybody who wants to tackle this is to write it as a third party package. Good luck.

[iamacarpet]

Thanks @ianlancetaylor, that is very helpful!

I'll do some more research and see if it is something I can work on, but what you've said about the dynamic linker sounds like it would be a lot of work.

Is it possible that when you load the library, it could call the dynamic linker on it's own for it's dependancies? I was reading that the ELF header links to the dynamic linker the binary requires and if the requirement is to set up it's own environment, we wouldn't care how it's specific implementation handles it, would we?

[ianlancetaylor]

The ELF header of a dynamically linked executable refers to the dynamic linker, yes. But that is not true of a shared library. And since your driving desire is to be able to build programs without a C cross-compiler, you must be creating statically linked executables. So there isn't any place for you to look to find the dynamic linker.

But let's say you could find the real dynamic linker. That still wouldn't help, because the dynamic linker is designed to start the program, and in your case the program is already started. The dynamic linker doesn't have a way to run within a program that is already started. And since the dynamic linker is highly optimized for what it does, it doesn't have anything like the hooks you would need to make it act differently.

[iamacarpet]

Thanks again @ianlancetaylor, you saved me a lot of hours of research before I would have come to that stumbling block myself.

I'll put it on the back burner for now and keep researching when I've the resources to invest.

[binarycrusader]

Realistically, it's impossible to support this on Solaris. On Solaris, the dynamic linker must be used; it's the only one that's sufficiently aware of system configuration and that supports the many types of relocations that might be needed. Any sort of attempt to workaround this is likely to end in tears, especially since on Solaris, libc must be used for "system calls".

[notti]

After implementing a pcap version for windows in gopacket (=call into libpcap) that doesn't require cgo, I thought it would be nice to have that in linux too (gopacket often gets crosscompilation questions...).

Well I landed here and thought: This can't be that complicated - or can it?

Since it was mentioned above as a starting point, I had a look at the dynamic linker and also came to the conclusion, that reimplementing that one is a no-starter. But we actually we want to get rid of cgo - not the dynamic linker.

go can already help us there: //go:cgo_import_dynamic (I didn't know at the time and wrote my own relinker that could also do that...). So getting dlopen and dlsym is simple.

Next up calling C functions (including dlopen and dlsym):
asmcgocall already works for that, but has two issues:

1. It can only pass one argument and returning values is limited
2. Thread Local Storage (TLS) is not available and one runs into issues very quickly (already printf("%f", 1) needs that under glibc).

Ok so I found out one can convince the go runtime not touching TLS by providing something in _cgo_init (//go:linkname is our friend here). But that only works for thread 1... So I found out that we could also set runtime.is_cgo to true and implement _cgo_thread_start, _cgo_notify_runtime_init_done, _cgo_setenv, and _cgo_unsetenv. Should be simple to do - but cgo implements those in C.

To solve this chicken and egg problem I created trampoline functions in go assembly that convert the C calling conventions to go calling conventions (and vice versa), loaded all the necessary C functions via dynamic symbols and also wrote wrappers for those (so we can call C from go), like in the solaris and darwin implementation in the runtime. => I could reimplement these functions in golang.

So now the only thing that's missing is to provide some kind of libffi functionality for calling arbitrary functions. For that I created a mechanism where one can specify arguments, argument types and return via a struct.

Since writing everything several times is cumbersome, I created a proof of concept library including full description and everything under https://github.com/notti/nocgo/

Everything is carefully split into parts that should make implementing further architectures and OSes simple with sharing most of the code.

Working stuff

• Loading libraries
• Calling functions (integer, floating point, and pointers as arguments and return values; no structs, complex types, and callbacks)
• Accessing global variables
• amd64 support
• 386 support
• Linux (glibc) support
• FreeBSD support (Sadly this doesn't work fully outside the runtime, since FreeBSD needs two exported symbols, which is not allowed outside cgo and the stdlib - for now compiling freebsd stuff works by adding -gcflags=github.com/notti/nocgo/fakecgo=-std to the build process)
• cgo support (yeah this also works when cgo is there)

Missing stuff

• OS support (import right symbols and maybe some tweaks)
• Arch support (just implement the calling convention)

Stuff I don't like

• I'm not completely happy with the way arguments are specified - but I couldn't come up with something better
• The internal call specification is not written very good (some stuff could be moved from assembly to golang)
• Being in the runtime would help with lot's of stuff (the _cgo-symbol implementations are not allowed to use writebarriers and outside the runtime go:nowritebarrierrec is not allowed)

TLDR

I created a proof of concept at: https://github.com/notti/nocgo/

What do you guys think about this solution? Is this viable? Suggestions? Could have something like this or parts of it chances of getting into the runtime?