New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: dlopen/dlsym without CGo #18296

Open
iamacarpet opened this Issue Dec 13, 2016 · 9 comments

Comments

Projects
None yet
4 participants
@iamacarpet

iamacarpet commented Dec 13, 2016

Hello,

Would it be possible to implement dlopen/dlsym functionality to match the Windows DLL loading functionality, for Linux/*nix platforms?

It has been discussed that this is possible with CGo, but that doesn't play as well with a simple build and cross-compile environment as your pure-Go tool-chain does.

Discussed in golang-nuts: https://groups.google.com/forum/#!topic/golang-nuts/QDEczMhlQBU
As I'd like to make this library for SQLite without CGo (https://github.com/iamacarpet/go-sqlite3-win64) cross platform.

They suggested following the code path from one of the libc implementations and re-writing it into Go.
This for the moment is quite a way beyond my capabilities, so I'd really love some help.

Regards,
iamacarpet

@cznic

This comment has been minimized.

Contributor

cznic commented Dec 13, 2016

It wouldn't buy you much. It's not possible in the general case to call C from Go without CGO (or equivalent mechanism). IOW, the problem is not in dlopen/dlsym.

@iamacarpet

This comment has been minimized.

iamacarpet commented Dec 13, 2016

@cznic please excuse my ignorance, but could you elaborate on that for me?

Using the Windows "syscall.LazyDLL" and "syscall.LazyProc" constructs, I can attach and call shared libraries that are written in C, using pointers and the "unsafe" package, while manually translating the type differences.

Is the way shared library loading works on Linux and other non-Windows platforms fundamentally different and incompatible with this method?

@cznic

This comment has been minimized.

Contributor

cznic commented Dec 13, 2016

Heh, I am the ignorant one. I know next to nothing about Windows. On unix, one of the problems is the required stack switch. Without that the C routine will/can crash.

@iamacarpet

This comment has been minimized.

iamacarpet commented Dec 13, 2016

Thanks @cznic,

So I guess the question is, the stack switch you mentioned, is that already something that is required & has been overcome to allow it to work on Windows, or does the OS handle all that for us?

Please excuse me if I'm wrong, it's quite foreign to me, but I've tried to follow the Windows code path and it looks like this:

https://github.com/golang/sys/blob/master/windows/dll_windows.go (LoadDLL -> loadlibrary)
https://github.com/golang/sys/blob/master/windows/asm_windows_amd64.s (loadlibrary -> syscall.loadlibrary)
https://github.com/golang/go/blob/master/src/runtime/syscall_windows.go (syscall_loadlibrary -> cgocall)
https://github.com/golang/go/blob/master/src/runtime/cgocall.go (cgocall -> asmcgocall)
https://github.com/golang/go/blob/master/src/runtime/asm_amd64.s (asmcgocall).

I don't know enough to say if that is the required stack switch taking place, or if the example we have here is anything like what is required for a Linux/Unix implementation.

Although, it does look like they are using the same "cgocall" on Solaris too, so perhaps it's not so different?

@ianlancetaylor ianlancetaylor changed the title from dlopen/dlsym without CGo to runtime: dlopen/dlsym without CGo Dec 13, 2016

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Dec 13, 2016

The stack switch on Windows and Solaris takes place in the call to asmcgocall. It's done that way on those systems because there is no alternative. That approach is much slower than the syscall approach used on other Unix systems. Still, in theory it could be used with any dlopen implementation because we would force people to use a Call method along the lines of syscall.Proc.Call on Windows.

A bigger problem is that the dynamic linker, which implements the usual dlopen, is a complex program that is closely tied to the system C library. Making dlopen work without invoking the dynamic linker would require implementing precisely what the dynamic linker does, and that is highly system and version dependent. I don't see that as feasible.

On the plus side, this code does not actually have to live in the Go runtime. It needs some runtime support, but that is available using the unsafe package and the go:linkname magic comment. So the first step for anybody who wants to tackle this is to write it as a third party package. Good luck.

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Dec 13, 2016

@iamacarpet

This comment has been minimized.

iamacarpet commented Dec 13, 2016

Thanks @ianlancetaylor, that is very helpful!

I'll do some more research and see if it is something I can work on, but what you've said about the dynamic linker sounds like it would be a lot of work.

Is it possible that when you load the library, it could call the dynamic linker on it's own for it's dependancies? I was reading that the ELF header links to the dynamic linker the binary requires and if the requirement is to set up it's own environment, we wouldn't care how it's specific implementation handles it, would we?

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Dec 13, 2016

The ELF header of a dynamically linked executable refers to the dynamic linker, yes. But that is not true of a shared library. And since your driving desire is to be able to build programs without a C cross-compiler, you must be creating statically linked executables. So there isn't any place for you to look to find the dynamic linker.

But let's say you could find the real dynamic linker. That still wouldn't help, because the dynamic linker is designed to start the program, and in your case the program is already started. The dynamic linker doesn't have a way to run within a program that is already started. And since the dynamic linker is highly optimized for what it does, it doesn't have anything like the hooks you would need to make it act differently.

@iamacarpet

This comment has been minimized.

iamacarpet commented Dec 13, 2016

Thanks again @ianlancetaylor, you saved me a lot of hours of research before I would have come to that stumbling block myself.

I'll put it on the back burner for now and keep researching when I've the resources to invest.

@binarycrusader

This comment has been minimized.

Contributor

binarycrusader commented Jun 8, 2017

Realistically, it's impossible to support this on Solaris. On Solaris, the dynamic linker must be used; it's the only one that's sufficiently aware of system configuration and that supports the many types of relocations that might be needed. Any sort of attempt to workaround this is likely to end in tears, especially since on Solaris, libc must be used for "system calls".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment