Fix filenames and paths used in DLL shared library generation

[nirbheek]

• DLLs should always go into bindir
• If import libraries are created, those should go into libdir
• DLLs should never contain soversion or version in the filename
• Only .so files should have aliases generated for them (dylibs and DLLs do not use aliases)

[jpakkane]

This change breaks two things:

• installing C# libraries on Linux fails because they end in .dll and this change causes the installer to try to use a file name with .so suffix
• installing libraries on OSX fails (test 27), probably because the suffix is 'dylib' and this breaks this change somehow

[nirbheek]

I guess we need some way to find out what platform/compiler/language the SharedLibrary target is targetting. Perhaps by implementing them as child types or a set of variables that indicate the target platform and current compiler/language being used, or some other method. Checking for the current OS is not enough because one might be cross-compiling with cross-mingw on Linux, for instance (we do this in GStreamer).

I actually got a distinct feeling in various parts of the code that platform-specific things were difficult to do because such information wasn't available in a consistent/well-layered manner. Am I reading things incorrectly?

For instance, in this patch, calling detect_c_compiler() felt really hacky to me since we're not necessarily compiling with a C compiler, and library suffix/prefix is really specific to the target, not the environment.

[jpakkane]

The environment info is not easily accessible because cross compilation was not a part of the design from day 1. The reason for this was that it was not obvious how it should be designed (and also to avoid doing a big design up front).

There are many places in the code that check for platform stuff in a hacky way but who really should be querying properties of host_machine.

And unfortunately library suffix and the like actually do come from the environment (and others). For example the object file suffix is .obj on Windows. Except for MinGW where it is .o. And when cross compiling from Linux to Win it might be either one of the two.

[nirbheek]

That's sort of what I mean by the suffix/prefix/etc being specific to the target (which should know what compiler/host/target is being used) rather than the environment (which is used for the entire project). For instance, plenty of projects compile C/C++ code and also bindings that use them, such as Java or C#.

Do you have thoughts on how to go about improving this? I've been trying to think of a re-structuring that would work, but I don't know nearly enough about the code to come up with something. :(

Often when I do something platform-specific in Meson, the hacks required kill me a little inside, but I'm sure we can fix that!

[jpakkane]

I guess the obvious first step is to open a bug and try to enumerate how things should work. Something like this:

• C source, linux build, linux host -> .so
• C source, linux build, win host -> .dll
• C# source, any build -> .dll

And so on. Especially the parts that currently go wrong.

[nirbheek]

I've made a list of what build outputs should be and where they should go. I couldn't find a better place to put it, so here it is in comment form:

Library names and installation directories

NOTE: <- means a symbolic link of the form: target <- symlink

C/C++/Asm source

Shared library, name 'foo', version 'X[.Y[.Z]]'

host	prefix	suffix	install_dir	filenames
linux	'lib'	'.so'	libdir	libfoo.so.X.Y.Z <- libfoo.so.X <- libfoo.so
android	'lib'	'.so'	libdir	libfoo.so
osx	'lib'	'.dylib'	libdir	libfoo.X.dylib
mingw	'lib'	'.dll'	bindir	libfoo-X.dll
msvc	''	'.dll'	bindir	foo-X.dll

iOS does not support shared libraries

OS X:

• If there's no version, the filename is libfoo.dylib
• dylib internal versioning should be done with clang -current_version X.Y.Z; see also: -compatibility_version
• There are limits on the range of versioning: X is [0, 65535], Y and Z are [0, 255] (inclusive)

Linux:

• Only the major version 'X' (also stored as the DT_SONAME tag inside the library) is used for versioning dependencies
• The -soname,libfoo.so.X param to ld is used for writing the DT_SONAME tag
• If only a major version is specified, the only filenames are libfoo.so.X <- libfoo.so
• If no versioning is specified, the filenname is libfoo.so

Debug symbol file, name 'foo'

host	prefix	suffix	install_dir
msvc	''	'.pdb'	bindir
*	???	???	???

.pdb files must be installed in the same directory as the executable or shared library

Import library, name 'foo'

host	prefix	suffix	install_dir
mingw	'lib'	'.dll.a'	libdir
msvc	''	'.lib'	libdir

Static library, name 'foo'

host	prefix	suffix	install_dir
linux	'lib'	'.a'	libdir
android	'lib'	'.a'	libdir
ios	'lib'	'.a'	libdir
osx	'lib'	'.a'	libdir
mingw	'lib'	'.a'	libdir
msvc	'lib'	'.a'	libdir

MSVC convention for static vs import libraries has not been consistent in the past, but starting with MSVC 2015, the new CRT follows this convention: foo.lib is import library, libfoo.lib is static library

However, GCC does not understand this convention and will not find a file named libfoo.lib if given the argument -lfoo. Since MSVC does not care what the static library is called (it has to be specified on the command line anyway), we've decided to use libfoo.a for all platforms.

People who want to use an alternative naming scheme can use the name_prefix and name_suffix keyword arguments to change lib and a to something else.

Executable, name 'foo'

host	suffix	install_dir
mingw	'.exe'	bindir
msvc	'.exe'	bindir
*	''	bindir

C# source

Shared library, name 'foo', version 'X[.Y[.Z]]'

host	prefix	suffix	install_dir	filenames
*	''	'.dll'	libdir	foo.dll

Note: The actual installation directory is usually a subdir of libdir, and the subdir is distro-specific. Probably best to have it default to libdir.

Executable, name 'foo'

host	suffix	install_dir
win	'.exe'	bindir
*	'.exe'	libdir

Note: The actual installation directory on !Windows is usually a subdir of libdir, and the subdir is OS-specific. Probably best to have it default to libdir.

Debug symbol file, name 'foo' (no versioning)

host	prefix	suffix	install_dir
win	'lib'	'.pdb'	libdir
*	???	???	???

Note: Debug symbol files location for Windows is from the gtk-sharp installer provided by upstream (must be same as library being debugged). For others, it is unknown.

[LRN]

Shared library, name 'foo', version 'X[.Y[.Z]]'

host	prefix	suffix	install_dir	filenames
mingw	'lib'	'.dll'	bindir	libfoo-X.Y.Z.dll <- libfoo-X.dll <- libfoo.dll

That is, technically, the "correct" way (see below for why it's not immediately useful). However, most projects use the libfoo-X.dll form only, as this is how libtool names DLLs by default.

DLLs normally go into bindir, but plugins are usually installed into libdir/name-of-the-application-or-library-the-plugin-is-for/ (the application is responsible for finding the plugins there). IIRC, meson allows install_dir to be changed by the user, so this should be OK.

It is not stressed in the comments above, but import libraries are not versioned in any way, and i know of no practical way to install multiple versions of an import library (from different versions of a -dev package). I wonder how various GNU distributions handle this...

The name of the DLL that a binary links to comes from the import library (if an import library is used; in case of direct DLL linking, it comes from the DLL file itself, probably). Anyway, if the import library libfoo.dll.a contains the name libfoo-3.dll, then linking to that import library does, in effect, link to libfoo-3.dll. The name that an import library contains seems to depend entirely on what -o option was. That is, gcc ... -Wl,--out-implib=libfoo.dll.a -o libfoo-1.dll will make an implib libfoo.dll.a that links to libfoo-1.dll.

This is why having real libfoo-X.Y.Z.dll and less-specifically-versioned symlinks libfoo-X.dll and libfoo.dll is not practical - programs linked with -lfoo will use the implib, and implib is hardcoded to link to the DLL it was made for, which in this case would be libfoo-X.Y.Z.dll. To link to libfoo-X.dll one would need to actually repeat the linking step multiple times, producing a real libfoo[-X[.Y[.Z]]].dll shared libs and libfooSOMETHING.dll.a implibs (SOMETHING to ensure that they don't overwrite each other), then deleting all DLLs except one, replacing them with symlinks. Projects would still need to specify -lfooSOMETHING to link to the more narrowly-versioned library. All that sounds really crazy and no one in their right mind would do this (and indeed, i haven't seen anyone doing this).

So yes, libfoo-X.dll is what 99.9% of all libtool-using projects have on Windows.

CMake-built DLLs and implibs use anything, but can be coerced into using libfoo-X.dll and libfoo.dll.a by specifying some special arguments (at least 80% of existing CMake-using projects do not specify these special arguments).

Niche and custom buildsystems use arbitrary naming (usually fooX.dll or foo.dll).

MinGW does not use soname. If that option is provided, ld will silently ignore it.

Direct linking to DLLs without an import library is supported, but:

1. Either bindir (where DLLs are) must be in -L path, or libfoo.dll.a files must be symlinks to their respective DLLs
2. See the numbered list at the end of the section (the link above) for reasons why import libraries might be used instead of direct DLL linkage
3. I vaguely remember some problem in replacing all .dll.a files with links to DLLs, but i can't remember what the problem was (maybe autotools tests didn't work correctly? Or some tooling assumed that .dll.a files are actually ARchives, not PE files? Can't remember).

It is possible to produce an import library for existing DLL (i.e. without the necessity of --out-implib at link time), but it requires a .def file. Such .def file can be generated from a DLL too, or written by hand. That said, this process can have many gotchas and exotic corner-cases that i am not aware of (i used this trick only for %1 of all packages i've made).

Debug symbol file, name 'foo'

host	method	prefix	suffix	install_dir
mingw	debug-link	*	*	*
mingw	build-id	*	*	*

MinGW-gcc and MinGW-gdb seem to have full support for separate debug info files
I have used debug-link method (and can confirm that it works) with the /foo/bar/libfoo-X.dll -> /foo/bar/libfoo-X.dll.dbg convention myself (objcopy and gdb have some undocumented kinks, at least on Windows, which makes generating separate debug info files harder than it should be, but it's still doable), but if documentation is to be believed, debuginfo files can be located in a specially-designated global directory too.
build-id method seems to be working as well (my gcc-4.9.2 does not recognize --build-id option, but ld does, when passed via -Wl,), although i have absolutely no experience in using it and did not check that mingw-gdb picks up the debuginfo files successfully.

Either way, the name of the debuginfo file is baked into a shared library, and is either arbitrary (debug-link) or random (build-id; unless you specify a particular build-id via --build-id=0x...).

[nirbheek]

Thanks for the detailed comment LRN! Most of what you've said matches my research so I'm glad to have independent verification for all this. I'll just add a few comments here:

DLL naming

So yes, libfoo-X.dll is what 99.9% of all libtool-using projects have on Windows.

You are correct, and this has also been my experience with almost all projects built with MinGW including gstreamer (libgstreamer-1.0.so.0 vs libgstreamer-1.0-0.dll) and glib (libglib-2.0.so.0 vs libglib-2.0-0.dll). I agree that we should replicate this naming in Meson but only when built with MinGW.

When building with MSVC, we should stick with the foo.dll syntax (gstreamer-1.0.dll, glib-2.0.dll, etc) and we should write the versioning inside the DLL itself as Microsoft recommends. I don't think this affects linking in any way since you're supposed to always use the import library for linking on Windows. The fact that linking directly to DLLs is supported by GCC is just a convenience thing. It doesn't work with MSVC anyway and is too difficult to special-case support in a clean way for GCC considering that they're not installed in the library path.

I also think that Meson should always generate both .lib and .dll.a import libraries while targetting Windows.

• MSVC always creates foo.lib import libraries and we should use implib if it is available (or perhaps add our own reimplementation of this) to generate the libfoo.dll.a import library so that GCC can link to MSVC DLLs.
• GCC can create libfoo.dll.a import libraries if requested (as you say), so we should generate the foo.lib MSVC import library so that MSVC can link to GCC DLLs.

We do both these in Cerbero (on top of Meson) while building GStreamer, and I have both GCC linking to MSVC libraries and vice-versa working flawlessly.

I'm not entirely sure how to handle static libraries yet (since there's a name-collision there with import libraries), but we can figure that out. I have some ideas.

Debugging symbol files on Windows

I think for now we should keep things simple and just support split-debug files in the same path as the DLLs themselves. So:

$prefix/bin/libfoo-X.dll & $prefix/bin/libfoo-X.dll.dbg for GCC (same for executables)

$prefix/bin/foo.dll & $prefix/bin/foo.dll.pdb for MSVC (same for executables)
(I have patches for this that I need to clean up and submit—depends on this PR being cleaned and merged first).

I'm not sure if we should give users the option of splitting the debug information out or not. With MSVC pdb syntax splitting is required and I don't know of any drawbacks of always splitting the debug info out.

[LRN]

DLL naming

GCC can create libfoo.dll.a import libraries if requested (as you say), so we should generate the foo.lib MSVC import library so that MSVC can link to GCC DLLs.

Generating foo.lib requires link.exe from MS toolchain. Usually (in GTK+, for example) this is done only if link.exe is available (there's a configure check that looks for link.exe). The reverse is also true (you need some parts of GNU toolchain to generate libfoo.dll.a), but is less of an issue, as GNU toolchain is free software.

I'm not entirely sure how to handle static libraries yet (since there's a name-collision there with import libraries), but we can figure that out. I have some ideas.

You've stated earlier that for MinGW static libraries are named libfoo.a, while import libraries are libfoo.dll.a. You've also stated that for MSVC there is a convention that static libraries are foo.lib and import libraries are libfoo.lib. I see no collisions here.

Debugging symbol files on Windows

and I don't know of any drawbacks of always splitting the debug info out

Neither do i.

I think for now we should keep things simple and just support split-debug files in the same path as the DLLs themselves

I think build-id is, potentially, a better way, since it allows one to just dump debug info into some global directory without worrying about collisions. Package maintainers will appreciate that. That said, you need a modern version of gdb (7.11 works, 7.9 doesn't). Anyway, one only needs to slip -Wl,--build-id[=...] option to the linker, which will cause .buildid section to appear in the file. Afterwards, one just needs to
objdump -p binaryfile | grep "format RSDS signature" | sed -e 's/.format RSDS signature \([a-z0-9]\+\) .\+/\1/'
to obtain the signature, then
objcopy --only-keep-debug binaryfile .build-id/XX/ZZ..ZZ.debug
(where XX is first two characters (one hex byte) of the signature and ZZ..ZZ is the rest of the signature) and
objcopy --strip-debug binaryfile
at which point one would just need to move the .build-id/XX/ZZ..ZZ.debug file into ${prefix}/lib/debug/.build-id/XX/ZZ..ZZ.debug.

Now, meson could be taught to automate that process, but should it? If so, how much?
Probably the ideal way is to:

1. See if there's a .buildid section (or detect that there's a --build-id argument being passed to the linker; looking for the section is foolproof though, as --build-id could be slipped via some means that meson has no control over).
2. If it exists, do the build-id dance as outlined above, generating the .build-id/XX/ZZ..ZZ.debug file and (during install phase) installing it into ${prefix}/lib/debug (the directory probably being overridable somehow).
3. Otherwise do the debug-link dance as outlined in the bug-report above (or do the shorter version of the same dance, which will still work, but will result in bogus warning from gdb; though i haven't tested it on more modern versions, maybe it's been fixed?) up to the point where .dbg file is generated next to the .exe/.dll. If package maintainer wants to move the file into ${prefix}/lib/debug/absolute-path-to-file, then package maintainer can do so manually (also, it seems highly impractical on Windows, where absolute-path-to-file is subject to relocation; i'm not sure gdb applies the same relocation rules to that path as to other directories it uses, although i haven't checked).

This does the "right thing".

If meson supports customizing debuginfo filename (i read MSDN, it seems that it does allow .pdb files to be arbitrarily-named), this facility can be used to give custom names to .pdb (MSVC) and .dbg (MinGW debug-link) files. Otherwise, establish some convention (xxx.(exe|dll).pdb and xxx.(exe|dll).dbg seem OK).

If some of the behaviour outlined above wouldn't be implemented, ensure that the part that is implemented is also disableable (i.e. if one wants to make use of build-id debuginfo separation by manually post-processing binaries, and meson support automating that by itself, one should be able to tell meson to just leave binaries with debuginfo and don't do anything to them).

[nirbheek]

Generating foo.lib requires link.exe from MS toolchain

This is not true. You can use dlltool that's shipped with MinGW to do this. We already do this in Cerbero for GStreamer Windows binaries. We also generate .dll.a using dlltool in Cerbero when we only have a .lib import library.

You've stated earlier that for MinGW static libraries are named libfoo.a, while import libraries are libfoo.dll.a. You've also stated that for MSVC there is a convention that static libraries are foo.lib and import libraries are libfoo.lib. I see no collisions here.

The convention is new and most people don't follow it yet, and it's not clear whether they will. The old convention was to have static libraries also be called foo.lib and to just never build both at the same time.

However I think we should go with the new convention anyway.

Debugging symbol files on Windows

All of this is very pie-in-the-sky stuff. We don't even properly generate .pdb files on Windows right now which means we have no debugging symbols at all (I have patches for this that I need to clean up).

Let's take it step by step. Generate .pdb files, install .pdb files on Windows, then add support for splitting out to .dbg with MinGW and on UNIX, then everything else. I'm sure Meson will support all this once more pressing issues are fixed. ;)

[jpakkane]

Can you repoke this so Travis will run the tests? Thanks.

[nirbheek]

I'm in the process of reworking this somewhat. Will re-push when I'm done.

[nirbheek]

The latest changes in this branch implement most of what was discussed above. What's missing is:

1. Debug file handling. I have some patches for this in my branch, but they need to be cleaned up. The implementation is very ugly and incomplete.

2. Generating .dll.a import libraries when building with the MSVC toolchain using dlltool (if available), and .lib import libraries when building with MinGW using dlltool or lib.exe (if and whichever is available).

Lots of the tests fail, but many of them fail because they are wrong. I'll be going through them one by one and fixing them. Will comment again when everything is green.

[nirbheek]

Well. After a long journey, all tests pass, and a bunch of additional tests have also been added to test the naming scheme. I think we can finally get this merged now.

[jpakkane]

raise AssertionError('BUG: Tried to link to something that's not a library')

All of these should be one of Meson internal exception types instead.

Shared library version checking is probably clearer if done with regex rather than split + int. Something like fullmatch r'\d+(\.\d+){0, 2}'?

get_import_filename_gen(self) mentions dlltool but it is not referenced anywhere else. Please clarify what this means.

Not a fan of @save_result. It seems very finicky and error prone. If the targets need compiler info to determine their stuff maybe we should pass that in as arguments then. Or determine the values from the outside and pass those as the arguments?

It seems that the tests do not check the case of installing shared libraries from a subproject. We probably don't need one for all platforms, linuxlike should be sufficient.

Fortran shared library test fails with MinGW.

MSVC dll versioning test fails with MinGW because no-installed-files is missing.

Test 6 fails in run_tests.py line 368 with mixed operands (int and str) with Visual Studio 2010 backend. Ninja works.

[nirbheek]

raise AssertionError('BUG: Tried to link to something that's not a library')

All of these should be one of Meson internal exception types instead.

Do we have a way of "asserting" via this? I used AssertionError because I wanted to encourage users to report those errors as bugs. They're supposed to never happen, so I'm treating those as literal asserts. Seeing the word "BUG" makes people report bugs. ;)

Shared library version checking is probably clearer if done with regex rather than split + int. Something like fullmatch r'\d+(.\d+){0, 2}'?

True. Fixed!

get_import_filename_gen(self) mentions dlltool but it is not referenced anywhere else. Please clarify what this means.

This was supposed to be part of a feature to build .dll.a files with dlltool when using the MSVC toolchain (which only generates .lib files) and .lib files with dlltool when using the MinGW toolchain, but I decided to get this working first since that's just a "nice-to-have" feature. I've removed the offending code.

Not a fan of @save_result. It seems very finicky and error prone. If the targets need compiler info to determine their stuff maybe we should pass that in as arguments then. Or determine the values from the outside and pass those as the arguments?

I did try to implement that first, but it was very invasive, so I thought I'd get this in first and then try to work on that. The @save_result stuff is not really needed; it's just an optimization so I've removed it for now. We can look into fixing this properly next.

It seems that the tests do not check the case of installing shared libraries from a subproject. We probably don't need one for all platforms, linuxlike should be sufficient.

New test added.

Fortran shared library test fails with MinGW.

I'll look into this. Could you share the error that you get?

MSVC dll versioning test fails with MinGW because no-installed-files is missing.

Fixing this.

Test 6 fails in run_tests.py line 368 with mixed operands (int and str) with Visual Studio 2010 backend. Ninja works.

This was fixed in d61656d. I've rebased the branch and you should see the actual test failures now.

[nirbheek]

Fortran shared library test fails with MinGW.

This should be fixed now. I wasn't able to test this because my MinGW Fortran toolchain is broken, so please test it if you can.

MSVC dll versioning test fails with MinGW because no-installed-files is missing.

This is fixed now.

I've added several patches that fix bugs in the vs backends. Some were pre-existing, others were because of the DLL versioning patches in this patchset. I've included them all in here and they're all already being tested by our test suite.

There is still one test that fails with the vs backends: common/113 generatorcustom. This is a pre-existing bug (I didn't touch that code at all), and I'm not sure how to fix it, so I've abandoned it for now. Will file a bug for this and poke @nioncode about it.

I have found more bugs in the vs backends that I will be posting in a new PR with tests for that.

[nirbheek]

I've just posted those commits here instead since they fix some bugs due to changes in this patchset, and also fix pre-existing bugs related to the directory structure and it's hard to separate the two.

[jpakkane]

No installed files still seems to be failing. I added a no-installed-files to test 8 and put this in meson.build:

install_data('no-installed-files', install_dir : '')

This passes run_tests even though it should not. Other than that this is looking pretty good.

[nirbheek]

No installed files still seems to be failing. I added a no-installed-files to test 8 and put this in meson.build:
install_data('no-installed-files', install_dir : '')
This passes run_tests even though it should not. Other than that this is looking pretty good.

I've fixed this with the last two commits. I have a test for the same pending in my tree, but that depends on fixing failing tests to also check for failure during install and such. Perhaps we can add a separate set of tests for failing compile, failing check, and failing install later.

[0x1997]

This PR broke libdir in set in project().
e.g. project('mylib', 'cpp', default_options: ['libdir=lib'])
The target will always be install to lib64 dir on 64-bit Linux.

[0x1997]

Line 515, should be build.StaticLibrary.

[0x1997]

A quick fix 0x1997@a875520

[nirbheek]

That's interesting. I think that fix only masks the problem. Going to look into this and will also add a test for it.

[0x1997]

@nirbheek Environment is initialized before Interpreter, then the interpreter processes the project() call, and update coredata in parse_default_options(). So when Environment is initialized, libdir in coredata is the default value.
This is just a quick dirty fix. The result of get_libdir() can be cached after the first access.

[nirbheek]

Yes, that's what I mean by the fix only masking the problem. We should probably initialize Environment after all options have been processed.

[kusma]

I know I'm extremely late to the party here, but shouldn't this have consulted host_machine instead of target_machine? The docs indicate that target_machine is only for compiler-type projects, but I currently need to set it to get Visual Studio to cross-compile anything at all...