support WINDOWS_EXPORT_ALL_SYMBOLS like cmake

[santagada]

CMake has a very useful property called WINDOWS_EXPORT_ALL_SYMBOLS: https://cmake.org/cmake/help/v3.9/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html

That uses a coff symbol dumper (cmake internal command) that creates .def files for .obj files exporting every symbol in them. That is the only simple way to generate complete shared libraries (.dll on windows) with MSC as it only links exported symbols (explicit exports or using this .def file).

Does meson already supports this? If not this is a feature request, I can help with providing a C++ tool to do the exports either by writting it myself or just repackaging the CMake code.

[guruDanny67]

gtkmm does this (also on the companion glibmm, pangomm, cairomm, ...) using a program named gendef, compiled from gendef.cpp in any of the projects. This one uses dumpbin.exe from the Visual Studio installation.

[nirbheek]

This is generally very bad practice to do, since it will export all symbols, which means you cannot enforce ABI/API stability. I don't think we should support this, especially since you can always keep a static definitions list if you really don't want to use __declspec(dllexport).

[santagada]

The use we have is to create shared libraries to internal applications that take many minutes to link. So we don't care about defining a ABI/API. IIRC that behavior is supported using GCC, its just MSC not even allowing you to use it.

[guruDanny67]

Maybe is enought to have a prelink and a postlink command so this function can be implemented by the user and can be useful also in another situations (for example if someone wants to digitally sign the executable / dll at the end of the link phase).

[rib]

Just to note that not having an equivalent feature can make it more tricky to build existing third-party libraries that use cmake and use WINDOWS_EXPORT_ALL_SYMBOLS, such as libflann.

Even though I agree it seems like poor practice to use this option; if it's a third party library which I don't maintain then I don't really want to be patching libflann to improve how it handles exports.

I think the reason libflann uses it is because they themselves are embedding a third-party lz4 decompression API in their library and expect it to be exported since their public headers then depend on the LZ4_ symbols. The lz4 implementation they embed doesn't have any kind of symbol export defines.

Seems like a very questionable design choice and I don't want to defend it - just adding a data point for considering whether there's some value in a feature like this, even if only for the sake of simplifying building/using existing projects currently using cmake, with minimal changes to an upstream project.

[nirbheek]

One workaround for such cases is to create a .def file from the import library outputted by cmake and use that when building with Meson. That means you have to update it when new symbols are added, but that's something you can automate.

[rib]

Thanks, yeah, maybe something to consider if I hit another project using WINDOWS_EXPORT_ALL_SYMBOLS. For me, in this particular case, it would be a bit awkward since I'm actually trying out cross-compiling from Linux to Windows using a cross-file based on clang -target x86_64-pc-windows -fms-compatibility... so I'm not really in a good position to run a normal windows + msvc cmake build :-)

I ended up resorting to patching libflann's embedded lz4 and lz4hc headers to add the necessary exports and, to be fair, that wasn't a huge amount of effort. I'll cross my fingers and hope the changes rebase cleanly when necessary, but since the libflann project doesn't seem to be actively developed I guess it'll be ok.

When considering automating I did see that that for my cross-compile case that llvm-nm -extern-only -defined-only *.obj seems like another possible way of dumping a list of the symbols that could probably be massaged into a .def EXPORTS list too.

[eli-schwartz]

Maybe is enought to have a prelink and a postlink command so this function can be implemented by the user and can be useful also in another situations (for example if someone wants to digitally sign the executable / dll at the end of the link phase).

When I first saw this issue, reading this comment made me think that e.g. dumpbin operates on the DLL, creates a def file, and then the idea is you relink the DLL using a def file generated from the first version of the DLL.

But now I'm not so sure that's actually needed? e.g. the llvm-nm suggestion is to glob the .obj files.

https://docs.microsoft.com/en-us/cpp/build/reference/dumpbin-reference?view=msvc-170 mentions:

You can use DUMPBIN to examine COFF object files, standard libraries of COFF objects, executable files, and dynamic-link libraries (DLLs).

But then https://docs.microsoft.com/en-us/cpp/build/reference/dash-exports?view=msvc-170 mentions:

This option displays all definitions exported from an executable file or DLL.

So it's not clear to me (from a Linux user's perspective) what options there are for automatically determining viable symbols of a DLL... before building the DLL.

I think modifying existing outputs is probably not an option on the table. But if there is some way to create the def file right before the link stage, that would be neat.

How does cmake handle this anyway?

Does meson already supports this? If not this is a feature request, I can help with providing a C++ tool to do the exports either by writting it myself or just repackaging the CMake code.

BTW: Meson is written in pure python, so generally the idea would be to write a python tool so that it can be packaged with Meson. Alternatively, if there is some way to do it with the toolchain that Microsoft is providing anyway, that's another possible route.

[dcbaker]

I think dumpbin /symbols might work

[neheb]

maybe dlltool -l . No idea. The GNU dlltool has more options but that won't fly with clang's MSYS2 backends.

dlltool -h
OVERVIEW: llvm-dlltool

USAGE: llvm-dlltool [options] file...

OPTIONS:
  -D <value> Specify the input DLL Name
  -d <value> Input .def File
  -f <value> Assembler Flags
  -k         Kill @n Symbol from export
  -l <value> Generate an import lib
  -m <value> Set target machine
  -S <value> Assembler

[TyBalduf]

Hopefully not the wrong place to ask, but its been mentioned here that including a .def file is at least a workaround for this issue. How do I get meson to find a .def file that I have generated? What options do I need to put in the meson.build or the commandline args in order for it to actually find the .def? I have tried to look around through the documentation and the code itself and haven't been able to come up with a clear answer. I imagine its fairly simple and its just my naivety about meson.

In an attempt to make some small projects MSVC compatible, I have been able to get away with passing the .def using meson configure build -Dc_link_args=-def:def/file/path.def after meson setup (to avoid the def file causing a failure in the compiler sanity check), but I realize this a super hacky way of doing things and I have run into case where even this doesn't work and it simply ignores the passed in def file.

[dcbaker]

meson has a vs_module_def keyword argument for the executable() or shared_library() in question. Generally you would either check the .def file in and pass it as a string executable(..., vs_module_def : 'mylib.def') or generate it with a custom_target and pass that output to the vs_module_def argument

[neheb]

@TyBalduf vs_module_def accepts custom_target, which is what's used to generate such a file. An example is in the recently commited libsndfile wrap.

[anarazel]

Maybe is enought to have a prelink and a postlink command so this function can be implemented by the user and can be useful also in another situations (for example if someone wants to digitally sign the executable / dll at the end of the link phase).

When I first saw this issue, reading this comment made me think that e.g. dumpbin operates on the DLL, creates a def file, and then the idea is you relink the DLL using a def file generated from the first version of the DLL.

But now I'm not so sure that's actually needed? e.g. the llvm-nm suggestion is to glob the .obj files.

https://docs.microsoft.com/en-us/cpp/build/reference/dumpbin-reference?view=msvc-170 mentions:

You can use DUMPBIN to examine COFF object files, standard libraries of COFF objects, executable files, and dynamic-link libraries (DLLs).

But then https://docs.microsoft.com/en-us/cpp/build/reference/dash-exports?view=msvc-170 mentions:

This option displays all definitions exported from an executable file or DLL.

So it's not clear to me (from a Linux user's perspective) what options there are for automatically determining viable symbols of a DLL... before building the DLL.

Yes, dumpbin /symbols on .objs works. Postgres has been using it for a long time to generate .def symbols for postgres itself (so it can dynamically load extension libraries).

Unfortunately the output of dumbin /symbols needs a bit of filtering. Here's what we have, fwiw. Quite possibly some of the filtering is just cargo-culting from the early aughts.
https://github.com/anarazel/postgres/blob/a2f0e678474234d4c6d7f88e1d533ee4d0d8b1de/src/tools/msvc/gendef.pl#L32

There's some potential issues with commandline lengths, depending on where you glob. For the meson port of postgres I'm now just passing a static library to the above tool to work around those, and then use the objects from the static library as executable(..., objects: ). Alternatively doing the globbing "in" dumpbin, rather than the shell, might also work around the problems.

[eli-schwartz]

Thanks, that's a useful example and I like the fact that we can just use dumpbin rather than writing a PE/COFF parser.

This should be feasible to reimplement in python and run internally, in between the *.obj generation and the final link.exe

[anarazel]

Thanks, that's a useful example and I like the fact that we can just use dumpbin rather than writing a PE/COFF parser.

Heh, yea, that doesn't sound like fun. And this isn't needed when building with mingw, so depending on dumpbin shouldn't be an issue.

This should be feasible to reimplement in python and run internally, in between the *.obj generation and the final link.exe

That would presumably provide most of the the infrastructure for the avoid-unnecessary-shared-library-relinking stuff?

Nobody here might care, for quite defensible reasons, but AIX needs pretty much the same, except that gcc doesn't generate the export/import files either.

[eli-schwartz]

That would presumably provide most of the the infrastructure for the avoid-unnecessary-shared-library-relinking stuff?

The def file (re)generation would only happen when the object files are rebuilt, so that should be fine, yeah.

Nobody here might care, for quite defensible reasons, but AIX needs pretty much the same, except that gcc doesn't generate the export/import files either.

That sounds like "patches welcome". :) For Windows stuff I'd just blind code it given examples and then check if CI passes...

[egorpugin]

FYI
There is old obj->symbols code that is used by cmake and some other projects. I use it too.
https://github.com/Kitware/CMake/blob/master/Source/bindexplib.h
https://github.com/Kitware/CMake/blob/master/Source/bindexplib.cxx

[eli-schwartz]

@anarazel any plans to implement something like gendef.pl in python code as part of meson and run it internally by meson as a build edge after building all object files but before linking them?

[anarazel]

@eli-schwartz

@anarazel any plans to implement something like gendef.pl in python code as part of meson and run it internally by meson as a build edge after building all object files but before linking them?

Unfortunately not in the near term. Perhaps @tristan957 is interested?

[tristan957]

I could probably port the script to python, but I don't know what "as a build edge after building all object files but before linking them" means.

Perl makes me :'(

[eli-schwartz]

It should output a .def file and use it as a link_depends, but the input is the built objects.

Normally:

• compile all .o files
• link all .o files into a library

Now:

• compile all .o files
• run a custom target
• link all .o files into a library

(This is impossible to do from the meson.build DSL since you don't have a reference to the .o files until you run the library() function, and then it's too late to define a custom_target() that gets used inside the previous library() function.)

[anarazel]

(This is impossible to do from the meson.build DSL since you don't have a reference to the .o files until you run the library() function, and then it's too late to define a custom_target() that gets used inside the previous library() function.)

Yea. Doing it from "userspace" is quite ugly. Because of this (and awful dtrace behaviour on macos - it has to modify .o files, yuck), we build a static library of all postgres backend code, generate the .def file from that, and only then build the actual executable. Which runs into annoying edge cases, e.g. (IIRC) msvc doesn't like building an executable with just a static library.

I have wondered if there's something the DSL could add to make things like this easier. Even if some WINDOWS_EXPORT_ALL_SYMBOLS were added, there are other tasks that want to run between building .o files and linking to an executable / library.

Perl makes me :'(

You're not alone in that...

[tristan957]

I have wondered if there's something the DSL could add to make things like this easier. Even if some WINDOWS_EXPORT_ALL_SYMBOLS were added, there are other tasks that want to run between building .o files and linking to an executable / library.

Maybe have a link boolean kwarg on executable()/library(), and then it is the job of the of the script to call build_tgt.link() before using it?

[mgautierfr]

Hi all,

I have (chatgpt) ported the script gendef.pl from @anarazel to python: https://gist.github.com/mgautierfr/11eb53379bf063f965614dcc9b6baef7

I have (quickly) tested it with one of my library and it seems to works well, I can build a dll (and unittest) without issues (I have compilation warning about "export of deleting destructor" LNK4102, we may have to filter them from .def).
I had to build first a static library first, then generate .def and then I was able to use the .def with vs_module_defs and build a dll.

It would be nice to integrate into meson. I will have no time on the next few months to do it (neither I know how to do it). Feel free to reuse the script to do it if you which to. If you don't I will have a look when I come back.

---

Copyright is @anarazel/postgres one. My personal contribution/translation is public domain (as far as it can be)

[eli-schwartz]

Chatgpt cannot be public domain. :(

[mgautierfr]

Chatgpt cannot be public domain. :(

Kind of off-topic but do you have source for this ?

---

Anyway, if I manually convert it to python from perl, are we ok or we are (I am) doomed because I have seen the result of chatgpt ?
From comment in perl script, and some inference of perl instruction, I could convert the script easily (it will take probably more time but, I hope, with a better result)

[eli-schwartz]

Kind of off-topic but do you have source for this ?

Its license is "derivative work, without informing the consumer what it is derived from". This applies to all LLMs regardless of type of output (code, essays, artwork...) and the basic unit of defense used by the companies building LLMs to justify the legality is that it falls under the Fair Use Doctrine because consuming copyrighted work is the only economically viable way to build an LLM. The industry is awash in copyright lawsuits because not everyone finds that argument convincing.

Anyway, if I manually convert it to python from perl, are we ok or we are (I am) doomed because I have seen the result of chatgpt ?

I assume you haven't done an in-depth study of the chatgpt code and therefore there is no taint and it's fine.

G-d knows I can't remember the details of code I only read one time a week ago, and would have to rewrite it from scratch anyway... to be on the safe side, you could do exactly that, and wait a week until you no longer remember what chatgpt showed you. :)