New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't want platform libraries to be included when linking statically to a dependency #2765

Open
ocrete opened this Issue Dec 11, 2017 · 11 comments

Comments

Projects
None yet
4 participants
@ocrete

ocrete commented Dec 11, 2017

The "-static-libtool-libs" argument to libtool tells it only statically link libraries that have a ".la" files, but to dynamically link other libraries.

The use-case is the following:

  • A "device" SDK containing dynamic and static versions libc, libm, libpthread, openssl, etc and the headers
  • An "app SDK" built using Cerbero that contains GLib, GStreamer, etc

The device contains the .so files for libraries that are also in its SDK. I want to create the smallest possible binary for my device using GStreamer. So I want the files coming from the "app SDK" to be statically linked into my application, but the files from the "platform SDK" to be static.

Currently, the way I do it is by making sure there are no ".la" files in the "device SDK", but that the "app SDK" always has a .la file.. Then I use the "-static-libtool-libs" option to libtool and it just works.

I suggest doing the same with .pc files. I think the trick is to take the output of "pkg-config --libs --static" and of "pkg-config --libs", and if a library is in both, add it to the list of static libraries. If a library is only in "pkgconfig --libs --static" then add it to the general list.. When all the libraries have been processed, remove all "static" libs from the general list. Then we'll have 2 lists, the "static" list.. which must be statically linked, and the "general list" which must be dynamically linked...

@xclaesse

This comment has been minimized.

Show comment
Hide comment
@xclaesse

xclaesse Dec 12, 2017

Contributor

This has something similar to PR #2711 conceptually. The key is that pkgconfig dependencies could be used both as static and shared library. So meson has to make a choice and currently always link dynamically.

It makes sense to have a project wide option to tell which one to choose when both are available. I think this issue should use the same config than the one I introduced for both_library() case.

Contributor

xclaesse commented Dec 12, 2017

This has something similar to PR #2711 conceptually. The key is that pkgconfig dependencies could be used both as static and shared library. So meson has to make a choice and currently always link dynamically.

It makes sense to have a project wide option to tell which one to choose when both are available. I think this issue should use the same config than the one I introduced for both_library() case.

@xclaesse

This comment has been minimized.

Show comment
Hide comment
@xclaesse

xclaesse Dec 12, 2017

Contributor

Actually PkgConfigDependency already has a static argument and the code to invoke pkg-config --static. So it is just a matter to add a global setting to set self.static to true by default on all ExternalDependency objects. Right?

Contributor

xclaesse commented Dec 12, 2017

Actually PkgConfigDependency already has a static argument and the code to invoke pkg-config --static. So it is just a matter to add a global setting to set self.static to true by default on all ExternalDependency objects. Right?

@ocrete

This comment has been minimized.

Show comment
Hide comment
@ocrete

ocrete Dec 12, 2017

Except that linking statically means putting all the deps explicitly on the GCC command line. But using the output of pkg-config --static won't work as some of those may be static. also the order must be respected when linking statically.

ocrete commented Dec 12, 2017

Except that linking statically means putting all the deps explicitly on the GCC command line. But using the output of pkg-config --static won't work as some of those may be static. also the order must be respected when linking statically.

@ocrete

This comment has been minimized.

Show comment
Hide comment
@ocrete

ocrete Dec 12, 2017

Another option for this is to instead do it path based, so I can say "libraries in ~/cerbero/build/ are statically linked, but those in ~/devicesdk/ are dynamically linked"

ocrete commented Dec 12, 2017

Another option for this is to instead do it path based, so I can say "libraries in ~/cerbero/build/ are statically linked, but those in ~/devicesdk/ are dynamically linked"

@nirbheek

This comment has been minimized.

Show comment
Hide comment
@nirbheek

nirbheek Dec 12, 2017

Member

If you set static: true on a pkg-config dependency(), Meson will do some magic to find a static library corresponding to the paths outputted by pkg-config --libs and pass the full path to it to gcc. I implement this a couple of weeks ago, and this will work with 0.44.0. If it doesn't work, please file a bug.

The same should also work for cc.find_library().

Do you need anything else?

Member

nirbheek commented Dec 12, 2017

If you set static: true on a pkg-config dependency(), Meson will do some magic to find a static library corresponding to the paths outputted by pkg-config --libs and pass the full path to it to gcc. I implement this a couple of weeks ago, and this will work with 0.44.0. If it doesn't work, please file a bug.

The same should also work for cc.find_library().

Do you need anything else?

@ocrete

This comment has been minimized.

Show comment
Hide comment
@ocrete

ocrete Dec 12, 2017

Nope, that doesn't work, because you take the whole output of "pkg-config --libs --static" and try to build all of those as static. But this is not what I want, I just want some of them to be.

Here is an example

  • I build a normal GStreamer cerbero tree
  • I use my Fedora's /usr/ as the "device" to target
  • I want my test application to statically link the content of the cerbero tree

My meson.build is

project('test','c')
mydep = dependency('libpng', static: true)
executable('test2',['test.c'], dependencies: mydep)

Then I run meson as

PKG_CONFIG_PATH=~/cerbero/build/dist/linux_x86_64/lib/pkgconfig meson build

But that fails because it tries to find a static library for libm which is in the Libs.private.. Which is what I want to avoid. It does:

Meson encountered an error in file meson.build, line 3, column 0:
Static library not found for 'm'

ocrete commented Dec 12, 2017

Nope, that doesn't work, because you take the whole output of "pkg-config --libs --static" and try to build all of those as static. But this is not what I want, I just want some of them to be.

Here is an example

  • I build a normal GStreamer cerbero tree
  • I use my Fedora's /usr/ as the "device" to target
  • I want my test application to statically link the content of the cerbero tree

My meson.build is

project('test','c')
mydep = dependency('libpng', static: true)
executable('test2',['test.c'], dependencies: mydep)

Then I run meson as

PKG_CONFIG_PATH=~/cerbero/build/dist/linux_x86_64/lib/pkgconfig meson build

But that fails because it tries to find a static library for libm which is in the Libs.private.. Which is what I want to avoid. It does:

Meson encountered an error in file meson.build, line 3, column 0:
Static library not found for 'm'
@nirbheek

This comment has been minimized.

Show comment
Hide comment
@nirbheek

nirbheek Dec 12, 2017

Member

For platform libraries, we should actually add a blacklist. For instance, no one wants -ldl -lm -lc -lpthread to be statically linked unless they are building with a custom toolchain and with no libc.

Would have to think how this would be implemented.

Member

nirbheek commented Dec 12, 2017

For platform libraries, we should actually add a blacklist. For instance, no one wants -ldl -lm -lc -lpthread to be statically linked unless they are building with a custom toolchain and with no libc.

Would have to think how this would be implemented.

@nirbheek nirbheek changed the title from Meson equivalent to libtool's -static-libtool-libs to Don't want platform libraries to be included when linking statically to a dependency Dec 12, 2017

@ocrete

This comment has been minimized.

Show comment
Hide comment
@ocrete

ocrete Dec 12, 2017

I'd rather not having something magical for platform libraries, as maybe we want to use other libraries that are already in the platform such as glib or openssl (to make a smaller binary)

ocrete commented Dec 12, 2017

I'd rather not having something magical for platform libraries, as maybe we want to use other libraries that are already in the platform such as glib or openssl (to make a smaller binary)

@xclaesse

This comment has been minimized.

Show comment
Hide comment
@xclaesse

xclaesse Dec 12, 2017

Contributor

I think what we need is an exclude list like ['/usr', '/lib']. So if the fullpath of libfoo.a has one of them as prefix, link it dynamically instead.

Contributor

xclaesse commented Dec 12, 2017

I think what we need is an exclude list like ['/usr', '/lib']. So if the fullpath of libfoo.a has one of them as prefix, link it dynamically instead.

@rhd

This comment has been minimized.

Show comment
Hide comment
@rhd

rhd Dec 15, 2017

Contributor

The same should also work for cc.find_library().

This doesn't support static: true - even though I wish it did!

Contributor

rhd commented Dec 15, 2017

The same should also work for cc.find_library().

This doesn't support static: true - even though I wish it did!

xclaesse pushed a commit to xclaesse/meson that referenced this issue Dec 21, 2017

Xavier Claessens
PkgConfigDependency: Add option to choose between shared or static
This adds 2 global project options:
- default_link: Tells that we prefer static link when possible. This
option could be used in more places where both static and shared
libraries are available (e.g. future both_library()).
- static_paths: List of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.

xclaesse pushed a commit to xclaesse/meson that referenced this issue Dec 22, 2017

Xavier Claessens
PkgConfigDependency: Add option to choose between shared or static
This adds 2 global project options:
- default_link: Tells that we prefer static link when possible. This
option could be used in more places where both static and shared
libraries are available (e.g. future both_library()).
- static_paths: List of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.

xclaesse pushed a commit to xclaesse/meson that referenced this issue Dec 22, 2017

Xavier Claessens
PkgConfigDependency: Add option to choose between shared or static
This adds 2 global project options:
- default_link: Tells that we prefer static link when possible. This
option could be used in more places where both static and shared
libraries are available (e.g. future both_library()).
- static_paths: List of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.

@shawnl shawnl referenced this issue Dec 24, 2017

Merged

1.3.3 (#1) #2

xclaesse pushed a commit to xclaesse/meson that referenced this issue Dec 24, 2017

Xavier Claessens
PkgConfigDependency: Add option to choose between shared or static
This adds 2 global project options:
- default_link: Tells that we prefer static link when possible. This
option could be used in more places where both static and shared
libraries are available (e.g. future both_library()).
- static_paths: List of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.

xclaesse pushed a commit to xclaesse/meson that referenced this issue Dec 24, 2017

Xavier Claessens
PkgConfigDependency: Add option to choose between shared or static
This adds 2 global project options:
- default_link: Tells that we prefer static link when possible. This
option could be used in more places where both static and shared
libraries are available (e.g. future both_library()).
- static_paths: List of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.

xclaesse pushed a commit to xclaesse/meson that referenced this issue Dec 28, 2017

Xavier Claessens
PkgConfigDependency: Add option to choose between shared or static
This adds 2 global project options:
- default_link: Tells that we prefer static link when possible. This
option could be used in more places where both static and shared
libraries are available (e.g. future both_library()).
- static_paths: List of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.

xclaesse pushed a commit to xclaesse/meson that referenced this issue Jan 1, 2018

Xavier Claessens
PkgConfigDependency: Add option to choose between shared or static
This adds 2 global project options:
- default_link: Tells that we prefer static link when possible. This
option could be used in more places where both static and shared
libraries are available (e.g. future both_library()).
- static_paths: List of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.

xclaesse pushed a commit to xclaesse/meson that referenced this issue Jan 2, 2018

Xavier Claessens
PkgConfigDependency: Add option to choose between shared or static
This adds 2 global project options:
- default_link: Tells that we prefer static link when possible. This
option could be used in more places where both static and shared
libraries are available (e.g. future both_library()).
- static_paths: List of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.

xclaesse pushed a commit to xclaesse/meson that referenced this issue Apr 4, 2018

Xavier Claessens
Add 'static_paths' global option
It is a list of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.

xclaesse pushed a commit to xclaesse/meson that referenced this issue Apr 5, 2018

Xavier Claessens
Add 'static_paths' global option
It is a list of path prefixes where it is allowed to use static
libraries. By default it has "/" which means all paths are allowed.
Sometimes it is useful to have there the path to an SDK that we want to
static link and keep dynamic link for everything else (e.g. plateform
libraries).

Closes mesonbuild#2765.
@xclaesse

This comment has been minimized.

Show comment
Hide comment
@xclaesse

xclaesse Sep 27, 2018

Contributor

I'm facing a similar case for glib CI: https://gitlab.gnome.org/GNOME/glib/merge_requests/360

I would like in the CI to install glib into a prefix, then build a simple app that static link on the installed glib build, to verify the installed .pc is correct and static links works (spoiler alert, current they don't).

My test project looks like this:

project('test-static-link', 'c')

app = executable('test-static-link', 'app.c',
  dependencies : dependency('gio-2.0', static : true)
)
test('test-static-link', app)

The problem is the "static : true" kwarg is a all-or-nothing switch. Setting it to true means it will try to static link all glib dependencies (e.g. libffi) but fedora (used as base for our CI docker) doesn't package static libraries, so meson is printing WARNING message that it cannot find .a file for them. I would like to be able to tell "static link glib/gobject/gio that are installed into a custom prefix, and dynamic link their dependencies that comes from the distro".

I'm not convinced anymore PR #2816 is the best way to achieve this, I think there are easier and less invasive ways. I think we just need to make static kwarg accept other types than a a simple boolean. For example dependency('foo', static : ['/home/user/my-prefix/lib']) that would mean "static link only if the .a is found in that directory".

We could then do something like (would require nicer API to avoid double dependency lookup):

tmp = dependency('foo')
libdir = tmp.get_pkgconfig_variable('libdir')
dep = dependency('foo', static : [libdir])
Contributor

xclaesse commented Sep 27, 2018

I'm facing a similar case for glib CI: https://gitlab.gnome.org/GNOME/glib/merge_requests/360

I would like in the CI to install glib into a prefix, then build a simple app that static link on the installed glib build, to verify the installed .pc is correct and static links works (spoiler alert, current they don't).

My test project looks like this:

project('test-static-link', 'c')

app = executable('test-static-link', 'app.c',
  dependencies : dependency('gio-2.0', static : true)
)
test('test-static-link', app)

The problem is the "static : true" kwarg is a all-or-nothing switch. Setting it to true means it will try to static link all glib dependencies (e.g. libffi) but fedora (used as base for our CI docker) doesn't package static libraries, so meson is printing WARNING message that it cannot find .a file for them. I would like to be able to tell "static link glib/gobject/gio that are installed into a custom prefix, and dynamic link their dependencies that comes from the distro".

I'm not convinced anymore PR #2816 is the best way to achieve this, I think there are easier and less invasive ways. I think we just need to make static kwarg accept other types than a a simple boolean. For example dependency('foo', static : ['/home/user/my-prefix/lib']) that would mean "static link only if the .a is found in that directory".

We could then do something like (would require nicer API to avoid double dependency lookup):

tmp = dependency('foo')
libdir = tmp.get_pkgconfig_variable('libdir')
dep = dependency('foo', static : [libdir])

xclaesse pushed a commit to xclaesse/meson that referenced this issue Sep 27, 2018

Xavier Claessens
dependency: Allow passing paths to 'static' kwarg
Passing 'static:true' to dependency() will try to static link the
dependency, but also dependencies of the dependency, etc. This could
lead to even static link libc and other basic system libraries, or warn
if those static libraries cannot be found on the system.

To give more flexibility, 'static' kwarg now accepts a list of paths
where it is acceptable to use static libraries. If the static library is
found outside one of those paths, then it won't be used and it will be
kept dynamic.

Often the path will be ${libdir} from the dependency's .pc file, so we
support '${variable}' in paths that will be replaced by the value of
that variable from the pkg-config file.

Closes mesonbuild#2765.

@xclaesse xclaesse referenced a pull request that will close this issue Sep 27, 2018

Open

dependency: Allow passing paths to 'static' kwarg #4276

xclaesse pushed a commit to xclaesse/meson that referenced this issue Sep 27, 2018

Xavier Claessens
dependency: Allow passing paths to 'static' kwarg
Passing 'static:true' to dependency() will try to static link the
dependency, but also dependencies of the dependency, etc. This could
lead to even static link libc and other basic system libraries, or warn
if those static libraries cannot be found on the system.

To give more flexibility, 'static' kwarg now accepts a list of paths
where it is acceptable to use static libraries. If the static library is
found outside one of those paths, then it won't be used and it will be
kept dynamic.

Often the path will be ${libdir} from the dependency's .pc file, so we
support '${variable}' in paths that will be replaced by the value of
that variable from the pkg-config file.

Closes mesonbuild#2765.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment