Use cross-compilation flags when checking for system directories #3894

bruce-richardson · 2018-07-16T13:54:31Z

When building 32-bit on a 64-bit system the "system directories" for libraries are the 64-bit ones unless the appropriate CFLAGS e.g. -m32, as passed to the compiler. Doing so is tricky, as the flags seem to be stored in the environment, rather than the compiler object. This draft patch is a first-attempt to implement this, and works ok for 32-bit and 64-bit builds on linux using GCC.

If the problem of incorrect system paths can be solved, we may be able to remove the build checks in find_library() and hopefully fix #3881.

ignatenkobrain · 2018-07-16T13:55:45Z

mesonbuild/compilers/c.py

-    def get_library_dirs(self):
-        key = tuple(self.exelist)
+    def get_library_dirs_real(self, env):
+        code=''


[Flake8]

[E225] missing whitespace around operator

Sure. I didn't run flake8 and clean up the code much, as I'm looking for initial feedback as to whether this a correct approach or not to fixing the issues. [Hence "DRAFT" in the title :-)]

These comments are automated...

ePirat · 2018-07-17T22:44:00Z

Seems related to bug #3720?

nirbheek

This approach is good, and is a better fix for #3720 than #3916.

CCing @jpakkane

This also needs a test.

nirbheek · 2018-08-07T12:02:44Z

mesonbuild/compilers/c.py

+                    paths = [os.path.realpath(p) for p in libstr.split(':') if os.path.exists(os.path.realpath(p))]
+            return paths
+
+    def get_library_dirs(self, env=None):


Is get_library_dirs() called from anywhere that doesn't have access to the environment? If no, we should make this a required argument so we don't have hidden bugs. Then you can also remove the conditional below in _get_compiler_check_args().

It is generally available, except in the case for mesonbuild/backend/backends.py, where we are filtering out rpath dirs which are in the system directory path. There _libdir_is_system calls get_library_dirs() without access to any environment. [That in turn is called by rpaths_for_bundled_shared_libraries which similarly doesn't have any env info]

As an aside: Is there a reason why the cross build compiler flags are stored in an environment object, rather than in a compiler object for the cross-compiler in question? I would have thought that for cross builds we would have multiple compiler objects, each tracking it's own args.

As an aside: Is there a reason why the cross build compiler flags are stored in an environment object, rather than in a compiler object for the cross-compiler in question?

Just cruft from ages past. I think someone recently opened a PR to change this.

mesonbuild/backend/backends.py

The Backend object has the environment object available as self.environment, so you should pass that.

nirbheek · 2018-08-07T12:04:38Z

mesonbuild/compilers/compilers.py

+    def get_library_dirs(self, env=None):
+        if env is None:
+            env = os.environ.copy()
+            env['LC_ALL'] = 'C'


This is a totally different environment.

Agreed. However, in the case we don't have an environment available, we need some fallback scheme.

What I mean is, that the env passed to get_library_dirs() in compilers/c.py is an Environment object, while this is os.environ (a dict). We should not conflate the two. Just rename this to osenv or something.

Or, better yet, if you change this function to use _build_wrapper() like c.py, this is no longer a problem.

nirbheek · 2018-08-07T12:05:02Z

mesonbuild/compilers/compilers.py

+    def get_library_dirs(self, env=None):
+        if env is None:
+            env = os.environ.copy()
+            env['LC_ALL'] = 'C'
        stdo = Popen_safe(self.exelist + ['--print-search-dirs'], env=env)[1]


You will want to change this to also use _build_wrapper() and the get_program_dirs() below too.

bruce-richardson · 2018-08-10T16:08:05Z

mesonbuild/compilers/c.py

+            return paths
+
+    def get_library_dirs(self, env=None):
+        key = tuple(self.exelist) # todo include environment here


Help requested as to how to manage the environment here. Is it possible to hash it in with the compiler to save the results for later?

The Environment object is hashable, afair

nirbheek · 2018-08-19T15:32:25Z

Well, this is fun. The unit tests are failing in a spectacular fashion, but only if they are run one after the other. Not if they are run individually. Probably because we're messing up some global state.

This means that we will take into account all the flags set in the cross file when fetching the list of library dirs, which means we won't incorrectly look for 64-bit libraries when building for 32-bit. Signed-off-by: Nirbheek Chauhan <nirbheek@centricular.com> Closes mesonbuild#3881

Shared libraries may not always be linkable without the use of other libraries, so don't make it a linker error.

nirbheek · 2018-08-19T23:18:58Z

@mtp401 @ChuckDaniels87 @ignatenkobrain Can you please test this branch to see if that fixes your issues?

jpakkane · 2018-08-20T21:05:47Z

Am I correct in understanding that the underlying problem was that compile/link arguments from the cross file were not being used in library lookup tests?

If yes then that is strange because it should do that. Is it only for find_library or are the args missing also from other tests?

ismaelgv · 2018-08-20T23:05:24Z

@nirbheek this fixes #3958. Thanks!

nirbheek · 2018-08-21T11:42:11Z

Am I correct in understanding that the underlying problem was that compile/link arguments from the cross file were not being used in library lookup tests?

The algorithm for cc.find_library() used to be: "if not extra_dirs: try linking to -lfoo, if that fails: search manually in extra_dirs, if that fails: search in system dirs ($CC --print-search-dirs)".

The "try linking to -lfoo" part has always read cross-args from the cross file or CFLAGS/LDFLAGS if compiling natively, and still does. $CC --print-search-dirs never did, but no one reported a problem with that.

Then, in 0.47, I made PkgConfigDependency start passing extra_dirs to find_library so that we skip the link check completely and only do manual searching. This is good, except that since we don't pass cross-compile args such as -m64 or -m32 to $CC --print-search-dirs, it prints the location to both 32-bit and 64-bit args, which is what @bruce-richardson reported.

We merged his fix which does a link check against the library found after manually searching to ensure that it's the right arch, but that broke searching for pkg-config libraries that have missing symbols, so I papered around it by only doing the link check for libraries found in the system paths, not extra_dirs.

Then people reported #3881 and #3958, where libraries in the system paths had undefined symbols, which meant we couldn't do a link check at all.

That leads us to this fix. Now we use _build_wrapper (the same function used by all compiler checks) to call $CC --print-search-dirs, so cross-compile and CFLAGS/LDFLAGS are passed to it, and we always get the right library search dirs. No need for a link check against the found library.

HOWEVER, there is still one piece missing in all this. cc.find_library() looks for -L flags specified in LDFLAGS, but cc.find_library(..., dirs: ...) does not since it only does manual searching.

The next step (which can't go into the stable release, and hence isn't in this PR) which will complete the fix for #3720 is to also collect -L flags from the cross file or from LDFLAGS for native and add them to the end of extra_dirs.

Note that has_function() has always used _build_wrapper() so it reads link flags correctly. Not sure what that's about. Would need a test case to see that one.

jpakkane · 2018-08-22T16:43:06Z

Gaah! I really wish the library situation was not so completely terrible as it is. But given that it is and we can't change it, we need to work with it. Feel free to merge once you are convinced that this works.

nirbheek · 2018-08-22T16:53:31Z

I agree, I really wish we had an introspection API or something for compilers/linkers instead of having to guess/emulate/duplicate their behaviour. Something like "just tell me where this library resolves to".

ignatenkobrain reviewed Jul 16, 2018

View reviewed changes

nirbheek reviewed Aug 7, 2018

View reviewed changes

nirbheek mentioned this pull request Aug 7, 2018

Use global args in tests #3916

Closed

bruce-richardson commented Aug 10, 2018

View reviewed changes

nirbheek added this to the 0.47.2 milestone Aug 11, 2018

nirbheek mentioned this pull request Aug 11, 2018

Meson 0.47.1 breaks cross builds using Boost Python #3881

Closed

nirbheek force-pushed the fix_32-bit_library_search_path branch from 3806b68 to df3e828 Compare August 19, 2018 15:18

nirbheek force-pushed the fix_32-bit_library_search_path branch 3 times, most recently from e586202 to 0c974c2 Compare August 19, 2018 18:08

find_library: Allow undefined symbols while linking

c98156d

Shared libraries may not always be linkable without the use of other libraries, so don't make it a linker error.

nirbheek force-pushed the fix_32-bit_library_search_path branch 2 times, most recently from 14f8deb to fc54046 Compare August 19, 2018 19:28

nirbheek changed the title ~~DRAFT: use cross-compilation flags when checking for system directories~~ Use cross-compilation flags when checking for system directories Aug 19, 2018

Test that system shlibs with undefined symbols can be found

388d086

nirbheek force-pushed the fix_32-bit_library_search_path branch from fc54046 to 388d086 Compare August 19, 2018 22:17

nirbheek mentioned this pull request Aug 20, 2018

Fix linking of shared/static libs with static libs #3939

Closed

nirbheek merged commit 5e93393 into mesonbuild:master Aug 22, 2018

Ericson2314 mentioned this pull request Mar 24, 2019

Don't collect preprocssor flags separately from compiler flags #5123

Merged

eli-schwartz mentioned this pull request Oct 21, 2022

compiler.find_library() function considers unusable libraries to be found #10936

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use cross-compilation flags when checking for system directories #3894

Use cross-compilation flags when checking for system directories #3894

bruce-richardson commented Jul 16, 2018

ignatenkobrain Jul 16, 2018

bruce-richardson Jul 16, 2018

QuLogic Jul 16, 2018

ePirat commented Jul 17, 2018

nirbheek left a comment

nirbheek Aug 7, 2018

bruce-richardson Aug 10, 2018

nirbheek Aug 11, 2018

nirbheek Aug 11, 2018

nirbheek Aug 7, 2018

bruce-richardson Aug 10, 2018

nirbheek Aug 11, 2018

nirbheek Aug 11, 2018

nirbheek Aug 7, 2018

bruce-richardson Aug 10, 2018

bruce-richardson Aug 10, 2018

nirbheek Aug 11, 2018

nirbheek commented Aug 19, 2018

nirbheek commented Aug 19, 2018

jpakkane commented Aug 20, 2018

ismaelgv commented Aug 20, 2018

nirbheek commented Aug 21, 2018 •

edited

jpakkane commented Aug 22, 2018

nirbheek commented Aug 22, 2018

Use cross-compilation flags when checking for system directories #3894

Use cross-compilation flags when checking for system directories #3894

Conversation

bruce-richardson commented Jul 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ePirat commented Jul 17, 2018

nirbheek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirbheek commented Aug 19, 2018

nirbheek commented Aug 19, 2018

jpakkane commented Aug 20, 2018

ismaelgv commented Aug 20, 2018

nirbheek commented Aug 21, 2018 • edited

jpakkane commented Aug 22, 2018

nirbheek commented Aug 22, 2018

nirbheek commented Aug 21, 2018 •

edited