Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-6317] Package Manager's .pc parsing makes it difficult to selectively shadow system libraries #4927

Open
Lukasa opened this issue Nov 7, 2017 · 12 comments
Labels

Comments

@Lukasa
Copy link
Contributor

Lukasa commented Nov 7, 2017

Previous ID SR-6317
Radar rdar://problem/40235197
Original Reporter @Lukasa
Type Bug
Environment

Linux 1779df4f8699 4.9.49-moby #1 SMP Wed Sep 27 23:17:17 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Swift version 4.0.2 (swift-4.0.2-RELEASE)

Additional Detail from JIRA
Votes 0
Component/s Package Manager
Labels Bug
Assignee None
Priority Medium

md5: e849c854eac92b5e5de3cdf3e167fa11

Issue Description:

I'm running a Swift project where I want to link mostly against system libraries (e.g. the system copy of icu-uc), but want to selectively shadow one or two other system libraries with different versions. This should be something that can mostly be solved by using pkg-config, as pkg-config allows us to give SwiftPM library search paths. This means, for example, if you're using icu-uc.pc you should expect to see the following flags added:

$ pkg-config --libs icu-uc
-licuuc -licudata

However, SwiftPM's parsing of .pc files doesn't behave exactly like that. Instead it behaves like pkg-config with the optional PKG_CONFIG_ALLOW_SYSTEM_LIBS flags set:

$ env PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 pkg-config --libs icu-uc
-L/usr/lib/x86_64-linux-gnu -licuuc -licudata

This means that if some libraries are being added that shadow system libraries, and some are not, you will get both -L/my/shadow/path and -L/usr/lib/x86_64-linux-gnu in the command line. Depending on the order in which they appear this may mean that the attempt to shadow the system libraries fails, as the system library path may end up first in the search path again.

I'd propose that SwiftPM should behave more like pkg-config without the PKG_CONFIG_ALLOW_SYSTEM_LIBS environment variable set, and refuse to put the system library path in a -L flag.

@aciidb0mb3r
Copy link
Member

aciidb0mb3r commented Nov 7, 2017

We already do this for macOS by removing /usr/include and /usr/lib. I wonder if there is a way to ask pkg-config about the system libary paths so we can strip them off. However, that won't work if pkg-config is not installed on the system.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 23, 2017

I don't think we can do that for pkg-config anyway: this seems to be data compiled into the binary and not exposed through any of its interfaces. However, you can use the following dirty hack to work out whether a given package is using the system library path:

root@e1f25860f73b:/code# pkg-config --libs openssl
-lssl -lcrypto
root@e1f25860f73b:/code# env PKG_CONFIG_SYSTEM_LIBRARY_PATH="" pkg-config --libs openssl
-L/usr/lib/x86_64-linux-gnu -lssl -lcrypto

That is, you can override the system library path with the empty string.

Of course, if we were going to rely on this or something like it then we'd have to ask ourselves why we weren't just shelling out to pkg-config directly in the first instance, which would clearly resolve the issue.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 23, 2017

The easiest proposed fix, if you're interested in having it, would just be to defer to pkg-config directly on systems that have it installed. That is, if we can locate a pkg-config on the path then rather than use the custom logic in PkgConfig.swift we just call it directly.

The downside of that mechanism is that we will inevitably behave differently on different systems, which can lead to strange and difficult-to-debug issues.

Another proposal, then, is to just manually provide these for supported Linux distributions. They can be obtained by grabbing source packages for the supported pkg-config installers and looking in their build scripts. For example, Ubuntu 14.04's pkg-config package provides the following as the system library path:

/lib:/lib/i386-linux-gnu:/lib/x86_64-linux-gnu/4.8:/lib/x86_64-linux-gnux32:/lib32:/libx32:/usr/lib:/usr/lib/i386-linux-gnu:/usr/lib/x86_64-linux-gnu/4.8:/usr/lib/x86_64-linux-gnux32:/usr/lib32:/usr/libx32

This somewhat opaque string is the result of a build script that basically does this:

OPTS=$(gcc -print-multi-lib | sed -n -e's/.*;@/-/p')
for opt in $OPTS; do \
     gcc "$opt" -print-search-dirs | sed -n -e's/^libraries: =//p' \
     | sed -e's/:/\n/g' | xargs -n1 readlink -f | grep -v 'gcc\|/[0-9.]\+$$'; \
    done | sort -u | tr '\n' : | sed 's/:$$//'
echo ""

For Ubuntu 16.04 this is:

/lib:/lib/i386-linux-gnu:/lib/x86_64-linux-gnu/5:/lib/x86_64-linux-gnux32:/lib32:/libx32:/usr/lib:/usr/lib/i386-linux-gnu:/usr/lib/x86_64-linux-gnu/5:/usr/lib/x86_64-linux-gnux32:/usr/lib32:/usr/libx32

For Ubuntu 16.10 this is:

/lib:/lib/i386-linux-gnu:/lib/x86_64-linux-gnu/6:/lib/x86_64-linux-gnux32:/lib32:/libx32:/usr/lib:/usr/lib/i386-linux-gnu:/usr/lib/x86_64-linux-gnu/6:/usr/lib/x86_64-linux-gnux32:/usr/lib32:/usr/libx32

You'll note these strings are not quite identical: in particular, they contain paths that are versioned by what I presume to be the version of libc on the system. This does lead to the annoyance that we actually have to maintain three of these opaque strings. It gets worse if we want to support different architectures for these distributions, or different distributions: each entry in the arch/distro/version matrix gets its own opaque string. That will get annoying, fast.

That makes my vote for fixing this problem go towards just letting pkg-config do this when it's installed on the system, and having the Swift code available as a fallback for systems without a pkg-config.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 23, 2017

After some discussion with my colleagues we've come up with an alternative: we can ask clang what it thinks the system library paths are. This will probably work a bit better, because 100% of systems compiling Swift with SwiftPM that require .pc parsing will have a clang installed (otherwise they won't be able to build), and they will use clang to do that build. The end result is that gcc's list might not quite match clang's list, and in that case we'd like to use clang's instead of gcc's.

It was even suggested that we might be able to do this at SwiftPM build time, to avoid the overhead of checking each time we run SwiftPM. I'll investigate both these options to see how viable they are.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 23, 2017

Ok, so assuming we're using the regular ld linker, we can just ask ld. We get output like this (Ubuntu 14.04):

# ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012
SEARCH_DIR("/usr/x86_64-linux-gnu/lib64")
SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib/x86_64-linux-gnu")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib")
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")

So that would make a runtime check possible and fairly straightforward to parse.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 23, 2017

So CMAKE_SYSTEM_LIBRARY_PATH should be the correct variable to pass, assuming we're built on the right system. I want to double-check though.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 23, 2017

In point of fact, we'd need to replicate cmake's whole logic here, because it's non-trivial:

   <prefix>/lib/<arch> if CMAKE_LIBRARY_ARCHITECTURE is set, and
   <prefix>/lib for each <prefix> in CMAKE_SYSTEM_PREFIX_PATH
   CMAKE_SYSTEM_LIBRARY_PATH
   CMAKE_SYSTEM_FRAMEWORK_PATH

This probably gets us pretty close to the correct answer in most cases, as well. That said, asking ld is probably easier in general.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 24, 2017

Hrm. The downside to "just ask ld" is that ld may not be the linker in use. Swift already supports ld and gold as Linkers on Linux, and there seems to be an open Swift PR that adds support for, amongst other things, lld. This complicates matters somewhat, and suggests that the build system may be the better place to do this work.

Probably the easiest approach is to ask the linker what its paths are as part of the build script. I'll investigate how tricky this is to do.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 24, 2017

However, swiftc allows you to override the linker in use, which means SwiftPM does too (-Xswiftc -use-ld=gold, for example).

If we think lld, gold, and ld.bfd will all disagree on what the system library paths are, we would really have to do this dynamically to ensure we agree with the actual linker in use. If we think they won't disagree, it may be enough to just ask ld.bfd (or any of the other linkers) to get the answer.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 24, 2017

I suppose while I'm enumerating all the options, we could also parse the output of ldconfig -v, which lets us ask the runtime linker what it thinks the search paths are.

@Lukasa
Copy link
Contributor Author

Lukasa commented Nov 27, 2017

Oh, I should note this problem also affects cflags: PKG_CONFIG similarly checks a PKG_CONFIG_ALLOW_SYSTEM_CFLAGS environment variable to determine whether to emit the default include path. I suspect that SwiftPM will encounter similar problems here.

@aciidb0mb3r
Copy link
Member

aciidb0mb3r commented May 14, 2018

@swift-ci create

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
@shahmishal shahmishal transferred this issue from apple/swift May 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants