From 308ab787267adfd4ff4c4979f09ca01ba7abb197 Mon Sep 17 00:00:00 2001 From: Max Howell Date: Wed, 16 Mar 2016 17:33:01 -0700 Subject: [PATCH 1/2] Proposal: SwiftPM System Module Search Paths --- ...NNNN-swiftpm-system-module-search-paths.md | 207 ++++++++++++++++++ 1 file changed, 207 insertions(+) create mode 100644 proposals/NNNN-swiftpm-system-module-search-paths.md diff --git a/proposals/NNNN-swiftpm-system-module-search-paths.md b/proposals/NNNN-swiftpm-system-module-search-paths.md new file mode 100644 index 0000000000..ce0ec44e48 --- /dev/null +++ b/proposals/NNNN-swiftpm-system-module-search-paths.md @@ -0,0 +1,207 @@ +# SwiftPM System Module Search Paths + +* Proposal: [SE-NNNN](https://github.com/apple/swift-evolution/blob/master/proposals/NNNN-swiftpm-system-module-search-paths.md) +* Author: [Max Howell](https://github.com/mxcl) +* Status: **Awaiting review** +* Review manager: Anders Bertelrud + + +## Introduction + +Swift is able to `import` C libraries in the same manner as Swift libraries. + +For this to occur the library must be represented by a clang module-map file. + +The current system for using these module-map files with SwiftPM works, but with +a number of caveats that must be addressed. + + +[swift-evolution thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013201.html) + +## Terminology + +* **Swift Package**: A package consumed by SwiftPM that comes with sources that SwiftPM builds into modules +* **Swift System Package**: A package consumed by SwiftPM that refers to a modular system library not installed by SwiftPM + + +## Motivation + +The current implementation of Swift System Packages have a number of problems: + + 1. Install locations vary across platforms and `.modulemap` files require absolute paths + 2. `/usr/lib:/usr/local/lib` is not always a sufficient `-L` linker search path + 3. `/usr/include:/usr/local/include` is not always a sufficient `-I` C compiler search path + 4. Installing the system library is left up to the end-user to figure out + +For example to import a module map representing the GTK library, the include search +path must be supplemented with `-I/usr/include/gtk` so that a number of includes in +the `gtk.h` header can be sourced for the complete modular definition of GTK. + +For example to import a module map representing the GTK library a user must first have +a copy of GTK and its headers installed. On Debian based systems the install name for +this system package is `libgtk-3-0-dev` which is not entirely intuitive. + +For example, Homebrew and MacPorts on OS X install to prefixes other than `/usr`. +`.modulemap` files must specify headers with absolute paths. The standard we +encourage with modulemaps is for the headers to be specified with an assumed +prefix of `/usr`, but you will not find eg. `jpeglib.h` at `/usr/include/jpeglib.h` +if it is installed with Homebrew or MacPorts. + + +## Proposed Solution + +We propose that SwiftPM gains the ability to read `.pc` files written for the +cross-platform `pkg-config` tool. These files describe the missing search paths +that SwiftPM requires. + +We propose that SwiftPM gains the ability to read `.pc` files written for the +cross-platform `pkg-config` tool. These files specify the install location of +system libraries and will allow SwiftPM to preprocess the modulemap changing +the specified header prefixes. + +We propose that `Package.swift` is supplemented with metadata that provides the +package-install-name for specific platforms. + + +## Detailed Design + +### Solving Path/Flags Issues + +A system library should provide a pkg-config file (`.pc`) which describes: + + 1. Its install location + 2. Supplementary flags that should be used when compiling against this library + 3. Supplementary flags that should be used when linking against this library + +If SwiftPM read the `.pc` file that comes with packages, this solves problems 1 through 3. + +Of the tickets we currently have open describing issues using Swift-system-module-packages, +reading the `.pc` file would fix all of them. + +It is a convention to name the `.pc` file after the library link-name, so we can determine +which `.pc` file to ask `pkg-config` for by parsing the `.modulemap` file in the Swift +package. However sometimes this is not true, (eg. GTK-3 on Ubuntu), so we will allow an +override in the `Package.swift` file, for example: + +```swift +let package = Package( + name: "CFoo", + pkgConfigName: "gtk-3" +) +``` + +Thus we would search for a filename: `gtk-3.pc`. + +We don’t want to depend on `pkg-config`, so we will implement the reading of `.pc` files +according to the pkg-config specification, including: + + 1. Obeying the correct search .pc file search paths + 2. Following overrides due to any `PKG_CONFIG_PATH` environment variable + + +### Providing Package Install Names + +`Package.swift` would be supplemented like so: + +```swift +let package = Package( + name: "CFoo", + pkgConfigName: "foo", + providers: [ + .Brew(installName: "foo"), + .Apt(installName: "libfoo-dev"), + ], +) +``` + +Thus, in the event of build failure for modules that depend on this +package we would provide additional help to the user: + +``` +error: failed to build module `bar' +note: you may need to install `foo' using your system-packager: + + apt-get install libfoo-dev +``` + +Since the syntax to provide this information uses an explicit enum we can +add code for each enum to detect which system packagers should be +recommended. The community will need to write the code for their own +platforms. It also means that if a specific packager requires additional +parameters, they can be added on a per enum basis. + +#### Install-names are not standard + +`apt` is used across multiple distirbutions and the install-names for +tools vary. Even for the same distribution install-names may vary +across releases (eg. from Ubuntu 15.04 to Ubuntu 15.10) or even on +ocassion at finer granularity. + +We will not add explicit handling for this, but one can imagine the +enums for different system packagers could be supplemented in a backwards +compatible way to provide specific handling as real-world uses emerge, eg: + +```swift +case Apt(installName: String) + +// …could be adapted to: + +struct Debian: Linux {} +struct Ubuntu: Debian { + enum Variant { + case Gubuntu + case Kubuntu(Version) + } + enum Version { + case v1510 + case v1504 + } +} +case Apt(installName: String, distribution: Linux? = nil) +``` + +## Impact on Existing Code + +There will be no impact on existing code as this feature simply improves +an existing feature making new code possible. + + +## Alternatives Considered + +A clear alternative is allowing additional flags to be specified in a system-module package’s `Package.swift`. + +However since these paths and flags will vary by platform this would because a large matrix that is quite a maintenance burden. Really this information is recorded already, in the system package itself, and in fact almost all packages nowadays provide it in a `.pc` `pkg-config` file. + +Also we do not want to allow arbitrary flags to be specified in `Package.swift`, this allows packages too much power +to break a large dependency graph with bad compiles. The only entity that understands the whole graph and can manage +the build without breakage is SwiftPM, and allowing packages themselves to add arbitrary flags prevents SwiftPM from +being able to understand and control the build ensuring reliability and preventing “Dependency Hell”. + + +## Unsolved Problems + +Some (usually more legacy) system libraries do not provide `.pc` files instead they may provide +a tool named eg. `foo-config` that can be queried for compile and link flags. We do not yet +support these tools, and would prefer to take a wait and see approach to determine how +important supporting them may be. + +Some libraries on OS X do not come with `.pc` files. Again we'd like to see which libraries +are affected before potentially offering a solution here. + + +## Future Directions + +The build system could be made more reliable by having the specific packager provide the information that this +proposal garners from `pkg-config`. For example, Homebrew installs everything into independent directories, using these +directories instead of more general POSIX search paths means there is no danger of edge-case search path collisions and the wrong libraries being picked up. + +If this was done `pkg-config` could become just one option for providing this data, and be used only as a fallback. + +--- + +We do not wish to provide a flag to automatically install dependencies via the +system packager. We feel this opens us up to security implications beyond the +scope of this tool. + +Instead we can provide JSON output that can be parsed and executed by some +other tooling developed outside of Apple. From deb0686f0e840d3853ba9e900ec4efb5f353afa1 Mon Sep 17 00:00:00 2001 From: Max Howell Date: Fri, 1 Apr 2016 16:56:57 -0700 Subject: [PATCH 2/2] Revisions from internal feedback --- ...NNNN-swiftpm-system-module-search-paths.md | 37 ++++++++++--------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/proposals/NNNN-swiftpm-system-module-search-paths.md b/proposals/NNNN-swiftpm-system-module-search-paths.md index ce0ec44e48..5d7eae7f9e 100644 --- a/proposals/NNNN-swiftpm-system-module-search-paths.md +++ b/proposals/NNNN-swiftpm-system-module-search-paths.md @@ -18,15 +18,18 @@ a number of caveats that must be addressed. [swift-evolution thread](https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20160321/013201.html) + ## Terminology -* **Swift Package**: A package consumed by SwiftPM that comes with sources that SwiftPM builds into modules -* **Swift System Package**: A package consumed by SwiftPM that refers to a modular system library not installed by SwiftPM +* **SwiftPM Source Package**: A package consumed by SwiftPM that comes with sources that SwiftPM builds into modules +* **SwiftPM System Package**: A package consumed by SwiftPM that refers to a modular system library not installed by SwiftPM +* **System Package**: A package provided by a system packager like, eg. `apt`, `pacman` or `brew`. +* **System Packager**: A system package manager like, eg. `apt`, `pacman` or `brew`. ## Motivation -The current implementation of Swift System Packages have a number of problems: +The current implementation of SwiftPM System Packages have a number of problems: 1. Install locations vary across platforms and `.modulemap` files require absolute paths 2. `/usr/lib:/usr/local/lib` is not always a sufficient `-L` linker search path @@ -39,7 +42,7 @@ the `gtk.h` header can be sourced for the complete modular definition of GTK. For example to import a module map representing the GTK library a user must first have a copy of GTK and its headers installed. On Debian based systems the install name for -this system package is `libgtk-3-0-dev` which is not entirely intuitive. +this System Package is `libgtk-3-0-dev` which is not entirely intuitive. For example, Homebrew and MacPorts on OS X install to prefixes other than `/usr`. `.modulemap` files must specify headers with absolute paths. The standard we @@ -73,15 +76,15 @@ A system library should provide a pkg-config file (`.pc`) which describes: 2. Supplementary flags that should be used when compiling against this library 3. Supplementary flags that should be used when linking against this library -If SwiftPM read the `.pc` file that comes with packages, this solves problems 1 through 3. +If SwiftPM read the `.pc` file that comes with System Packages, this solves problems 1 through 3. -Of the tickets we currently have open describing issues using Swift-system-module-packages, +Of the tickets we currently have open describing issues using SwiftPM System Packages, reading the `.pc` file would fix all of them. It is a convention to name the `.pc` file after the library link-name, so we can determine -which `.pc` file to ask `pkg-config` for by parsing the `.modulemap` file in the Swift -package. However sometimes this is not true, (eg. GTK-3 on Ubuntu), so we will allow an -override in the `Package.swift` file, for example: +which `.pc` file to ask `pkg-config` for by parsing the `.modulemap` file in the SwiftPM Package. +However sometimes this is not true, (eg. GTK-3 on Ubuntu), so we will allow an override in +the `Package.swift` file, for example: ```swift let package = Package( @@ -92,14 +95,14 @@ let package = Package( Thus we would search for a filename: `gtk-3.pc`. -We don’t want to depend on `pkg-config`, so we will implement the reading of `.pc` files -according to the pkg-config specification, including: +We don’t want to introduce a new dependency (on `pkg-config`) to Swift, so we will +implement the reading of `.pc` files according to the pkg-config specification, including: 1. Obeying the correct search .pc file search paths 2. Following overrides due to any `PKG_CONFIG_PATH` environment variable -### Providing Package Install Names +### Hinting At System-Package Install-Names `Package.swift` would be supplemented like so: @@ -115,7 +118,7 @@ let package = Package( ``` Thus, in the event of build failure for modules that depend on this -package we would provide additional help to the user: +SwiftPM Package we would output additional help to the user: ``` error: failed to build module `bar' @@ -127,7 +130,7 @@ note: you may need to install `foo' using your system-packager: Since the syntax to provide this information uses an explicit enum we can add code for each enum to detect which system packagers should be recommended. The community will need to write the code for their own -platforms. It also means that if a specific packager requires additional +platforms. It also means that if a specific system-packager requires additional parameters, they can be added on a per enum basis. #### Install-names are not standard @@ -168,9 +171,9 @@ an existing feature making new code possible. ## Alternatives Considered -A clear alternative is allowing additional flags to be specified in a system-module package’s `Package.swift`. +A clear alternative is allowing additional flags to be specified in a SwiftPM System Package’s `Package.swift`. -However since these paths and flags will vary by platform this would because a large matrix that is quite a maintenance burden. Really this information is recorded already, in the system package itself, and in fact almost all packages nowadays provide it in a `.pc` `pkg-config` file. +However since these paths and flags will vary by platform this would because a large matrix that is quite a maintenance burden. Really this information is recorded already, in the System Package itself, and in fact almost all System Packages nowadays provide it in a `.pc` `pkg-config` file. Also we do not want to allow arbitrary flags to be specified in `Package.swift`, this allows packages too much power to break a large dependency graph with bad compiles. The only entity that understands the whole graph and can manage @@ -191,7 +194,7 @@ are affected before potentially offering a solution here. ## Future Directions -The build system could be made more reliable by having the specific packager provide the information that this +The build system could be made more reliable by having the specific system packager provide the information that this proposal garners from `pkg-config`. For example, Homebrew installs everything into independent directories, using these directories instead of more general POSIX search paths means there is no danger of edge-case search path collisions and the wrong libraries being picked up.