New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
libcxx : upgrade to 6.0.0 and use CMake (WIP) #1853
Conversation
This is a combined upgrade and overhaul of port:libcxx : - upgrade to 6.0.0 - build using the preferred/official CMake buildsystem instead of through a restored and patched buildit script. One advantage of this way of building is that there is no stdlib.h bug and one can use recent port:clang versions (tested with Clang 5.0.1). - adds a +runtime variant which allows (older (libc++-based)) systems to test and run applications with a current libc++/libc++abi version via DYLD_INSERT_LIBRARIES .
(This allows the use of LTO even with +universal)
Notifying maintainers: |
Note: I use the +runtime variant myself, but am more too lazy to strip it right away than motivated to champion its committal. |
Travis Build #2353 Errored. Lint results
Port libcxx fail on xcode9.3. Log |
@jeremyhu you should probably review this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can't be accepted as-is due to the use of cmake.
lang/libcxx/Portfile
Outdated
@@ -59,7 +123,7 @@ platform darwin { | |||
post-activate { | |||
set dirs / | |||
if {${os.subplatform} eq "macosx"} { | |||
lappend dirs ${developer_dir}/SDKs/MacOSX[join [lrange [split ${macosx_version} .] 0 1] .].sdk | |||
lappend dirs ${developer_dir}/SDKs/MacOSX${macosx_version}.sdk |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please do not revert 5bdf46b.
|
||
if {[tbool use_cmake]} { | ||
revision [expr ${revision} + 1] | ||
PortGroup cmake 1.1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cmake requires libcxx; therefore libcxx cannot require cmake, because that would be a circular dependency and MacPorts doesn't support that and its behavior in that case is undefined. See https://trac.macports.org/ticket/56519.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CMake has started requiring C++11 only relatively recently, and doesn't require libcxx for that on 10.8 and earlier only.
Wouldn't it make sense to provide a pre-C++11 version of cmake for this kind of situation (via [sub]port or variant) provided that version is new enough to build libc++)? We're only talking about a build dependency here (= bootstrap situation), and the CMake version used for building a port should not affect the end result (= no reproducible build principle issues).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes to this port need to be tested against every single macOS, starting from 10.5 and all the way up to 10.13. As Ryan explained, CMake introduces a circular dependency and is problematic for this particular port.
The changes to this port need to be tested against every single macOS, starting from 10.5 and all the way up to 10.13.
I've added a WIP tag to the PR title because evidently that will have to be a combined effort.
The most straightforward way to avoid the circular dependency would be to initialise
```
set use_cmake [file exists /usr/lib/libc++.1.dylib]
```
or some more elaborate scheme involving a `+bootstrap` variant. The whole circular dependency issue is (AFAIU) relevant only during the process of doing the libc++ conversion on older Mac OS X versions, something that probably already involves a number of "special case derogations" to general dogma, no?
(BTW, sorry about the reverted commit, somehow I missed that change, will address it when I get back to my Mac.)
|
Either way, if Jeremy doesn't mind keeping the buildit script alive I can of course not argue with that.
I do think however that the possibility to build libc++ with newer clang versions should count for something, so in that case one could also consider a construction like
```
# check if we have a (bootstrapped) environment providing both libc++ and CMake
set use_cmake [[file exists /usr/lib/libc++.1.dylib] && [file exists $prefix/bin/cmake]]
```
or
```
# check if we have a (bootstrapped) environment providing both libc++ and CMake
if {[file exists /usr/lib/libc++.1.dylib] && [file exists $prefix/bin/cmake]} {
variant use_cmake description {Build libc++ using cmake, gives more choice of compilers} {}
}
set use_cmake [[variant_exists use_cmake] && [variant_isset use_cmake]]
```
|
that is, when a libc++ copy is already installed. Otherwise, fall back to the bootstrap build using the buildit script.
Travis Build #2362 Errored. Lint results
Port libcxx fail on xcode9.3. Log |
Travis Build #2363 Errored. Lint results
Port libcxx fail on xcode9.3. Log |
Very cool stuff, and what you're doing looks like it might be quite useful to me in building a version of libcxx with gcc6 that uses the gcc ABI https://libcxx.llvm.org/docs/BuildingLibcxx.html#libc-abi-feature-options to support thread_local with complex destructors on 10.4 to 10.6. Your ideas about checking for cmake and libc++ and enabling features certainly look like they could work. I also have a personal port of cmake 3.9 that does not require c++11 for bootstrapping purposes, exactly as you outlined. The real question comes down to what's reasonable to put out for general use in a publicized and supported repo, and what's more of a niche product that people who want it would have to seek out. I suspect, in the end, that will be the thing that keeps us going along with the buildit script and our current system, although I do enjoy this exploration you've done, and I personally plan to try using it. |
Very cool stuff, and what you're doing looks like it might be quite useful to me in building a version of libcxx with gcc6 that uses the gcc ABI <https://libcxx.llvm.org/docs/BuildingLibcxx.html#libc-abi-feature-options> to support thread_local with complex destructors on 10.4 to 10.6.
For that you might also be interested in my patches to port:gcc7 to make it support -stdlib=libc++ (https://github.com/RJVB/macstrop/tree/master/lang/gcc7; there's a simpler version of the same for port:gcc6 and I expect that the more elaborate gcc7 patch should apply to gcc6 with minimal modification).
It sounds like you're trying to achieve what I have been tinkering with on Linux. Interestingly it appears to be possible to mix and match libstdc++ and libc++ when libc++ is built against the libstdc++ ABI. I've built several complex Qt5/KF5 applications against such a libc++ and they work just fine even if most of the libraries they use are built against libstdc++.
Here's my Linux port for libc++; it also installs the libc++ headers, everything (naughtily) into /usr instead of $prefix: https://github.com/RJVB/lnxports/tree/master/lang/libcxx . Maybe it has something you can use.
I've been considering to take the +runtime variant a step further and figure out a way to let the linker use my $prefix/lib/libc++.dylib (on Mac) rather than the system one. In practice I think that only has a real interest when using `g++ -stdlib=libc++` (no need to pull in GCC's libsupc++ to resolve a few symbols that where only added to libc++ 4). For (almost) all other uses it's good enough to inject the custom libc++.
A question: does one need to install the matching "c++/v1" header files when actually linking against $prefix/lib/libc++.dylib, or will clang++-mp-X continue to use its one copy from $prefix/libexec/llvm-X anyway? This seems relevant to the libc++ conversion also, yet the port doesn't include the headers in the payload tarball.
|
The real question comes down to what's reasonable to put out for general use in a publicized and supported repo,
and what's more of a niche product that people who want it would have to seek out.
Fair enough, but that probably applies mostly to the +runtime variant. Which is why I've indicated that I'm not adamant to get that one in. The rest is mostly transparent to end users. Being able to use the official build system with its options use a compiler as new as clang 5.0.1 doesn't seem a niche thing to me though.
|
Did anyone test the changes on any other system besides 10.9? (The interesting ones are those below 10.9 in any case.) In any case this port is so central that it really needs to be well-tested and certainly approved by Jeremy. |
It's really unfortunate that Jeremy did not yet had a chance to review this, but while it would be ok for any random port to call this maintainer timeout, this really really really really needs to be tested on every single system from 10.5 to 10.8 at least. In particular, what happens when cmake gets a revbump at the same time as this port is outdated, and whether the bootstrapping instructions for |
++
And there's no hurry, either.
what happens when cmake gets a revbump at the same time as this port is outdated
With luck "base" will first update cmake (EDIT: and that's indeed what seems to happen when I activate an older libcxx version and port:cmake is outdated), but in practice the order shouldn't matter here once the system has been bootstrapped. Either way a port:libcxx upgrade will only upgrade the tarball; the user will still have to make the final step, no? That effectively postpones the libcxx upgrade until the end.
Testing this should be possible on any system, by re-activating an older cmake port version.
|
I'm not going to disagree, but if @jeremyhu doesn't have time for MacPorts these days we're going to need to get a new domain expert for this stuff. |
Jeremy is irreplaceable. He has a super crazy busy job as an Apple compiler engineer, but comes back to help us out when he has lulls in his schedule (eg. Xmas holidays, for example). We most definitely certainly do not want to ever piss him off enough to make him abandon MacPorts. This PR can sit here for years, rather than lose Jeremy. It's an interesting idea, but it's not key or central to MacPorts in any way. |
I don't want to piss him (or anyone else!) off. I'm just noting that it's problematic if there's one person alone who knows how to do something. What if he's hit by a bus? |
@RJVB why did you close the PR? |
Spring cleaning of projects too unlikely to go anywhere?
|
@pmetzger: It's probably similar as if you get hit by a bus: we end up with 1000+ open pull requests again (unless you are able to find your candidate for replacement right now). It's not that @jeremyhu is the only one capable of reviewing this PR. I would gladly commit it if someone does sufficient testing. Testing on a single post-10.8 OS is not sufficient though. We need someone willing to go through the complete bootstrapping process on at least 10.6, 10.7, 10.8. Closing the PR does not really help either. |
I think that if anyone were really motivated to give this thorough testing across the board, s/he would have spoken up by now. I'm guessing that the bootstrap issue makes it infeasible to do something like commit this on a testing branch and then let the CI system serve up any errors that it encounters.
That said, would it be possible to do a cmake 3.9.x universal build (probably using static libraries, or see below) that runs on all target platforms that do not have a system libc++, and that 's configured to run from some temporary location (which could be a mounted dmg)? If so, the port could include a bootstrap procedure for those platforms that fetches that cmake build and uses it for the first build. If a universal build is not feasible there's always the possibility to host multiple images and fetch the appropriate one.
Kitware themselves only provide an x86_64 build (https://cmake.org/files/v3.9/cmake-3.9.6-Darwin-x86_64.dmg) so that doesn't really help.
This probably isn't really any less complex than backporting the buildit script and using that, but it should be much more of a single-shot implementation effort needing little maintenance during libc++ upgrades. Backporting the buildit script OTOH is something that requires time at each update and may (will...) even become more and more difficult as the sources continue to evolve.
I'm willing to try to make a PoC implementation of such a bootstrap procedure, if someone else takes care of the bootstrap cmake build.
About such a cmake bootstrap build:
- install a MacPorts copy with prefix=/Volumes/cmake_bootstrap
- do a minimal cmake 3.9.x build+install into that prefix
- prune that prefix
- convert it to a dmg?
That should get all required libraries in the right location automatically, without need for static linking.
|
This is a combined upgrade and overhaul of port:libcxx :
through a restored and patched buildit script.
to test and run applications with a current libc++/libc++abi version
via DYLD_INSERT_LIBRARIES .
Description
One advantage of this way of building is that there is no stdlib.h bug and one can use recent port:clang versions (tested with Clang 5.0.1).
I have left the legacy build code and patches accessible by unsetting the
use_cmake
flag, if anyone wants to compare the builds (or if there's a reason to keep the legacy mode on older OS X systems).Type(s)
Tested on
Mac OS X 10.9.5
MacPorts clang 5.0.1