New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

macOS gfortran binary note update in FAQ #682

Closed
coatless opened this Issue Apr 25, 2017 · 12 comments

Comments

Projects
None yet
3 participants
@coatless
Contributor

coatless commented Apr 25, 2017

:\

Important note: R 3.4.0 El Capitan binaries are using Clang 4.0.0 and GNU Fortran 6.1 to provide OpenMP parallelization support and C++17 standard features. If you want to compile R packages from sources, please download GNU Fortran binary from the official GNU Fortran Binaries page - in particular OS X 10.11 gfortran 6.1. We are also providing Clang 4.0.0 binaries for OS X 10.11 and higher in our libs directory (note that the offical Clang 4.0.0 binaries only support macOS 10.12) and will provide an Apple Installer package here soon. For the time being you may want to either create ~/.R/Makevars such as

https://cran.r-project.org/bin/macosx/tools/

@eddelbuettel

This comment has been minimized.

Show comment
Hide comment
@eddelbuettel

eddelbuettel Apr 25, 2017

Member

I kinda knew this was brewing but forgot about it leading up to R 3.4.0.

So what is currently borked, and what currently works?

Member

eddelbuettel commented Apr 25, 2017

I kinda knew this was brewing but forgot about it leading up to R 3.4.0.

So what is currently borked, and what currently works?

@kylebmetrum kylebmetrum referenced this issue Apr 25, 2017

Closed

R 3.4.0 #224

@kevinushey

This comment has been minimized.

Show comment
Hide comment
@kevinushey

kevinushey Apr 25, 2017

Contributor

FWIW, it looks like old workflows for building from sources still work fine. The main thing that scares me is now R 3.4.0 is shipping its own copy of the C++ standard library, so R packages that use system libraries built with the system compiler will effectively end up running against two copies of libc++. This strikes me as ... somewhat insane, to be honest, but I haven't bumped into any problems in my initial testing.

Users who want to build against the same compiler toolchain as used to build R can follow the instructions at https://cran.r-project.org/bin/macosx/tools/, but I don't think this is strictly necessary.

Contributor

kevinushey commented Apr 25, 2017

FWIW, it looks like old workflows for building from sources still work fine. The main thing that scares me is now R 3.4.0 is shipping its own copy of the C++ standard library, so R packages that use system libraries built with the system compiler will effectively end up running against two copies of libc++. This strikes me as ... somewhat insane, to be honest, but I haven't bumped into any problems in my initial testing.

Users who want to build against the same compiler toolchain as used to build R can follow the instructions at https://cran.r-project.org/bin/macosx/tools/, but I don't think this is strictly necessary.

@coatless

This comment has been minimized.

Show comment
Hide comment
@coatless

coatless Apr 25, 2017

Contributor

Seems like nothing is broken per say. Just a lack of features are present under the current recommended approach. I'll need to look into this a bit more later today.

Contributor

coatless commented Apr 25, 2017

Seems like nothing is broken per say. Just a lack of features are present under the current recommended approach. I'll need to look into this a bit more later today.

@kevinushey

This comment has been minimized.

Show comment
Hide comment
@kevinushey

kevinushey Apr 25, 2017

Contributor

Right -- IIUC the main deficiency is that Apple Clang still does not ship with OpenMP support; R-Core was tired of waiting on Apple for OpenMP and so decided to just build R with LLVM clang.

Contributor

kevinushey commented Apr 25, 2017

Right -- IIUC the main deficiency is that Apple Clang still does not ship with OpenMP support; R-Core was tired of waiting on Apple for OpenMP and so decided to just build R with LLVM clang.

@coatless

This comment has been minimized.

Show comment
Hide comment
@coatless

coatless Jun 25, 2017

Contributor

So, I've toyed around with the new setup. I think it would be beneficial to restructure the documentation so that all macOS users are directed to clang4 and gfortran 6.1 instead of maintaining highly specific sections of documentation that emphasize tools without OpenMP support. To that end...

  • Emphasize the short installer I wrote for the clang4 pre-built binaries put out by CRAN's macOS maintainers.
    • I sent an email to Simon about using it potentially as the installer and got a response indicating it might be possible with a couple changes (e.g. users must be the ones to modify ~/.R/Makevars 😢 ).
    • Details about the construction can be found here: https://github.com/coatless/r-macos-clang.
  • Rewrite the gfortran fixed binaries entry to emphasize gfortran 6.1.
    • The caveat is versions between R 3.0.0 -- R 3.3.* require the addition of an FLIBS flag as it looks for gfortran 4.8.2 in /usr/local/.
  • Change all OS X references to macOS (unless referring to an OS name, e.g. OS X Mavericks vs. macOS Sierra)
Contributor

coatless commented Jun 25, 2017

So, I've toyed around with the new setup. I think it would be beneficial to restructure the documentation so that all macOS users are directed to clang4 and gfortran 6.1 instead of maintaining highly specific sections of documentation that emphasize tools without OpenMP support. To that end...

  • Emphasize the short installer I wrote for the clang4 pre-built binaries put out by CRAN's macOS maintainers.
    • I sent an email to Simon about using it potentially as the installer and got a response indicating it might be possible with a couple changes (e.g. users must be the ones to modify ~/.R/Makevars 😢 ).
    • Details about the construction can be found here: https://github.com/coatless/r-macos-clang.
  • Rewrite the gfortran fixed binaries entry to emphasize gfortran 6.1.
    • The caveat is versions between R 3.0.0 -- R 3.3.* require the addition of an FLIBS flag as it looks for gfortran 4.8.2 in /usr/local/.
  • Change all OS X references to macOS (unless referring to an OS name, e.g. OS X Mavericks vs. macOS Sierra)
@eddelbuettel

This comment has been minimized.

Show comment
Hide comment
@eddelbuettel

eddelbuettel Jun 25, 2017

Member

Ok, I'll take the long view here. Rcpp is coming up on its 10th anniversary. Pdf vignettes are meant to be long-lasting documentation, and not something we have revise at every full moon just because some less-than-fully-cooked engineer in Cupertino came up with another braindead scheme for macOS. Or whatever it is called then.

So if we cannot squash this in the FAQ (where 'frequently' is meant to refer to how often questions are asked, rather than how often answers are replaced) then maybe we should just refer to the wiki?

Member

eddelbuettel commented Jun 25, 2017

Ok, I'll take the long view here. Rcpp is coming up on its 10th anniversary. Pdf vignettes are meant to be long-lasting documentation, and not something we have revise at every full moon just because some less-than-fully-cooked engineer in Cupertino came up with another braindead scheme for macOS. Or whatever it is called then.

So if we cannot squash this in the FAQ (where 'frequently' is meant to refer to how often questions are asked, rather than how often answers are replaced) then maybe we should just refer to the wiki?

@coatless

This comment has been minimized.

Show comment
Hide comment
@coatless

coatless Jun 25, 2017

Contributor

Regarding the long view, there is not an Rtools equivalent for macOS that automatically builds the development environment required for macOS. Unfortunately, a lot of it is piece meal (install Xcode CLI, install gfortran, and now install a different compiler). Though, we're rapidly approaching that step with the new clang4 installer package. In fact, we would only need to include the gfortran 6.1 binary... Mmm...

With this being said, the documentation regarding macOS should at the very least be updated to stress:

  1. macOS page of cran https://cran.r-project.org/bin/macosx/tools/
  2. ~/.R/Makevars variables: CC, CXX, LDFLAGS, and FLIBS

Generally speaking, the expected paths of system tools of the macOS build are influx due to Apple not shipping the Xcode bundled clang with OpenMP.

Contributor

coatless commented Jun 25, 2017

Regarding the long view, there is not an Rtools equivalent for macOS that automatically builds the development environment required for macOS. Unfortunately, a lot of it is piece meal (install Xcode CLI, install gfortran, and now install a different compiler). Though, we're rapidly approaching that step with the new clang4 installer package. In fact, we would only need to include the gfortran 6.1 binary... Mmm...

With this being said, the documentation regarding macOS should at the very least be updated to stress:

  1. macOS page of cran https://cran.r-project.org/bin/macosx/tools/
  2. ~/.R/Makevars variables: CC, CXX, LDFLAGS, and FLIBS

Generally speaking, the expected paths of system tools of the macOS build are influx due to Apple not shipping the Xcode bundled clang with OpenMP.

@coatless

This comment has been minimized.

Show comment
Hide comment
@coatless

coatless Jun 25, 2017

Contributor

Actually, thinking more about it...

We can do an Rtools esq like build for macOS.

In a preinstall script we can put into place Xcode CLI tools installation:

touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress;
PROD=$(softwareupdate -l |
  grep "\*.*Command Line" |
  head -n 1 | awk -F"*" '{print $2}' |
  sed -e 's/^ *//' |
  tr -d '\n')
softwareupdate -i "$PROD" --verbose;

c.f. https://apple.stackexchange.com/questions/107307/how-can-i-install-the-command-line-tools-completely-from-the-command-line

Then, we can use the clang4 component build.

Finally, we can link the gfortran 6.1 package using productbuild...

# Analyze the separate package builds
productbuild --synthesize \
    --package clang-r.pkg --package gfortran61.pkg \
    Distribution.xml 

## Changes to Distribution.xml

# Rebuild the installer
productbuild --distribution ./Distribution.xml \
    --package-path . \
    ./Installer.pkg

@kevinushey thoughts?

Contributor

coatless commented Jun 25, 2017

Actually, thinking more about it...

We can do an Rtools esq like build for macOS.

In a preinstall script we can put into place Xcode CLI tools installation:

touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress;
PROD=$(softwareupdate -l |
  grep "\*.*Command Line" |
  head -n 1 | awk -F"*" '{print $2}' |
  sed -e 's/^ *//' |
  tr -d '\n')
softwareupdate -i "$PROD" --verbose;

c.f. https://apple.stackexchange.com/questions/107307/how-can-i-install-the-command-line-tools-completely-from-the-command-line

Then, we can use the clang4 component build.

Finally, we can link the gfortran 6.1 package using productbuild...

# Analyze the separate package builds
productbuild --synthesize \
    --package clang-r.pkg --package gfortran61.pkg \
    Distribution.xml 

## Changes to Distribution.xml

# Rebuild the installer
productbuild --distribution ./Distribution.xml \
    --package-path . \
    ./Installer.pkg

@kevinushey thoughts?

@eddelbuettel

This comment has been minimized.

Show comment
Hide comment
@eddelbuettel

eddelbuettel Jun 25, 2017

Member

Yes, we will probably have a July release per the usual schedule so by all means let's refresh this 'FUA' (= 'Frequently Updated Answer') to be current.

Member

eddelbuettel commented Jun 25, 2017

Yes, we will probably have a July release per the usual schedule so by all means let's refresh this 'FUA' (= 'Frequently Updated Answer') to be current.

@kevinushey

This comment has been minimized.

Show comment
Hide comment
@kevinushey

kevinushey Jun 25, 2017

Contributor

This all seems fine to me. That said, I'm not a big fan of R's decision to build with a non-default toolchain on macOS, and I don't think the onus is on Rcpp to re-document how that toolchain should be used (nor to provide installers for it).

Using the macOS system default compilers mostly just works; although one notable exception is that exception handling may not work across .sos built with different toolchains due to the different C++ standard library implementations used (jeroen/V8#37). But V8 is unique here in that the generated library links to libstdc++ rather than libc++, which is non-standard as far as modern software on macOS goes.

Contributor

kevinushey commented Jun 25, 2017

This all seems fine to me. That said, I'm not a big fan of R's decision to build with a non-default toolchain on macOS, and I don't think the onus is on Rcpp to re-document how that toolchain should be used (nor to provide installers for it).

Using the macOS system default compilers mostly just works; although one notable exception is that exception handling may not work across .sos built with different toolchains due to the different C++ standard library implementations used (jeroen/V8#37). But V8 is unique here in that the generated library links to libstdc++ rather than libc++, which is non-standard as far as modern software on macOS goes.

@coatless

This comment has been minimized.

Show comment
Hide comment
@coatless

coatless Aug 14, 2017

Contributor

I'm going to submit a PR that sunsets the macOS documentation in favor of pointing macOS users to the R Installation and Administration manual. In particular...

I'm also going to switch OS X references to macOS during this revision (to be consistent with R documentation).

Contributor

coatless commented Aug 14, 2017

I'm going to submit a PR that sunsets the macOS documentation in favor of pointing macOS users to the R Installation and Administration manual. In particular...

I'm also going to switch OS X references to macOS during this revision (to be consistent with R documentation).

@coatless

This comment has been minimized.

Show comment
Hide comment
@coatless

coatless Mar 23, 2018

Contributor

Just in... We now have a wonderful R toolchain installer for macOS that downloads, installs, and configures the xcode-cli, clang4, and gfortran binaries!

https://github.com/coatless/r-macos-rtools

Contributor

coatless commented Mar 23, 2018

Just in... We now have a wonderful R toolchain installer for macOS that downloads, installs, and configures the xcode-cli, clang4, and gfortran binaries!

https://github.com/coatless/r-macos-rtools

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment