macOS gfortran binary note update in FAQ

[coatless]

:\

Important note: R 3.4.0 El Capitan binaries are using Clang 4.0.0 and GNU Fortran 6.1 to provide OpenMP parallelization support and C++17 standard features. If you want to compile R packages from sources, please download GNU Fortran binary from the official GNU Fortran Binaries page - in particular OS X 10.11 gfortran 6.1. We are also providing Clang 4.0.0 binaries for OS X 10.11 and higher in our libs directory (note that the offical Clang 4.0.0 binaries only support macOS 10.12) and will provide an Apple Installer package here soon. For the time being you may want to either create ~/.R/Makevars such as

https://cran.r-project.org/bin/macosx/tools/

[eddelbuettel]

I kinda knew this was brewing but forgot about it leading up to R 3.4.0.

So what is currently borked, and what currently works?

[eddelbuettel]

I kinda knew this was brewing but forgot about it leading up to R 3.4.0.

So what is currently borked, and what currently works?

[kevinushey]

FWIW, it looks like old workflows for building from sources still work fine. The main thing that scares me is now R 3.4.0 is shipping its own copy of the C++ standard library, so R packages that use system libraries built with the system compiler will effectively end up running against two copies of libc++. This strikes me as ... somewhat insane, to be honest, but I haven't bumped into any problems in my initial testing.

Users who want to build against the same compiler toolchain as used to build R can follow the instructions at https://cran.r-project.org/bin/macosx/tools/, but I don't think this is strictly necessary.

[kevinushey]

FWIW, it looks like old workflows for building from sources still work fine. The main thing that scares me is now R 3.4.0 is shipping its own copy of the C++ standard library, so R packages that use system libraries built with the system compiler will effectively end up running against two copies of libc++. This strikes me as ... somewhat insane, to be honest, but I haven't bumped into any problems in my initial testing.

Users who want to build against the same compiler toolchain as used to build R can follow the instructions at https://cran.r-project.org/bin/macosx/tools/, but I don't think this is strictly necessary.

[coatless]

Seems like nothing is broken per say. Just a lack of features are present under the current recommended approach. I'll need to look into this a bit more later today.

[coatless]

Seems like nothing is broken per say. Just a lack of features are present under the current recommended approach. I'll need to look into this a bit more later today.

[kevinushey]

Right -- IIUC the main deficiency is that Apple Clang still does not ship with OpenMP support; R-Core was tired of waiting on Apple for OpenMP and so decided to just build R with LLVM clang.

[kevinushey]

Right -- IIUC the main deficiency is that Apple Clang still does not ship with OpenMP support; R-Core was tired of waiting on Apple for OpenMP and so decided to just build R with LLVM clang.

[coatless]

So, I've toyed around with the new setup. I think it would be beneficial to restructure the documentation so that all macOS users are directed to clang4 and gfortran 6.1 instead of maintaining highly specific sections of documentation that emphasize tools without OpenMP support. To that end...

• Emphasize the short installer I wrote for the clang4 pre-built binaries put out by CRAN's macOS maintainers.
  • I sent an email to Simon about using it potentially as the installer and got a response indicating it might be possible with a couple changes (e.g. users must be the ones to modify ~/.R/Makevars 😢 ).
  • Details about the construction can be found here: https://github.com/coatless/r-macos-clang.
• Rewrite the gfortran fixed binaries entry to emphasize gfortran 6.1.
  • The caveat is versions between R 3.0.0 -- R 3.3.* require the addition of an FLIBS flag as it looks for gfortran 4.8.2 in /usr/local/.
• Change all OS X references to macOS (unless referring to an OS name, e.g. OS X Mavericks vs. macOS Sierra)

[coatless]

So, I've toyed around with the new setup. I think it would be beneficial to restructure the documentation so that all macOS users are directed to clang4 and gfortran 6.1 instead of maintaining highly specific sections of documentation that emphasize tools without OpenMP support. To that end...

• Emphasize the short installer I wrote for the clang4 pre-built binaries put out by CRAN's macOS maintainers.
  • I sent an email to Simon about using it potentially as the installer and got a response indicating it might be possible with a couple changes (e.g. users must be the ones to modify ~/.R/Makevars 😢 ).
  • Details about the construction can be found here: https://github.com/coatless/r-macos-clang.
• Rewrite the gfortran fixed binaries entry to emphasize gfortran 6.1.
  • The caveat is versions between R 3.0.0 -- R 3.3.* require the addition of an FLIBS flag as it looks for gfortran 4.8.2 in /usr/local/.
• Change all OS X references to macOS (unless referring to an OS name, e.g. OS X Mavericks vs. macOS Sierra)

[eddelbuettel]

Ok, I'll take the long view here. Rcpp is coming up on its 10th anniversary. Pdf vignettes are meant to be long-lasting documentation, and not something we have revise at every full moon just because some less-than-fully-cooked engineer in Cupertino came up with another braindead scheme for macOS. Or whatever it is called then.

So if we cannot squash this in the FAQ (where 'frequently' is meant to refer to how often questions are asked, rather than how often answers are replaced) then maybe we should just refer to the wiki?

[eddelbuettel]

Ok, I'll take the long view here. Rcpp is coming up on its 10th anniversary. Pdf vignettes are meant to be long-lasting documentation, and not something we have revise at every full moon just because some less-than-fully-cooked engineer in Cupertino came up with another braindead scheme for macOS. Or whatever it is called then.

So if we cannot squash this in the FAQ (where 'frequently' is meant to refer to how often questions are asked, rather than how often answers are replaced) then maybe we should just refer to the wiki?

[coatless]

Regarding the long view, there is not an Rtools equivalent for macOS that automatically builds the development environment required for macOS. Unfortunately, a lot of it is piece meal (install Xcode CLI, install gfortran, and now install a different compiler). Though, we're rapidly approaching that step with the new clang4 installer package. In fact, we would only need to include the gfortran 6.1 binary... Mmm...

With this being said, the documentation regarding macOS should at the very least be updated to stress:

1. macOS page of cran https://cran.r-project.org/bin/macosx/tools/
2. ~/.R/Makevars variables: CC, CXX, LDFLAGS, and FLIBS

Generally speaking, the expected paths of system tools of the macOS build are influx due to Apple not shipping the Xcode bundled clang with OpenMP.

[coatless]

Regarding the long view, there is not an Rtools equivalent for macOS that automatically builds the development environment required for macOS. Unfortunately, a lot of it is piece meal (install Xcode CLI, install gfortran, and now install a different compiler). Though, we're rapidly approaching that step with the new clang4 installer package. In fact, we would only need to include the gfortran 6.1 binary... Mmm...

With this being said, the documentation regarding macOS should at the very least be updated to stress:

1. macOS page of cran https://cran.r-project.org/bin/macosx/tools/
2. ~/.R/Makevars variables: CC, CXX, LDFLAGS, and FLIBS

Generally speaking, the expected paths of system tools of the macOS build are influx due to Apple not shipping the Xcode bundled clang with OpenMP.

[coatless]

Actually, thinking more about it...

We can do an Rtools esq like build for macOS.

In a preinstall script we can put into place Xcode CLI tools installation:

touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress;
PROD=$(softwareupdate -l |
  grep "\*.*Command Line" |
  head -n 1 | awk -F"*" '{print $2}' |
  sed -e 's/^ *//' |
  tr -d '\n')
softwareupdate -i "$PROD" --verbose;

c.f. https://apple.stackexchange.com/questions/107307/how-can-i-install-the-command-line-tools-completely-from-the-command-line

Then, we can use the clang4 component build.

Finally, we can link the gfortran 6.1 package using productbuild...

# Analyze the separate package builds
productbuild --synthesize \
    --package clang-r.pkg --package gfortran61.pkg \
    Distribution.xml 

## Changes to Distribution.xml

# Rebuild the installer
productbuild --distribution ./Distribution.xml \
    --package-path . \
    ./Installer.pkg

@kevinushey thoughts?

[coatless]

Actually, thinking more about it...

We can do an Rtools esq like build for macOS.

In a preinstall script we can put into place Xcode CLI tools installation:

touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress;
PROD=$(softwareupdate -l |
  grep "\*.*Command Line" |
  head -n 1 | awk -F"*" '{print $2}' |
  sed -e 's/^ *//' |
  tr -d '\n')
softwareupdate -i "$PROD" --verbose;

c.f. https://apple.stackexchange.com/questions/107307/how-can-i-install-the-command-line-tools-completely-from-the-command-line

Then, we can use the clang4 component build.

Finally, we can link the gfortran 6.1 package using productbuild...

# Analyze the separate package builds
productbuild --synthesize \
    --package clang-r.pkg --package gfortran61.pkg \
    Distribution.xml 

## Changes to Distribution.xml

# Rebuild the installer
productbuild --distribution ./Distribution.xml \
    --package-path . \
    ./Installer.pkg

@kevinushey thoughts?

[eddelbuettel]

Yes, we will probably have a July release per the usual schedule so by all means let's refresh this 'FUA' (= 'Frequently Updated Answer') to be current.

[eddelbuettel]

Yes, we will probably have a July release per the usual schedule so by all means let's refresh this 'FUA' (= 'Frequently Updated Answer') to be current.

[kevinushey]

This all seems fine to me. That said, I'm not a big fan of R's decision to build with a non-default toolchain on macOS, and I don't think the onus is on Rcpp to re-document how that toolchain should be used (nor to provide installers for it).

Using the macOS system default compilers mostly just works; although one notable exception is that exception handling may not work across .sos built with different toolchains due to the different C++ standard library implementations used (jeroen/V8#37). But V8 is unique here in that the generated library links to libstdc++ rather than libc++, which is non-standard as far as modern software on macOS goes.

[kevinushey]

This all seems fine to me. That said, I'm not a big fan of R's decision to build with a non-default toolchain on macOS, and I don't think the onus is on Rcpp to re-document how that toolchain should be used (nor to provide installers for it).

Using the macOS system default compilers mostly just works; although one notable exception is that exception handling may not work across .sos built with different toolchains due to the different C++ standard library implementations used (jeroen/V8#37). But V8 is unique here in that the generated library links to libstdc++ rather than libc++, which is non-standard as far as modern software on macOS goes.

[coatless]

I'm going to submit a PR that sunsets the macOS documentation in favor of pointing macOS users to the R Installation and Administration manual. In particular...

• Redirect compiler instructions to: https://cran.r-project.org/doc/manuals/r-release/R-admin.html#macOS-packages

• Redirect gfortran to: https://cran.r-project.org/doc/manuals/r-release/R-admin.html#macOS

I'm also going to switch OS X references to macOS during this revision (to be consistent with R documentation).

[coatless]

I'm going to submit a PR that sunsets the macOS documentation in favor of pointing macOS users to the R Installation and Administration manual. In particular...

• Redirect compiler instructions to: https://cran.r-project.org/doc/manuals/r-release/R-admin.html#macOS-packages

• Redirect gfortran to: https://cran.r-project.org/doc/manuals/r-release/R-admin.html#macOS

I'm also going to switch OS X references to macOS during this revision (to be consistent with R documentation).

[coatless]

Just in... We now have a wonderful R toolchain installer for macOS that downloads, installs, and configures the xcode-cli, clang4, and gfortran binaries!

https://github.com/coatless/r-macos-rtools

[coatless]

Just in... We now have a wonderful R toolchain installer for macOS that downloads, installs, and configures the xcode-cli, clang4, and gfortran binaries!

https://github.com/coatless/r-macos-rtools