data.table does not compile on OSX 10.13

[hannes]

It seems the -fopenmp option to clang now causes an error in Apple LLVM 9.

> devtools::install_github("Rdatatable/data.table")
...
clang: error: unsupported option '-fopenmp'

> sessionInfo()
R version 3.4.2 (2017-09-28)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] crayon_1.3.2         digest_0.6.12        withr_2.0.0         
 [4] assertthat_0.2.0     R6_2.2.2             git2r_0.19.0        
 [7] httr_1.3.1           debugme_1.0.2        rlang_0.1.2.9000    
[10] curl_3.0             callr_1.0.0.9000     devtools_1.13.3.9000
[13] tools_3.4.2          pkgload_0.0.0.9000   compiler_3.4.2      
[16] processx_2.0.0.1     pkgbuild_0.0.0.9000  memoise_1.1.0       
[19] knitr_1.17           usethis_0.0.0.9000  

clang -v
Apple LLVM version 9.0.0 (clang-900.0.37)
Target: x86_64-apple-darwin17.0.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[renkun-ken]

See https://github.com/coatless/r-macos-clang

[mattdowle]

Also, CRAN provides a binary for MacOS and that now supports OpenMP, so you shouldn't need to compile. If you do need to compile, then our Mac Instructions have been reported to work. Please reopen if there's a problem with the instructions.

[smcinerney]

a) CRAN only seems to have MacOS binaries up to R 3.2, not 3.3 or 3.4?

options(install.packages.check.source = "no")
install.packages('data.table', type='mac.binary')
Warning: unable to access index for repository https://cran.cnr.berkeley.edu/bin/macosx/contrib/3.4:
cannot open URL 'https://cran.cnr.berkeley.edu/bin/macosx/contrib/3.4/PACKAGES'
Warning message:
package ‘data.table’ is not available (as a binary package for R version 3.4.2)

b) Then I tried manually downloading the binary from https://cran.r-project.org/bin/macosx/el-capitan/contrib/3.4/data.table_1.10.4-3.tgz and R CMD INSTALL data.table_1.10.4-3.tgz. But it fails on an unresolved dependency on libomp:

Loading required package: data.table
Error: package or namespace load failed for ‘data.table’ in dyn.load(file, DLLpath = DLLpath, ...):
unable to load shared object '/opt/anaconda3/lib/R/library/data.table/libs/datatable.so':
dlopen(/opt/anaconda3/lib/R/library/data.table/libs/datatable.so, 6): Library not loaded: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libomp.dylib
Referenced from: /opt/anaconda3/lib/R/library/data.table/libs/datatable.so
Reason: image not found

b2) I do have a /opt/anaconda3/lib/libomp.dylib. Was I supposed to manually supply it in R CMD INSTALL -l ..., or edit it in into ~/.R/Makevars ?

b3) libomp has been deprecated from brew, btw.

[mattdowle]

a) CRAN only seems to have MacOS binaries up to R 3.2, not 3.3 or 3.4?

If the binary does not even exist, then this is question for CRAN. Maybe try a different mirror? Let's resolve that first, before trying to compile.

[smcinerney]

But that's part of what I'm trying to tell you. Seems the bin/macosx/contrib/ tree under all CRAN mirrors is either nonexistent or only goes up to 3.2. No 3.3 or 3.4. Contrary to [the Mac Instructions](Mac Instructions) implying the current mac.binary is downloadable with install.packages() from CRAN.

On checking multiple mirrors, the root-cause is the issue is the (2017?) path fork into having subtrees ./el-capitan, ./mavericks. install.packages() does not know about that. So it's generic breakage of install.packages(..., type='mac.binary') against CRAN.
Workaround: install.packages(..., type='mac.binary') is pretty brittle, just manually download the package .tgz with browser or curl.
Either way, if you want to keep recommending install.packages(..., type='mac.binary') in data.table instructions, you need a caveat about how brittle it is. (It also suffers from the well-known internet-proxy issue under RStudio, among many other things). All this is preventing MacOS users from using your excellent package.

Kludging the el-capitan/mavericks subtree path is also not documented in the R for Mac OS X FAQ. Where is the official CRAN documentation on that split? (so I can mention it to them, and maybe raise a bug on install.packages)?

b) is also an issue, esp. under Anaconda. I did resolve it by installing brew llvm and editing ~/.R/Makevars. But I'm trying to flag the level of install pain caused by CRAN breaking the mac.binary paths, and that it will prevent people using data.table, at least with openmp.

[jangorecki]

@smcinerney I would start with reporting lack of documentation about that split to R core.

[smcinerney]

@jangorecki : I certainly will. Just letting you know inter alia, unless I missed some internal discussion on R-help list, the fact the split happened 2 years ago and noone much flagged it as breaking install.packages() on mac binaries seems to show people stopped using CRAN for installing mac binaries years ago. FYI. After we get that sorted we should revisit data.table's Mac instructions.

[mattdowle]

@smcinerney Any luck with CRAN on this? I have a Mac but have installed Linux on it. I don't use MacOS. The info about 'mac.binary' sounds very plausible, I just don't know. We rely on help from the community to solve it for Mac users. Please feel free to edit our wiki directly (there are no restrictions).

[jaapwalhout]

If it is of any help:

Updating data.table on macOS 1.12.6 and using R 3.4.4 gives me the following message:

> install.packages("data.table")

  There is a binary version available but the source version is
  later:
             binary source needs_compilation
data.table 1.10.4-3 1.11.2              TRUE

Do you want to install from sources the package which needs compilation?
y/n: 

[mattdowle]

Thanks @jaapwalhout. We'll just have to wait for CRAN MacOS on oldrel to happen. I mentioned it to them a few days ago.

[mattdowle]

I sent an email to cran@r-project.org about this earlier today. For others finding this in future, CRAN checks page for data.table shows that r-oldrel-osx-x86_64 is a month stale. old-rel means R 3.4.* MacOS binary are usually the slowest, taking a week or two. But 1 month is much more than usual.

[yiqinfu]

Hope this is relevant. I followed all the steps outlined in the installation doc but still encountered a stdio.h file not found error. That issue was fixed by installing the package at
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -- hope someone will find this helpful.

[mattdowle]

@yiqinfu Fantastic. Please could you add this info to the installations page. The wiki is open access and anybody can change it. You're in the best position to update it correctly. Just to double-check we mean this page: https://github.com/Rdatatable/data.table/wiki/Installation

[yiqinfu]

@mattdowle Just did! Also tweaked the formatting of the doc. Hope it's clear! Thank you for maintaining the package.

[renkun-ken]

@yiqinfu The package solves the problem for macOS 10.14!

@mattdowle Does data.table prefer llvm to clang6 suggested at https://cran.r-project.org/bin/macosx/tools? I feel using clang6 toolchain is much easier to setup and work best for all packages. Just download clang-6.0.0.pkg and gfortran-6.1.pkg, and write a ~/.R/Makevars and everything works perfectly for all package compilation.