Many packages are broken on Mac with CRAN version of Rcpp -- unless 10.11 SDK is used as on CRAN

[wch]

The CRAN version of httpuv will not build using the CRAN version of Rcpp, on Mac. Simple example:

install.packages("Rcpp")
install.packages("httpuv", type = "source")

(For more info, see rstudio/httpuv#260 (comment) and other comments in that issue.)

In that issue, I and others use the Mac system toolchain, but @kevinushey has tested with the CRAN recommended toolchain and ecountered the same problems.

I've learned that several other packages have similar problems: http://lists.r-forge.r-project.org/pipermail/rcpp-devel/2020-March/010412.html

In that discussion thread, it is claimed that these packages pass R CMD check on CRAN: http://lists.r-forge.r-project.org/pipermail/rcpp-devel/2020-March/010414.html

However, @jeroen has informed me that (A) CRAN does not have the resources to re-check downstream dependencies of new packages on Mac, and (B), it does not re-build the downstream dependencies either.

I don't know how I could confirm (A) from the public information that CRAN provides, but it is simple to confirm that (B) is true. These are the dates in https://www.r-project.org/nosvn/R.check/r-release-osx-x86_64/ :

• Rcpp_1.0.4.tgz: 2020-03-18
• httpuv_1.5.2.tgz: 2019-09-12
• dplyr_0.8.5.tgz: 2020-03-08

These dates show that Rcpp's downstream dependencies were NOT rebuilt on Mac.

So from all this, I feel fairly confident that the CRAN version of Rcpp does actually break its downstream dependencies on Mac. I know that the issue is fixed in the development version of Rcpp, and I think that this is very strong reason to release a new version of Rcpp to CRAN.

[coatless]

@wch agreed to the need to bump Rcpp to address downstream failures on macOS. I think the fix release was being held because new issues were being encountered after the 1.0.4 release. (c.f. R 3.3.x issue)

[wch]

@coatless Based on this comment, my understanding is that there is no plan for a patch release: hadley/purrrlyr#11 (comment)

[eddelbuettel]

I have been in communication with CRAN about this, and they

1. do not see breakage on macOS (on several of their sites / installations)
2. do not see a need for a patch release
   .

I actually contacted CRAN right after the first incremental dev release after 1.0.4 with a heads-up note --- but as everything appears to be fine on their end based on every communication I had with them.

That may of course change. Maybe, as you say, Simon is behind and if there is genuine breakage they too will see it and then request a new version I will most likely comply with any request of theirs.

[eddelbuettel]

This issue, I think, also fundamentally the same as #1046 which was closed by #1047.

[jjallaire]

I think if we know for sure that downstream dependencies are broken and we know that CRAN may not detect these breakages for some time it would be good to proactively release a patch.

[eddelbuettel]

Sure -- and in fact we have made four such patch release to the drat. Running the install.packages() command gets you the fourth, the other three are still in the same directory.

edd@rob:~$ Rscript -e 'install.packages("Rcpp", repos="https://rcppcore.github.io/drat")'
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://rcppcore.github.io/drat/src/contrib/Rcpp_1.0.4.4.tar.gz'
Content type 'application/gzip' length 2710270 bytes (2.6 MB)
==================================================
downloaded 2.6 MB

* installing *source* package ‘Rcpp’ ...
** using staged installation
** libs
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c api.cpp -o api.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c attributes.cpp -o attributes.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c barrier.cpp -o barrier.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c date.cpp -o date.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c module.cpp -o module.o
ccache g++ -I"/usr/share/R/include" -DNDEBUG -I../inst/include/    -fpic  -g -O3 -Wall -pipe -pedantic  -c rcpp_init.cpp -o rcpp_init.o
ccache g++ -Wl,-S -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o Rcpp.so api.o attributes.o barrier.o date.o module.o rcpp_init.o -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-Rcpp/00new/Rcpp/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (Rcpp)

The downloaded source packages are in
        ‘/tmp/RtmpIvgvlX/downloaded_packages’
edd@rob:~$ 

But I still defer to CRAN for the need of a CRAN release. So far they have not spoken up in need of one --- but in fact confirmed that everything looks ok to them.

Now, there could be a bias if in fact all macOS work is outsourced to the macOS maintainer. But we know that some of them (i.e. R Core, not just CRAN) develop on macOS too. I think it would bubble up.

[jjallaire]

Okay, but if we have better radar right now and we know that lots of R users are stuck unable to install packages I think we should be proactive. It's not about whether we defer to CRAN as I think CRAN is often notified proactively by maintainers about the need for a critical release before they manage to detect it. I think they appreciate the heads up so long as the fixes aren't trivial/spurious in terms of users affected and that they don't result from carelessness on the part of the maintainer.

Maybe the discussion then is really about the magnitude of the breakage (i.e. how many / what percent of Mac packages/users are affected). I don't have a good sense of that but perhaps others do.

[eddelbuettel]

What is preventing these users from issueing the one-line command below?

install.packages("Rcpp", repos="https://rcppcore.github.io/drat")

If the answer is "they cannot (generally) compile from source", could someone please provide me with a downloadable .tgz for the platform to support binary installations.

[jjallaire]

They have to know to issue that command in the first place. If an analyst in the wild gets a big stream of compilation errors they often are totally mystified and aren't even aware of the possibility that they could install patched versions from source (let alone where to get them, esp. if the error is from a downstream dependency of the package that failed).

[wch]

One more possible issue: If packages can't compile against CRAN Rcpp on Mac, then that also causes problems for releasing a new version of these downstream dependencies.

[eddelbuettel]

Yes. One of the first things I tried when the initial bug report came in was a rhub check on macOS for an Rcpp dependee -- it built fine. (May have been a cache artefact.) But my understanding was all this time that it a) it is not widespread and b) easy to fix via the drat. I will circle back with Vienna and mention this discussion. Thank you all for the input so far.

[langmm]

I just wanted to add that I am also experiencing this issue and would request a bump. While the temp fix to install from the drat works, it complicates both the CI builds for downstream dependencies (temporary patch that will need to be removed later, esp. for conda based builds) and the installation process for those who are new to programming/R and do not know to check Github issues, particularly in the case where Rcpp is installed as a dependency.

[coatless]

Sidenote: @langmm woah, NCSA is deploying Rcpp in production? This is awesome!

[jeroen]

It depends on which version of MacOS and xcode that you run. I have started seeing a lot of problems on MacOS 10.14 + xcode 11.3.

Here is an example log file: https://travis-ci.org/github/autobrew/autobrew.github.io/jobs/666857475 (search for: unknown type name 'uuid_t'; errors).

You can reproduce this problem on travis by using:

os: osx
osx_image: xcode11.3

This is especially unfortunate because xcode11.3 is precisely the version of toolchain that CRAN will be targeting for R 4.0. The difference is that CRAN runs a different version of MacOS than travis, so the problem may go unnoticed on the mac builder.

I think it is important that this is fixed on CRAN asap, even if the CRAN builder configuration is not affected. Many users that build packages from source will be.

[jozimck]

I am also experiencing this issue on MacOS 10.13.

[eddelbuettel]

@langmm: I respectfully disagree and think that what you state is not entirely correct. If you add the drat repo now, you get Rcpp 1.0.4.4 as desired. Suppose nothing were to change, then the next CRAN Rcpp version will be 1.0.5. And install.packages() will then pick it from CRAN even if you still point to the drat repo. That is how install.packages() works: highest package among declared repos work. So there is no downside. You may even get more fixes from the drat.

Saying "gee, all this is complicated" is also not entirely fair. Either one understands what the CI commands and setup do so that adding / removing a line is pocket change, or "for those new to R" one trusts other people to provide an appropriate setup. I am sure the folks providing the popular stanza for CI have no difficulty adding a repo.

But I do acknowledge the point that "yes, Rcpp may just be a dependency". Rcpp is also (effectively) an infrastructure package that is fairly widely deployed across CRAN---but the failure here is collective. I did ask for wider testing a week before the release, and the fact that neither those relying on R 3.3.* working for them nor those using macOS and building from source did notice issue pre-release is simply not good enough by all of us. Hindsight is 20/20 and I clearly should have asked a week (or two) earlier, but those of you on non-CRAN build paths (maybe including you on Conda) could have helped by testing. It is spilled milk now. Yet for each build issue reported since the Rcpp 1.0.4 release we did offer a bugfix build via the drat.

Let's see what CRAN comes back with now that @wch posted on r-devel and take it from there. Thanks for the feedback, always appreciated.

[wch]

There's a conversation about the CRAN Mac build machines here: https://stat.ethz.ch/pipermail/r-devel/2020-March/079198.html
And later moved to here: https://stat.ethz.ch/pipermail/r-sig-mac/2020-March/013278.html
I have replied to @s-u's latest post on that thread, but my message is currently held for moderation since I'm not a member of the r-sig-mac list.

The important bits:

the Mac CRAN build builds a package only if either is true:

1. the package has not passed checks
2. there is a new version of the package since last successful build+check

This confirms that none of the downstream dependencies would have been rebuilt after an Rcpp release. (I don't know for sure if the packages are run the R CMD check, but it seems very unlikely.)

However, he also said that he was able to build the CRAN version of httpuv using the CRAN version of Rcpp. [1]

It is still a mystery to me why Simon was able to build httpuv but no one else seems to be able to. He noted a possibility: they use the macOS 10.11 SDK, whereas others have been using other versions (I'm aware of people who have used 10.15 and 10.14). I don't know of anyone else who has yet tried with the 10.11 SDK but I will try to set up my system to use it.

The question now is whether downstream packages compile with the 10.11 SDK but not with later versions.

---

[1] In his email, Simon noted that the httpuv issue I linked to was originally related to a different problem. He is correct about that, but when I was investigating the issue (filed by someone else), I ran into the uuid_t/uid_t issue that was fixed by #1047. I could have made that clearer in the issue comments.

[langmm]

@eddelbuettel My experience may be different from yours and that is OK. I generally looks at CI testing as an opportunity to test the package from install to execution so that I know when/where users might experience issues (I mostly work with scientists who do not have formal training in programming and are trying to collaborate across programming languages). As a result, I try to mimic the expected install pattern for a typical user as much as possible. Since Rcpp is a dependency in my case, that means installing from CRAN or conda. That is why I do not see the drat install.packages() call as a permanent solution since it will not reflect the user's experience unless they know to install Rcpp from the drat. However, this is probably not the core motivation of most of your users.

[eddelbuettel]

@langmm As discussed the drat approach is needed only if you use R 3.3.* (time to update) or are on macOS, and even that seems not entirely (see e.g. Simon on r-devel, but I lack details here as I am not a macOS user).

It doesn't hold for any of the systems we tested on pre-release, nor did any of the users of such settings partake in pre-release testing. So it slipped. Next time we all as a community of users need to try to cast a wider net. I am sorry that the experience is now tainted for e.g macOS users; we generally try hard to avoid this (and managed so far). It would be really helpful if one or more of the macOS users could step forward and help with broader testing, maybe even with (almost all) reverse dependencies.

And I remain optimistic that folks who can do CI programmatically (by extending and adapting scripts for CI) may also be able to add a single line to access an updated tar.gz (or check out a repo via git if they prefer; I happen to like tar.gz releases better). For those relying on pre-made CI solutions maybe those providing these solutions can help with that step.

Lastly, I have not yet heard back from CRAN so let's wait and see a little what consensus emerges.

[langmm]

@eddelbuettel I aim to support R >=3.2 because 3.2 is the version that apt installs by default (again a large number of my target users are not very experienced programmers, so limiting the number of extra steps is important for minimizing the barrier to entry). I understand if Rcpp is no longer going to support R ≥ 3.0.2 and will adapt the installation for my package accordingly, but I had assumed support for R 3.2 based on the reported package info. Maybe the reported minimum supported version of R could be updated?

If it is useful, I am happy to supply a stripped version of my Travis CI setup that reproduces the installation error on Mac (I use SDK 10.9 for compat with conda-forge's compilers), or adapt it into a PR to add a Mac job to Rcpp’s .travel.yml (caveat: Mac jobs do take a while to run on Travis, though, so I can understand the desire to avoid that unless doing pre-release testing).

Again, my point was not that people can't add that line to their CI script -- that's what I have done so that I can continue development on my own project -- it's that the permanent addition of this line means that the CI install may not reflect the user's installation (i.e. the CI may have a version from drat that is not on CRAN/conda-forge), so it may not produce the same error.

[eddelbuettel]

@langmm Thanks for all you do for your users to support R, and Rcpp.

If supporting R 3.2 (or an even older version) is your goal, and that is the interpreter you use and provide, I recommend you also use a version of Rcpp released concurrent with, or during the lifetime, of the R release you aim for. That is essentially the MRAN approach, and it is consistent with how CRAN operates: working, and trusted, at a point in time. Not at random cuts across time spanning multiple years.

Jumping many years to a current package version is, to me, plainly asking for trouble. You then are far outside the grid of tested and supported versions. Which is defined here by CRAN.

I am all for expanding that matrix as community and contributor project, but you cannot really rely on me here. I test against the versions CRAN tests, I test against the versions RHub offers, and I run multiple, extensive reverse depends checks (with logs I commit here) but solely on Linux as that is what I have access to (thanks to a courtesy VM), and likely time for. It is also the platform I know best (by far).

But who knows, maybe out of this come an initiative by others to help with more testing. Until that happens I would suggest that stick to with actually tested versions and combinations. Rcpp 1.0.4 on R 3.2.* simply wasn't, and nobody every said it was.

(The reported minimal R version was due to a line last touched in 2013 in the DESCRIPTION file, this has since been fixed as stated above in this thread, and visible on the GH repo),

[davidchall]

Just to add a bit of extra info about the CRAN macOS machines...

I have a package that depends on Rcpp. On my local MacBook and using GitHub Actions CI for macOS builds (R 3.6, macOS 10.15), I get the #1046 error. So for development work I've been using unreleased Rcpp 1.0.4.1 for the last few days.

I submitted a new version of my package to CRAN a couple of days ago, and the macOS builds completed successfully. These CRAN builds will have used the buggy Rcpp 1.0.4. I think this shows that the CRAN macOS builds are not susceptible to issue #1046.

[wch]

With the CRAN version of Rcpp, I tested building the CRAN versions of httpuv and dplyr with different versions of the macOS SDK.

• With the 10.11 SDK, compiling dplyr and httpuv works
• With the 10.14 SDK, compiling dplyr and httpuv results in error (Updated 2020-03-30)
• With the 10.15 SDK, compiling dplyr and httpuv results in error
• With the 10.15 SDK and Rcpp 1.0.4.4 from drat, compiling dplyr and httpuv works. (I also needed to compile Rcpp because a binary package wasn't available)

This is true for both clang 7 and clang 11. So the key to making Rcpp's downstream dependencies compile isn’t the compiler toolchain, as per https://cran.r-project.org/bin/macosx/tools/, but the SDK version, which isn’t mentioned there.

[eddelbuettel]

Ahhhh. Thank you so much for that insight. Could you maybe add a third line:

• With the 10.15 SDK and Rcpp 1.0.4.4 from the drat it (works|results in error)

[wch]

@eddelbuettel I've tested that combination and added the entry above.

Some info about getting the 10.11 the SDK: My understanding is that, getting an "official" copy of the 10.11 SDK from Apple is prohibitively difficult (https://apple.stackexchange.com/a/358632). You would need to install an old version of Xcode, but the older versions of Xcode can only be installed with older versions of macOS. So those of us that don't have super-old versions of macOS need to the SDK from somewhere else. The most commonly-recommended source to get it from is this: https://github.com/phracker/MacOSX-SDKs/releases. But I have to say that the name of the GitHub account and the information in the user's profile doesn't exactly inspire confidence.

After unzipping the 10.11 SDK file to /Library/Developer/CommandLineTools/SDKs/MacOSX10.11.sdk, I added the following to my ~/.R/Makevars file. (I don't know if all of this is necessary, but it seemed to do the job.)

CFLAGS+=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.11.sdk
CXXFLAGS+=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.11.sdk
CPPFLAGS+=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.11.sdk

For clang7, the CPPFLAGS also had -I/usr/local/clang7/include.

---

So although it is possible to get CRAN versions of downstream packages to compile on users' macOS systems using the CRAN version of Rcpp, the procedure for doing so is a pretty unreasonable one, IMO.

[eddelbuettel]

As mentioned, if someone gets me a (macOS-binary) build of Rcpp it is a one-liner to add it to the drat.

[s-u]

I don't see how any of this is relevant. IMHO Rcpp should release the Rcpp bugfix on CRAN so those not using CRAN binaries and hit the issue can continue to use Rcpp.

@wch I still don't understand what you're trying to achieve. You can compile Rcpp with various SDKs and tools - the combinations you can use are numerous. If you just want to use Rcpp, then simply install it from the CRAN binary.

[wch]

@s-u Installing Rcpp isn't the issue. It can be installed, as you said, from a CRAN binary, and it can even be built and installed from the 1.0.4 sources on CRAN, on all platforms that I'm aware of.

The problem is with building packages that use Rcpp (like dplyr and httpuv). That does not work on the toolchain that the vast majority of Mac R users have installed -- I know it doesn't work with the 10.14 or 10.15 SDK.

What I've been trying to do is figure out why those packages can be built with the CRAN version of Rcpp on the CRAN build machines, but (apparently) not on anyone else's machines. We know now that the key difference is that the CRAN build machines use the 10.11 SDK.

[s-u]

@wch Ok, now I understand what you are after - so essentially no one on macOS can develop packages that depend on Rcpp with the current Rcpp release. Since this is mainly about development I'd say it's much easier to install the devel version or Rcpp than to mess with SDKs. That said, as far as I can tell 10.14 SDK works so you don't have to go all the way to 10.11 for that.

[pat-s]

We know now that the key difference is that the CRAN build machines use the 10.11 SDK.

Since I am also having a major pain with all of this here I'll add my thoughts on this:

• I do not see why there are many issues including numerous comments needed just to justify the resistance of a bugfix release if so many users are facing problems (yes I know that a CRAN release of Rcpp might take "a bit longer" than for other packages). However, given the importance of Rcpp in the R community and the accumulated wasted hours of many people trying to debug this, this has no relation to a quick patch release of Rcpp imo.
• I do not get why drat has been mentioned multiple as the "simple solution"/gold standard to fix all that - we all agree on the CRAN way and we all know that CI systems also rely on it. Nobody wants to adjust the whole testing chain and then revert after the next CRAN release.
• I am maintaining a package that is used by hundreds of people as the CI base package. While I was able to trigger a temporary installation of Rcpp-dev for the time being, I come back to conclusion of the previous points above.
• Mistakes happen. However, defending such to the extreme because there might be one version/toolchain out there that still works with that change while 90% others are broken are just a selfish justification. The same goes for "buy me Mac and I'm happy to test". CI tools exist exactly for this reason: Removing the need to have multiple machines on your desk running everything on a different OS. Travis exists since years and on GitHub Actions one can run with the latest SDK on macOS since months/roughly a year. This issue would have been caught early by this. So I'd conclude to "fix/up your CI game" rather than "requesting a Mac from the community".

Besides, I agree with all points of @wch here and would like to add the following regarding the CRAN toolchain (which might be somewhat off-topic):

• There should be more first-level documentation about the CRAN toolchain and how macOS users can install/mirror this setup. Currently it is really hard to mirror this or to find all bits an pieces of information about it. I've mentioned this in r-sig-mac multiple times lately with little success.
• Everyone knows that Apples SDKs are subject of major changes between the major releases. While it is great that CRAN tests on older machines to ensure backward comp, 90% of all users will always be on the last/second last version of the OS. That said, I'd really like to see a CRAN machine running the latest version of the OS including checks with new beta releases of macOS. Currently users/devs have to go the hard way and find out about possible breakage due to Apples new SDKs on their side because CRAN does not check for a working toolchain with the latest/upcoming SDK versions.

Since this is mainly about development I'd say it's much easier to install the devel version or Rcpp than to mess with SDKs.

It is also about testing on R-devel via CI. There you can only install from source and installing the latest Rcpp CRAN makes testing impossible at the moment for many packages (without adjustments).
Especially right now, just before R 4.0, it is more important than ever to have a functioning R-devel CI testing approach. Since macOS is the simplest way to get a binary installation of the R-devel tarball, this is used widely for R-devel testing within the community. And just right now many CI configs are broken/were broken because the need for a small patch release of Rcpp "is not seen". This is really a pity.

[s-u]

@pat-s I fully agree with you on the point 1) - as the quote attributed to Robert says: version numbers are cheap.

However, I disagree on point 2). The goal of CRAN is to provide stable binaries to users, it is not to provide a testbed for developers. As you said yourself, that is much better done with CI tools, not CRAN. But we can continue that discussion on R-SIG-Mac if you're interested. With R 4.0.0 we are going back to Apple toolchain so it should be quite easy to have an equivalent Travis setup.

[pat-s]

However, I disagree on point 2). The goal of CRAN is to provide stable binaries to users, it is not to provide a testbed for developers. As you said yourself, that is much better done with CI tools, not CRAN. But we can continue that discussion on R-SIG-Mac if you're interested. With R 4.0.0 we are going back to Apple toolchain so it should be quite easy to have an equivalent Travis setup.

I'm happy to help making the macOS experience better/easier for both users and devs.
I understand that CRAN has no obligation to check any kind of toolchain across various releases of popular operating systems. However, with tools like GitHub Actions (and possibly self-hosted runners) there is a toolstack to somewhat easily check essential parts of the build chain against latest versions of operating systems while the maintenance work for these images is done externally. Let's continue the discussion at R-SIG-Mac 🙂

[wch]

That said, as far as I can tell 10.14 SDK works so you don't have to go all the way to 10.11 for that.

For the record, Rcpp's downstream dependencies do not compile with the 10.14 SDK. I just tested and they have the same uuid_t/uid_t error. I've updated the entry above to reflect that.

I agree with those who think that Rcpp should release a new version to CRAN. Yes, CRAN provides stable binaries to users, but it is more than that. I don't think that provide source packages and allowing users to build packages from source is just a thing for developers. The install.packages() function even asks if you want to install packages from source if the latest version isn't available as a binary package.

The CRAN version of Rcpp is effectively broken for Mac users -- even though users can install binary packages that use Rcpp, they cannot use Rcpp itself without jumping hoops and using a dodgy source for the old SDK. I realize that releasing a new version of Rcpp takes a nontrivial amount of effort, but I also know that are people who are more than willing to help out with some of the tedious things, like reverse dependency checks on various platforms, to help solve these very real problems.

[pat-s]

@wch Simon announced yesterday that the R40 toolchain will use SDK 10.13, gfortran 8.2 and Apples native compiler: https://mac.r-project.org/

That said I installed SDK 10.13 from your link and I am able to install {Rcpp} from source with it (both R-release with clan7 and R-devel with /usr/bin/clang).

While this works, this is not really an easy approach for use in production for most people. Also a lot of custom efforts are needed on CI systems. But this is a different issue and belongs to R-SIG-MAC.

Anyway here is the CLI way to install SDK10.13

wget https://github.com/phracker/MacOSX-SDKs/releases/download/10.15/MacOSX10.13.sdk.tar.xz
tar fvxz MacOSX10.13.sdk.tar.xz
sudo mv MacOSX10.13.sdk /Library/Developer/CommandLineTools/SDKs/
rm -rf MacOSX10.13*

Afterwards add this to ~/.R/Makevars

CFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
CCFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
CXXFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk
CPPFLAGS=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk

I will probably write a blog post summarizing all of this so that things become more clear for people using macOS.

[eddelbuettel]

After some (as always unexplained) delay (of about a week), Rcpp 1.0.4.6 is now on CRAN.

[pat-s]

FWIW the new title of this issue is wrong. People arriving here should know that they can use SDK 10.13 (which is two years newer than 10.11).

Also, SDK 10.13 (High Sierra) is the SDK version CRAN will build R 4.0 with so it has an additional benefit of choosing this one over any other.

[eddelbuettel]

Just trying to keep track of developments:

• macOS will not have OpenMP support (@coatless, RcppArmadillo with macOS and OpenMP In a Post R 4.0.0 World RcppArmadillo#290)
• macOS does not allow you to debug code (@s-u, r-devel, 2020-04-21)

That's pretty impressive.

[kevinushey]

It is possible to use OpenMP with the newer versions of Apple Clang, although its support appears to be poorly advertised / documented. I have:

CPPFLAGS     += -I/usr/local/opt/libomp/include -Xclang -fopenmp
LDFLAGS      += -L/usr/local/opt/libomp/lib -lomp

in my ~/.R/Makevars, and that appears to be sufficient (using the version of libomp provided by LLVM 10 in this case, from Homebrew). I'm not sure if there's a better way around this though, and your luck may vary depending on the version of macOS / Xcode you have installed...

FWIW, CMake does something similar when finding + activating OpenMP on macOS:

https://github.com/Kitware/CMake/blob/8a8ebcdd707394981c9e70bf463ca7b1a7b36aac/Modules/FindOpenMP.cmake#L100

The inability to debug notarized applications is a pain, but it can still be worked around in various ways:

1. Disable System Integrity Protection (probably a bad idea),
2. Build R from sources yourself or install a non-notarized version (less bad idea).

All in all -- a huge pain, but at least there are still ways out.

[s-u]

@kevinushey is there any confirmation that SIP makes a difference? Last time I checked it didn't, but that may have been with betas. You can still use the our binary before notarization - it's the same binary at least. As for OpenMP, I was thinking about it to the point of shipping iomp, but apparently the binaries are incompatible between clang versions so they have to match as well which is a PITA.

@eddelbuettel I'm not happy about Apple's decisions for the last few years, but unfortunately there is not much we can do. The OpenMP issue is completely bonkers since clang supports it, they just take it out. On the debugging issue you can see where they are coming from - they are trying to create a rock-solid consumer platform, they are not catering to developers. It is true that at the end we may end up running R in Ubuntu Docker images ;). (That is, until Apple bans Docker on macOS ;))

[kevinushey]

@s-u I can confirm that disabling SIP allows lldb to attach to R with latest Catalina:

kevinushey@Kevins-MBP:~
$ system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: macOS 10.15.4 (19E287)
      Kernel Version: Darwin 19.4.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Computer Name: Kevin’s MacBook Pro
      User Name: Kevin Ushey (kevinushey)
      Secure Virtual Memory: Enabled
      System Integrity Protection: Disabled
      Time since boot: 4 minutes

kevinushey@Kevins-MBP:~
$ R -d lldb
(lldb) target create "/Library/Frameworks/R.framework/Resources/bin/exec/R"
Current executable set to '/Library/Frameworks/R.framework/Resources/bin/exec/R' (x86_64).
(lldb) run
Process 2225 launched: '/Library/Frameworks/R.framework/Resources/bin/exec/R' (x86_64)

R version 3.6.3 (2020-02-29) -- "Holding the Windsock"
Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin15.6.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> print("hello world!")
[1] "hello world!"
>

And confirming that this is indeed the notarized copy of R:

kevinushey@Kevins-MBP:~
$ codesign -d --entitlements :- `pgrep -nx R`
Executable=/Library/Frameworks/R.framework/Versions/3.6/Resources/bin/exec/R
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>com.apple.security.cs.disable-library-validation</key>
	<true/>
	<key>com.apple.security.cs.allow-dyld-environment-variables</key>
	<true/>
	<key>com.apple.security.automation.apple-events</key>
	<true/>
</dict>
</plist>

[pat-s]

That's pretty impressive.

Maybe it could be more helpful to listen to community feedback and ensure a smooth experience of one's own packages rather than continuously ranting against (niche) developer issues of an operating system of which no one has any influence on.

This would prevent further splitting of the R community, which this issue apparently did well on. But maybe I got it wrong until now and this is actually the whole purpose.

Just a thought though (to keep the tradition of the equivocal last sentence).

[eddelbuettel]

Some people continue to waste our (aggregate) time. I am locking this thread now.