Crippled Lapack Declaration on Windows

[kthohr]

Greetings,

A lot of Lapack routines were added with R version 3.3.0 (zgges, etc.), and the package configure script for Unix-alike systems correctly identifies this. However, for Windows systems, the assumption in RcppArmadilloConfig.h is that these routines are still missing. With R v3.5 being released in the next month or so, is now a good time to (safely) change this default behavior and assume most users have updated?

Best,
Keith

[eddelbuettel]

No, there isn't, sadly, as one @baptiste 's package still failed over a complex decomposition. As long as the (partial) LAPACK copy that ships with R and is used by default on Windows, we are bound to have these issues. I fought this twice already.

[eddelbuettel]

What is your underlying question? Are you missing a particular routine? Over the years it was generally just the somewhat more esoteric complex functions that were seen missing, and R Core helped twice and added them.

In general, until the R build changes and always uses an external (complete) LAPACK I fear that is little we can do better. Whenever we have left the guard down and said "we're good" we ended up being bitten by something we actually did not have.

[kthohr]

For sure. I guess what I should have asked is: can we update the list of unavailable Lapack symbols on the Armadillo side? (What gets ifdef'd out by ARMA_CRIPPLED_LAPACK.) For example, I've tested qz() in the complex case on Windows (vanilla R v4.3) and there are no issues.

Is @baptiste 's package calling routines that weren't covered by the updates?

[eddelbuettel]

It think when @baptiste was bitten last it was due to a change on the Armadillo side when @conradsnicta invoked a different (sparse?) solver. I am a little fuzzy on the details of that conversation -- may have been mostly on the r-package-devel or rcpp-devel mailing lists.

As for

there are no issues

that tends to be my sentiment too. As of today, we do have 458 packages on CRAN that use RcppArmadillo. In general, things just work.

This README in the R sources has a good list of what is included. Again, for RcppArmadillo, ex ante we do not know which library we are facing hence the need to dial down. So we set ARMA_CRIPPLED_LAPACK if we're on Windows, or if we see Rlapack.* being used. That #define then affects Conrad's code.

[kthohr]

Sorry I wasn't clear with that example. Right now qz() for complex-valued matrices is ifdef'd out on the Armadillo side, as zgges was missing until R version 3.3.0. However, I've tested RcppArmadillo on Windows with that guard removed and it works fine. So my feeling is that it should be okay to start removing some of the guards based on the 3.3 update.

[eddelbuettel]

And again, to be crystal clear, that guard is in the Armadillo code, correct?

We could test this in a local patch we could try to noodle through CRAN. If it all works we can discuss with Conrad if/where/how the #ifdef is still needed. A quick recursive grep reveals that it is only used in
inst/include/armadillo_bits/auxlib_meat.hpp but I fear we have to test each and every use case against different R versions and their Rlapack.*. (Or at least 3.3.0 and later...)

That is quite a bit of work. Are you volunteering?

[kthohr]

Yep, that guard is on the Armadillo side.

I'm happy to do the heavy lifting on this---I'm familiar with Rlapack and the internal Armadillo code. Would you prefer to start with the single qz case and then branch out to other functionality?

[eddelbuettel]

I really appreciate this. And the four-eyes principle is good. So how about something like this:

• we look at the different functions affected by the #ifdef guard
• we try to make sure each one is tickled by at least one unit test function
• we use win-builder and/or r-hub to test 'both sides' ie external LAPACK as well as cases where Rlapack is used

How does that sound?

[kthohr]

Sounds good to me.

[baptiste]

For the record since my cda package was mentioned: the package actually broke for Linux on CRAN, which apparently also rely on the internal R Lapack (I thought it did only for Windows). The package itself does not use any unusual routine, it just calls solve() on a standard complex linear system, but Armadillo recently introduced a new special-case for banded matrices that would require, if it was ever called, a Lapack routine that R doesn't provide. Linux CRAN checks find this routine in the headers missing, and break (in contrast Windows is always flagged as crippled). Someone kindly provided a workaround (to basically flag Lapack as crippled) on the mailing list so that it would pass CRAN checks like on Windows, but I never found the time to test this (when I tried on my Linux box an external Lapack was used, and when I tried to switch that didn't seem to affect anything; maybe I'd need to compile R with some flags for the purpose of this test).

Bottom line: cda is now archived on CRAN; I hope to return to it someday.

It sounds like we're facing opposing problems: you want to tell CRAN not to flag 'crippled' now that a routine is available, while I'd need to enforce the flag manually because some routine is missing.

[eddelbuettel]

I thought it did only for Windows

It's a compile time option. On the non-Windows systems, if you do configure --help you get (inter alia)

  --with-blas           use system BLAS library (if available), or specify it [no]
  --with-lapack         use system LAPACK library (if available), or specify it [no]

so the default may actually be to use the 'crippled' lapack (I think my local r-devel builds do that as I don't override it). But most distribution builds will probably use the external BLAS.

I supposed one can alter it on Windows too which must be how Microsoft injects the MKL.

Bottom line: "it's complicated" and when we build RcppArmadillo we have to track it.

[baptiste]

@conradsnicta I think a suitably edited version of your question would be a useful discussion to have on R-devel. I don't know the reasons myself, but my guess would be that R has always been very conservative with respect to numerical accuracy, and R-core may have found it necessary to manually patch/review some routines on a case-by-case basis. Or it could be that the effort of writing all the appropriate wrappers and ensuring their correctness has always been the limiting factor, leading to this cherry-picked, add-only-when-necessary situation.

Here's a link to the last time R added a bunch of Lapack routines; I'm trying to figure out from the (large) diff what was involved in the process.

[baptiste]

@conradsnicta as I said I don't know the reasons (and therefore should not have replied); I made a wild guess based on my recollection of discussions around flavours of Blas and Lapack (cf openBLAS and other alternatives), with R favouring a conservative (inefficient but more reliably accurate) choice by default. It may very well be completely irrelevant here; R-devel would have much more pertinent things to say.

[eddelbuettel]

This is a 30 year old discussion. Statistician prefer eg the QR solver with pivoting which is why a modified LINPACK gets used in the standard lm(). But these days with double precision you have to work hard for finding actual differences. The example fastLm() in RcppEigen has a worked example thanks to @dmbates , the one in for fastLm() in RcppArmadillo does not but at least I put a note in the documentation.

And for 99.95% of all users the shipped LAPACK subset is actually sufficient. In all those years the only user I encountered who had issues was @baptiste . Now, his uses were all legit so we tried (and managed) to get this improved -- which can be done. But running around, screaming murder and calling other people names rarely does.

[eddelbuettel]

The part I have control over -- how R is built and shipped for Debian and hence Debian derivatives -- does that: full external Lapack. Other parts and builds I cannot control, but I invite you to engage with R Core and argue for change there.