Eigen and permutations parallelization, Progressbar, instructions in readme

[lavakin]

Configured cpp to use OpenMP, so that Eigen can run in parallel and the permutation function can be parallelized using omp library. This might require additional software. Instructions have been added to README.

Created a progress bar to show the progress during computing the permutations. Seems like I managed to handle all the pitfalls of multiprocessing and it didn't slow it down 🥳. (Not the nicest implementation but still good enough I hope.)

Changed the default number of permutations to 10000 so that people will (be forced to) admire my progress bar for longer. (And the fit is more accurate ofc)

[kullrich]

Hi, have you looked e.g. at my implementation of the TAI calculation?

It uses the RcppThread library which comes with its own progressbar and a parallel for loop:

Rcpp::List rcpp_tei_parallel(const arma::sp_mat& expression,
                             Rcpp::NumericVector ps,
                             int ncores = 1){
  std::vector< std::string > psnames =  ps.attr("names");
  int n_col = expression.n_cols;
  Rcpp::NumericVector sumx(n_col);
  Rcpp::NumericVector teisum(n_col);
  Rcpp::NumericVector tei(n_col);
  RcppThread::ProgressBar bar(n_col, 1);
  RcppThread::parallelFor(0, n_col, [&] (int j) {
    for (size_t i = 0; i < expression.n_rows; i++) {
      sumx[j] += expression(i, j);
      teisum[j] += expression(i, j) * ps[i];
      tei[j] = teisum[j]/sumx[j];
    }
  }, ncores);
  return Rcpp::List::create(Rcpp::Named("sumx") = sumx,
                            Rcpp::Named("teisum") = teisum, Rcpp::Named("tei") = tei);
}

Of ocurse I do not know if it works with the RcppEigen

Best regards

Kristian

[kullrich]

The RcppThread also provide some benchmarks, but I have not looked deeply into it:

https://github.com/tnagler/RcppThread/tree/main/benchmarks

[lavakin]

Dear Kristian,

Thank you for your question and comments! If I understand correctly, your function computes the TAI value solely from the expression dataset and phylostrata, with the implementation of parallelization and a progress bar.

Regarding my TAI implementation, the parallelization is implicitly handled by the Eigen library, and it runs so quickly that adding a progress bar would not be necessary.

However, thanks to your comment, I looked into the possibility of using rcppthread to compute the permutations and display a progress bar. Unfortunately, the progress update within the for-loop is atomic and occurs after each iteration. Since generating a single permutation is fast, the threads would mostly wait for each other to update the progress counter, resulting in a loss of the benefits of parallelization. In my current, implementation each thread updates the progress bar after 200 iterations, and it might even be more beneficial to increase this number further.

While it would be technically possible to modify my code so that each iteration computes multiple permutations and then use the progress bar you suggested, such a solution would sacrifice elegance.

Once again, thank you for your comments and for bringing the rcppthread library to my attention.

Best,
Nikola

[HajkD]

Dear @lavakin

Thank you so much for finding this elegant solution for the Eigen library! This will be a super useful addition to the package.

@kullrich

Thank you also for the excellent discussion regarding rcppthread. If you do find an additional way to use this library with Eigen or even speed up some of the Eigen code for faster permutations then your suggestions would be greatly appreciated!

With many thanks and very best wishes,
Hajk

[lavakin]

Just a small comment: generating permutations is independent from Eigen (except for adding the generated permutations to the eigen matrix). Eigen is a library optimized for linear algebraic operations, not a general numerical library such as numpy.

Nikola

[kullrich]

Hi @lavakin,

the intention of the rcpp_tei_parallel function is to work with sparse data from single-cell expression.

Sometimes this data has million of cells, and the problem with the TAI.R function is the following line of code:

TAIProfile <- cpp_TAI(as.matrix(ExpressionMatrix), as.vector(Phylostratum))

So the ExpressionMatrix will be converted into a full matrix before the cpp_TAI function takes over, which normally would crash any personal machine with low memory profiles.

This is so far handled with the TEI.R function, which split and distribute the splitted sparseMatrix.

Best regards

Kristian

[HajkD]

Dear both,

That's an excellent point @kullrich, then let's also adjust cpp_TAI() accordingly, since at the time when I wrote it, there was no single-cell data.

@kullrich Would you like to give it a go or does anything speak against it @lavakin ?

With many thanks and very best wishes,
Hajk

[LotharukpongJS]

Dear all,

Thank you for the discussions and the work that has been put in to speed up (my favourite) package. I have installed the specified dependencies (brew install llvm libomp) on my Mac and loaded the new update to myTAI with devtools::load_all().

Unfortunately, I was unable to load myTAI and was left with an error.

   18 warnings generated.
   clang++ -mmacosx-version-min=10.13 -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o myTAI.so RcppExports.o code.o cpp11.o rcpp_funcs.o -lomp -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
   ld: library not found for -lomp
   clang: error: linker command failed with exit code 1 (use -v to see invocation)
   make: *** [myTAI.so] Error 1
   ERROR: compilation failed for package ‘myTAI’
─  removing ‘/private/var/folders/cx/w3nwmg5x1yl4l9pr2kv3mrwh0000gn/T/RtmpMtcQ8p/devtools_install_c4768643e2b/myTAI’
Error in `(function (command = NULL, args = character(), error_on_status = TRUE, …`:
! System command 'R' failed
---
Exit status: 1
stdout & stderr: <printed>
---
Type .Last.error to see the more details.

This was not the case in my last pull request.

With the assumption that users (desiring the developer version of myTAI) will refer solely to the instructions in README.md, this issue should be resolved as soon as possible.

Do you have any thoughts on how this can be resolved? On my part, I recommend reverting back to the commit before these changes have been implemented, e.g. f671be8, if the fix will take some time, lest other users be inconvenienced.

Many thanks in advance.

Best,
Sodai

zu Ihrer Information

> devtools::load_all()
ℹ Loading myTAI
ℹ 1 functions decorated with [[cpp11::register]]
✔ generated file cpp11.R
✔ generated file cpp11.cpp
ℹ Re-compiling myTAI (debug build)
── R CMD INSTALL ───────────────────────────────────────────────────────────────────────────────────────────────────
─  installing *source* package ‘myTAI’ ...
   ** using staged installation
   ** libs
   clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_MYTAI -DBOOST_NO_INT64_T -DBOOST_NO_INTEGRAL_INT64_T -DBOOST_NO_LONG_LONG -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT  -I/usr/local/opt/libomp/include -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppArmadillo/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppThread/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include' -I/usr/local/include  -Xpreprocessor -fopenmp -fPIC  -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c RcppExports.cpp -o RcppExports.o
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:1:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Core:540:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:2:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/LU:47:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:3:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Cholesky:12:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Jacobi:29:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:3:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Cholesky:43:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:4:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/QR:15:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Householder:27:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:4:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/QR:48:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:5:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/SVD:48:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Geometry:58:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:7:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Eigenvalues:58:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:26:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/SparseCore:66:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:27:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/OrderingMethods:71:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:29:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/SparseCholesky:43:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:32:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/SparseQR:34:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:33:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/IterativeLinearSolvers:46:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:32:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/CholmodSupport:45:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:35:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/KroneckerProduct:34:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:39:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/Polynomials:135:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from RcppExports.cpp:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:40:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/SparseExtra:51:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   18 warnings generated.
   clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_MYTAI -DBOOST_NO_INT64_T -DBOOST_NO_INTEGRAL_INT64_T -DBOOST_NO_LONG_LONG -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT  -I/usr/local/opt/libomp/include -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppArmadillo/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppThread/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include' -I/usr/local/include  -Xpreprocessor -fopenmp -fPIC  -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c code.cpp -o code.o
   clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_MYTAI -DBOOST_NO_INT64_T -DBOOST_NO_INTEGRAL_INT64_T -DBOOST_NO_LONG_LONG -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT  -I/usr/local/opt/libomp/include -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppArmadillo/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppThread/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include' -I/usr/local/include  -Xpreprocessor -fopenmp -fPIC  -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c cpp11.cpp -o cpp11.o
   clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src -DRCPP_DEFAULT_INCLUDE_CALL=false -DCOMPILING_MYTAI -DBOOST_NO_INT64_T -DBOOST_NO_INTEGRAL_INT64_T -DBOOST_NO_LONG_LONG -DRCPP_USING_UTF8_ERROR_STRING -DRCPP_USE_UNWIND_PROTECT  -I/usr/local/opt/libomp/include -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppArmadillo/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/Rcpp/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/cpp11/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppThread/include' -I'/Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include' -I/usr/local/include  -Xpreprocessor -fopenmp -fPIC  -Wall -g -O2  -UNDEBUG -Wall -pedantic -g -O0 -fdiagnostics-color=always -c rcpp_funcs.cpp -o rcpp_funcs.o
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:1:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Core:540:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:2:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/LU:47:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:3:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Cholesky:12:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Jacobi:29:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:3:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Cholesky:43:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:4:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/QR:15:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Householder:27:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:4:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/QR:48:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:5:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/SVD:48:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:6:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Geometry:58:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:30:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Dense:7:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Eigenvalues:58:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:26:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/SparseCore:66:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:27:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/OrderingMethods:71:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:29:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/SparseCholesky:43:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:32:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/SparseQR:34:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:31:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/Sparse:33:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/IterativeLinearSolvers:46:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:32:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/CholmodSupport:45:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:35:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/KroneckerProduct:34:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:39:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/Polynomials:135:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   In file included from rcpp_funcs.cpp:8:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigen.h:25:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/RcppEigenForward.h:40:
   In file included from /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/SparseExtra:51:
   /Library/Frameworks/R.framework/Versions/4.2/Resources/library/RcppEigen/include/unsupported/Eigen/../../Eigen/src/Core/util/ReenableStupidWarnings.h:14:30: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
       #pragma clang diagnostic pop
                                ^
   18 warnings generated.
   clang++ -mmacosx-version-min=10.13 -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o myTAI.so RcppExports.o code.o cpp11.o rcpp_funcs.o -lomp -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
   ld: library not found for -lomp
   clang: error: linker command failed with exit code 1 (use -v to see invocation)
   make: *** [myTAI.so] Error 1
   ERROR: compilation failed for package ‘myTAI’
─  removing ‘/private/var/folders/cx/w3nwmg5x1yl4l9pr2kv3mrwh0000gn/T/RtmpMtcQ8p/devtools_install_c4768643e2b/myTAI’
Error in `(function (command = NULL, args = character(), error_on_status = TRUE, …`:
! System command 'R' failed
---
Exit status: 1
stdout & stderr: <printed>
---
Type .Last.error to see the more details.

> sessionInfo()
R version 4.2.2 (2022-10-31)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Ventura 13.0.1

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] pkgdown_2.0.7  lintr_3.0.2    styler_1.10.0  renv_0.17.3    testthat_3.1.8 roxygen2_7.2.3 devtools_2.4.5
[8] usethis_2.1.6 

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.10       prettyunits_1.1.1 ps_1.7.5          utf8_1.2.3        rprojroot_2.0.3   digest_0.6.31    
 [7] mime_0.12         R6_2.5.1          pillar_1.9.0      rlang_1.1.1       lazyeval_0.2.2    curl_5.0.0       
[13] rstudioapi_0.14   miniUI_0.1.1.1    callr_3.7.3       urlchecker_1.0.1  R.utils_2.12.2    R.oo_1.25.0      
[19] desc_1.4.2        stringr_1.5.0     htmlwidgets_1.6.2 shiny_1.7.4       compiler_4.2.2    httpuv_1.6.11    
[25] xfun_0.39         pkgconfig_2.0.3   pkgbuild_1.4.0    htmltools_0.5.5   tibble_3.2.1      fansi_1.0.4      
[31] crayon_1.5.2      withr_2.5.0       later_1.3.1       brio_1.1.3        R.methodsS3_1.8.2 xtable_1.8-4     
[37] lifecycle_1.0.3   magrittr_2.0.3    cli_3.6.1         stringi_1.7.12    cachem_1.0.8      fs_1.6.2         
[43] promises_1.2.0.1  remotes_2.4.2     rex_1.2.1         xml2_1.3.4        decor_1.0.1       ellipsis_0.3.2   
[49] vctrs_0.6.2       cyclocomp_1.1.0   tools_4.2.2       R.cache_0.16.0    glue_1.6.2        purrr_1.0.1      
[55] processx_3.8.1    pkgload_1.3.2     fastmap_1.1.1     sessioninfo_1.2.2 cpp11_0.4.3       memoise_2.0.1    
[61] knitr_1.43        profvis_0.3.8   

$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1200.0.32.21)
Target: x86_64-apple-darwin22.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

[kullrich]

Hi, I resolved with this:

brew install lomp
cd /usr/local/lib
ln -s /usr/local/opt/libomp/lib/libomp.dylib ./libomp.dylib

Best regards
Kristian

[lavakin]

Interesting, I tried it in a separated environment and it compiled without problems.

please let me know if it works with Kristians fix and if yes, I’ll add it to the README.

If it would be an issue in the future, we can also think about building docker, which would ensure same behaviour through distributions.

Thanks a lot, Kristian for fixing the issue.

[HajkD]

Done! I added this to the README. Thank you so much, @kullrich!

@LotharukpongJS Does this work for you now?

I agree that building a docker would be great! But we need to also add this to the conda recepe.

Many thanks,
Hajk

[lavakin]

Dear both,

That's an excellent point @kullrich, then let's also adjust cpp_TAI() accordingly, since at the time when I wrote it, there was no single-cell data.

@kullrich Would you like to give it a go or does anything speak against it @lavakin ?

With many thanks and very best wishes, Hajk

Eigen has its own SparseMatrix class, so we can also have a look if there is an easy way to switch to that. (The documentation webpage doesnt work rn, but will have a look once its working again)

[HajkD]

Dear @lavakin

I also get:

Plot signature: ' TAI ' and test statistic: ' FlatLineTest ' running  20000  permutations.
Computing permutations
[=========================================] 100%   
Computing variances
Number of Eigen threads: 8
Computing permutations
[=========================================] 100%   
Computing variances
Number of Eigen threads: 8
Time: 12.3361010551453
Significance status of signature:  significant.
Now run 'FlatLineTest(..., permutations  =  20000 , plotHistogram = TRUE)' to analyse the permutation test performance 
Warning message:
In ks.test.default(filtered_vars, "pgamma", shape = shape, rate = rate) :
 ties should not be present for the Kolmogorov-Smirnov test

Would it be possible to find a quick solution to this warning?

Warning message:
In ks.test.default(filtered_vars, "pgamma", shape = shape, rate = rate) :
 ties should not be present for the Kolmogorov-Smirnov test

With many thanks,
Hajk

[lavakin]

I looked into it already at some point and as far as I understand it, it's a property of myTAI. Since we have a lot of repeated vaules in the phylostrata (because there are not many possible values), with a higher number of permutations, we will unavoidably get the same variance multiple times, because the same phylostrata will be assigned to the same genes (eventhough the permutations are different).

This might be also one of the reasons for a bad fit.
There is a way to prevent this, but then the permutations would not be randomly drawn anymore, so I don't think that would be a good idea.

But if you have a suggestion, what to do about it, I will be more than happy to fix it :).

[kullrich]

Hi @lavakin,

regarding the sparseMatrix from the RcppEigen package.

It would be great if one could just use it directly.

I created a small benchmark with the Matrix package and some matrix crossprod calculation, since given the PhyloExpressionData the dimensions do not change.

Here, if the RcppEigen does not directyl work with the sparseMatrix one could also write a R wrapper function to decide wether to use cpp_TAI or e.g. r_TAI.

r_TAI - function - using Matrix package

library(myTAI)
library(Matrix)

r_TAI <- function(eM, Phylostratum){
    Divisor <- colSums(eM)
    dM <- Matrix::Matrix(0,
        ncol=length(Divisor),
        nrow=length(Divisor),
        sparse=TRUE)
    diag(dM) <- 1/Divisor
    psM <- Matrix::Matrix(0,
        ncol=length(Phylostratum),
        nrow=length(Phylostratum),
        sparse=TRUE)
    diag(psM) <- Phylostratum
    # crossprod magic
    total <- colSums(t(t(eM) %*% psM) %*% dM)
    return(total)
}

And here an example creating sparse data:

# load example data
data(PhyloExpressionSetExample)
ExpressionMatrix <- PhyloExpressionSetExample[,-c(1,2)]
Phylostratum <- PhyloExpressionSetExample[,1]

# create sparse example
ExpressionMatrix[ExpressionMatrix>800]<-0

# create more expression data
test <- cbind(PhyloExpressionSetExample[,c(1,2)],
    do.call(cbind, rep(ExpressionMatrix, 2000)))
colnames(test)[-c(1,2)] <- seq(from=1, to=dim(test)[2]-2)
dim(test)

ExpressionMatrix <- as.matrix(test[,-c(1,2)])
ExpressionMatrix_sparse <- Matrix::Matrix(ExpressionMatrix, sparse=TRUE)
str(ExpressionMatrix_sparse)

and the benchmark:

# benchmark
system.time(r_TAI_results <- r_TAI(ExpressionMatrix, Phylostratum))
       User      System verstrichen 
     26.203      22.533      60.688
system.time(r_TAI_results_sparse <- r_TAI(ExpressionMatrix_sparse, Phylostratum))
       User      System verstrichen 
      5.945       3.670      11.641
system.time(cpp_TAI_results <- myTAI:::cpp_TAI(ExpressionMatrix, Phylostratum))
       User      System verstrichen 
      3.011      11.372      19.785

Note: All three different function have slightly different results due to rounding in the used packages and how R handles matrix versus sparseMatrix colSums I guess.

as.character(r_TAI_results[1])
[1] "3.77721412365141"
as.character(r_TAI_results_sparse[1])
[1] "3.77721412365141"
as.character(cpp_TAI_results[1])
[1] "3.77721412365142"

Best regards

Kristian

[LotharukpongJS]

Hi all,

@kullrich thank you for the solution.

@HajkD regarding the docker, I am not sure if it would make it complicated for some users (biologists) who just want to run myTAI to analyse their new RNA-seq data? One idea is to proceed with the updated multithreading only if OpenMP is detected in the system. Let me know what you think.

I notice some weird behaviours too. I see % signs in places that shouldn't exist.

> data("PhyloExpressionSetExample")
> ExprExample <- tf(PhyloExpressionSetExample, log2)
> PlotSignature(ExprExample, permutations = 20000)
Plot signature: ' TAI ' and test statistic: ' FlatLineTest ' running  20000  permutations.
Computing permutations
[=========================================] 100%   
Computing variances
Number of Eigen threads: 8
Computing permutations
[=========================================] 100%   60%   %   %   %                 ][=======================>                ] 
Computing variances
Number of Eigen threads: 8
Time: 1.83384306828181
Significance status of signature:  significant.
Now run 'FlatLineTest(..., permutations  =  20000 , plotHistogram = TRUE)' to analyse the permutation test performanceWarning message:
In ks.test.default(filtered_vars, "pgamma", shape = shape, rate = rate) :
  ties should not be present for the Kolmogorov-Smirnov test

Furthermore, myTAI doesn't pass devtools::test() and devtools::check(). @lavakin , do you mind having a look?

devtools::test()

[ FAIL 0 | WARN 3480 | SKIP 3 | PASS 200 ]

devtools::check()

── R CMD check results ───────────────────── myTAI 1.0.1.9000 ────
Duration: 4m 59.6s

❯ checking compilation flags in Makevars ... WARNING
  Non-portable flags in variable 'PKG_CXXFLAGS':
    -fopenmp

❯ checking for GNU extensions in Makefiles ... WARNING
  Found the following file(s) containing GNU extensions:
    src/Makevars
  Portable Makefiles do not use GNU extensions such as +=, :=, $(shell),
  $(wildcard), ifeq ... endif, .NOTPARALLEL See section ‘Writing portable
  packages’ in the ‘Writing R Extensions’ manual.

❯ checking package dependencies ... NOTE
  Packages suggested but not available for checking: 'taxize', 'Seurat'

❯ checking installed package size ... NOTE
    installed size is  5.6Mb
    sub-directories of 1Mb or more:
      data   2.0Mb
      doc    1.2Mb
      help   1.6Mb

❯ checking top-level files ... NOTE
  Non-standard file/directory found at top level:
    ‘pkgdown’

❯ checking compiled code ... NOTE
  File ‘myTAI/libs/myTAI.so’:
    Found ‘__ZNSt3__14coutE’, possibly from ‘std::cout’ (C++)
      Object: ‘rcpp_funcs.o’
  
  Compiled code should not call entry points which might terminate R nor
  write to stdout/stderr instead of to the console, nor use Fortran I/O
  nor system RNGs.
  
  See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual.

0 errors ✔ | 2 warnings ✖ | 4 notes ✖
Error: R CMD check found WARNINGs
Execution halted

Exited with status 1.

[HajkD]

Hi @LotharukpongJS,

I very much like the idea that the openmpi code is only used upon internal reference check whether or not a user has OpenMP installed or not. This way, less computer oriented users do not have to bother installing the OpenMP libraries.

@kullrich Thank you very much for this great suggestion! Let's also build this. I like it a lot!

With many thanks and very best wishes,
Hajk