multi-threading on server

[ljkhalil]

Since the last version of Quanteda 0.99, I have lost the multi-threading on server with a virtual environment on docker. ( 1 out of 24 thread).
The multi-threading is working on my laptop.

[kbenoit]

Have you tried setting the number of threads through:

quanteda_options(threads = 2)

for instance? It could be that on the virtual environment, it's setting the default to 1. This could happen if your VE has just two virtual cores, since the default upon loading is now to use half of the available cores.

What does the startup message show?

[ljkhalil]

Dear Prof, I think yes I did, I wrote in the R file: ``` library(quanteda) quanteda_options(threads = 23) ``` *The display is : * ``` quanteda version 0.99 Using 1 of 24 threads for parallel computing Attaching package: ‘quanteda’ The following object is masked from ‘package:utils’: View $threads [1] 1 $verbose [1] FALSE $print_dfm_max_ndoc [1] 20 $print_dfm_max_nfeature [1] 20 $base_docname [1] "text" $base_featname [1] "feat" $language_stemmer [1] "english" $language_stopwords [1] "English" what additional feature can I give you to help

On Wed, Sep 13, 2017 at 12:04 PM, Kenneth Benoit ***@***.***> wrote: Have you tried setting the number of threads through: quanteda_options(threads = 2) for instance? It could be that on the virtual environment, it's setting the default to 1. This could happen if your VE has just two virtual cores, since the default upon loading is now to use half of the available cores. What does the startup message show? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#966 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AecTqH-xtQIJbZgwq6vPRWOiaX4Yc_c3ks5sh6iegaJpZM4PV3cc>

```

[koheiw]

In which function are you seeing the problem? I saw teststat_collocations is not working in the parallel mode in recent version.

[kbenoit]

What is the output when you run the following?

library("quanteda")
quanteda_options(threads = 2)
quanteda_options("threads")
sessionInfo()

[ljkhalil]

Dear Professor, The output shows 1 out of 24 as you can see below: ############# Loading required package: methods quanteda version 0.99 Using 1 of 24 threads for parallel computing Attaching package: ‘quanteda’ The following object is masked from ‘package:utils’: View [1] 2 R version 3.4.1 (2017-06-30) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Debian GNU/Linux buster/sid Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1 locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] methods stats graphics grDevices utils datasets base other attached packages: [1] quanteda_0.99 loaded via a namespace (and not attached): [1] Rcpp_0.12.12 lubridate_1.6.0 lattice_0.20-35 [4] grid_3.4.1 plyr_1.8.4 gtable_0.2.0 [7] magrittr_1.5 scales_0.5.0 RcppParallel_4.3.20 [10] ggplot2_2.2.1 rlang_0.1.2 stringi_1.1.5 [13] lazyeval_0.2.0 data.table_1.10.4 Matrix_1.2-11 [16] fastmatch_1.1-0 tools_3.4.1 stringr_1.2.0 [19] munsell_0.4.3 compiler_3.4.1 colorspace_1.3-2 [22] tibble_1.3.4 Lionel Khalil

…

On 13 Sep 2017, at 22:31, Kenneth Benoit ***@***.***> wrote: What is the output when you run the following: library("quanteda") quanteda_options(threads = 2) quanteda_options("threads") sessionInfo() — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[kbenoit]

The

[1] 2

right before the "R version 3.4.1" indicates that you have successfully set the threads to 2. So that appears to be working.

You mention in the original question that you have 24 threads. If you try to set the number of threads higher than what is actually available, it will not change. You can query the available threads using

RcppParallel::defaultNumThreads()

[koheiw]

Multiprocessing using RcppParallel depends on an environmen variable called 'RCPP_PARALLEL_NUM_THREADS' in you OS.

Check if you get correct number from:

Sys.getenv('RCPP_PARALLEL_NUM_THREADS')

[koheiw]

Please also run quanteda:::qatd_cpp_tbb_enabled() to check if multiprocessing is enabled. If FALSE, your system may not have the Intel TBB library.

[ljkhalil]

Dear Professor, I have tried library("quanteda") quanteda_options(threads = 2) quanteda_options("threads") sessionInfo() Sys.getenv('RCPP_PARALLEL_NUM_THREADS') quanteda:::qatd_cpp_tbb_enabled() RcppParallel::defaultNumThreads() *Then I have* tried library("quanteda") quanteda_options(threads = *10*) quanteda_options("threads") sessionInfo() Sys.getenv('RCPP_PARALLEL_NUM_THREADS') quanteda:::qatd_cpp_tbb_enabled() RcppParallel::defaultNumThreads() *And both give me the same log with 2 threads out of 24 and the misleading **"Using 1 of 24 threads for parallel computing"* *##########################* Loading required package: methods quanteda version 0.99 *Using 1 of 24 threads for parallel computing* Attaching package: ‘quanteda’ The following object is masked from ‘package:utils’: View *[1] 2* *R version 3.4.1 (2017-06-30)* Platform: x86_64-pc-linux-gnu (64-bit) Running under: Debian GNU/Linux buster/sid Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1 locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] methods stats graphics grDevices utils datasets base other attached packages: [1] quanteda_0.99 loaded via a namespace (and not attached): [1] Rcpp_0.12.12 lubridate_1.6.0 lattice_0.20-35 [4] grid_3.4.1 plyr_1.8.4 gtable_0.2.0 [7] magrittr_1.5 scales_0.5.0 RcppParallel_4.3.20 [10] ggplot2_2.2.1 rlang_0.1.2 stringi_1.1.5 [13] lazyeval_0.2.0 data.table_1.10.4 Matrix_1.2-11 [16] fastmatch_1.1-0 tools_3.4.1 stringr_1.2.0 [19] munsell_0.4.3 compiler_3.4.1 colorspace_1.3-2 [22] tibble_1.3.4 *[1] "2"* *[1] TRUE* *[1] 24*

…

On Wed, Sep 13, 2017 at 12:04 PM, Kenneth Benoit ***@***.***> wrote: Have you tried setting the number of threads through: quanteda_options(threads = 2) for instance? It could be that on the virtual environment, it's setting the default to 1. This could happen if your VE has just two virtual cores, since the default upon loading is now to use half of the available cores. What does the startup message show? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#966 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AecTqH-xtQIJbZgwq6vPRWOiaX4Yc_c3ks5sh6iegaJpZM4PV3cc> .

[koheiw]

Please install the latest version 0.99.6 on Gtihub - we fixed this today.

[ljkhalil]

Dear Sir, It is working now. Thank you Lionel Khalil

…

On 14 Sep 2017, at 21:36, Kohei Watanabe ***@***.***> wrote: Please the latest version 0.996 on Gtihub. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.