Parallel option not working

[davidhen]

Hi Colin,

I was having a play around with this this morning. The standard function seems to work fine, but when I use the cores = .. argument I get an error as below. I've tried with 2, 4, and 8 cores with the same result.

# Programming benchmarks (5 tests):
# starting worker pid=10496 on localhost:11728 at 09:56:33.005
# starting worker pid=10495 on localhost:11728 at 09:56:33.005
# Error in unserialize(node$con) : error reading from connection
# Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
# Error in unserialize(node$con) : error reading from connection
# Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
# Execution halted
# Execution halted
# Error in makePSOCKcluster(names = spec, ...) : 
#  Cluster setup failed. 2 of 2 workers failed to connect.

What is really weird, is that when I created a reprex for this issue, it seemed to work...

library(benchmarkme)

benchmark_std(cores = 2L)
#> # Programming benchmarks (5 tests):
#> # Matrix calculation benchmarks (5 tests):
#> # Matrix function benchmarks (5 tests):
#>      user system elapsed          test test_group cores
#> 2   0.020  0.001   0.546           fib       prog     2
#> 3   0.004  0.001   0.383           fib       prog     2
#> 4   0.003  0.000   0.377           fib       prog     2
#> 21  0.003  0.000   1.172           gcd       prog     2
#> 31  0.003  0.000   0.831           gcd       prog     2
#> 41  0.003  0.001   0.905           gcd       prog     2
#> 22  0.004  0.001   0.694       hilbert       prog     2
#> 32  0.004  0.000   0.306       hilbert       prog     2
#> 42  0.004  0.000   0.402       hilbert       prog     2
#> 23  0.003  0.000   1.245      toeplitz       prog     2
#> 33  0.003  0.000   1.211      toeplitz       prog     2
#> 43  0.003  0.000   1.211      toeplitz       prog     2
#> 24  0.004  0.000   1.151     escoufier       prog     2
#> 34  0.004  0.000   1.098     escoufier       prog     2
#> 44  0.003  0.000   1.089     escoufier       prog     2
#> 25  0.003  0.000   0.990         manip matrix_cal     2
#> 35  0.003  0.000   0.725         manip matrix_cal     2
#> 45  0.004  0.000   0.714         manip matrix_cal     2
#> 211 0.003  0.000   0.956         power matrix_cal     2
#> 311 0.003  0.000   0.792         power matrix_cal     2
#> 411 0.003  0.000   0.715         power matrix_cal     2
#> 221 0.004  0.000   1.437          sort matrix_cal     2
#> 321 0.003  0.001   1.200          sort matrix_cal     2
#> 421 0.004  0.000   1.172          sort matrix_cal     2
#> 231 0.003  0.000   9.929 cross_product matrix_cal     2
#> 331 0.003  0.000   9.361 cross_product matrix_cal     2
#> 431 0.003  0.000   9.455 cross_product matrix_cal     2
#> 241 0.003  0.000   1.126            lm matrix_cal     2
#> 341 0.003  0.000   1.112            lm matrix_cal     2
#> 441 0.004  0.000   1.088            lm matrix_cal     2
#> 26  0.003  0.000  20.535      cholesky matrix_fun     2
#> 36  0.004  0.000  20.175      cholesky matrix_fun     2
#> 46  0.003  0.001  20.170      cholesky matrix_fun     2
#> 212 0.004  0.001   3.566   determinant matrix_fun     2
#> 312 0.003  0.000   3.493   determinant matrix_fun     2
#> 412 0.003  0.000   3.476   determinant matrix_fun     2
#> 222 0.003  0.000   0.871         eigen matrix_fun     2
#> 322 0.003  0.000   0.841         eigen matrix_fun     2
#> 422 0.004  0.000   0.821         eigen matrix_fun     2
#> 232 0.003  0.000   0.619           fft matrix_fun     2
#> 332 0.004  0.001   0.595           fft matrix_fun     2
#> 432 0.003  0.000   0.578           fft matrix_fun     2
#> 242 0.003  0.000   3.026       inverse matrix_fun     2
#> 342 0.003  0.000   3.000       inverse matrix_fun     2
#> 442 0.003  0.000   2.923       inverse matrix_fun     2

^{Created on 2020-11-18 by the reprex package (v0.3.0.9001)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS  10.16                
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_GB.UTF-8                 
#>  ctype    en_GB.UTF-8                 
#>  tz       Europe/London               
#>  date     2020-11-18                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package         * version    date       lib source                           
#>  assertthat        0.2.1      2019-03-21 [1] CRAN (R 4.0.0)                   
#>  benchmarkme     * 1.0.4      2020-05-09 [1] CRAN (R 4.0.2)                   
#>  benchmarkmeData   1.0.4      2020-04-23 [1] CRAN (R 4.0.2)                   
#>  cli               2.1.0      2020-10-12 [1] CRAN (R 4.0.2)                   
#>  codetools         0.2-16     2018-12-24 [1] CRAN (R 4.0.2)                   
#>  crayon            1.3.4      2017-09-16 [1] CRAN (R 4.0.0)                   
#>  digest            0.6.27     2020-10-24 [1] CRAN (R 4.0.2)                   
#>  doParallel        1.0.16     2020-10-16 [1] CRAN (R 4.0.2)                   
#>  dplyr             1.0.2      2020-08-18 [1] CRAN (R 4.0.2)                   
#>  ellipsis          0.3.1      2020-05-15 [1] CRAN (R 4.0.0)                   
#>  evaluate          0.14       2019-05-28 [1] CRAN (R 4.0.0)                   
#>  fansi             0.4.1      2020-01-08 [1] CRAN (R 4.0.0)                   
#>  foreach           1.5.0      2020-03-30 [1] CRAN (R 4.0.2)                   
#>  fs                1.5.0      2020-07-31 [1] CRAN (R 4.0.2)                   
#>  generics          0.0.2      2018-11-29 [1] CRAN (R 4.0.0)                   
#>  glue              1.4.2      2020-08-27 [1] CRAN (R 4.0.2)                   
#>  highr             0.8        2019-03-20 [1] CRAN (R 4.0.0)                   
#>  htmltools         0.5.0      2020-06-16 [1] CRAN (R 4.0.0)                   
#>  httr              1.4.2      2020-07-20 [1] CRAN (R 4.0.0)                   
#>  iterators         1.0.12     2019-07-26 [1] CRAN (R 4.0.2)                   
#>  knitr             1.30       2020-09-22 [1] CRAN (R 4.0.2)                   
#>  lattice           0.20-41    2020-04-02 [1] CRAN (R 4.0.2)                   
#>  lifecycle         0.2.0      2020-03-06 [1] CRAN (R 4.0.0)                   
#>  magrittr          2.0.1      2020-11-17 [1] CRAN (R 4.0.2)                   
#>  Matrix            1.2-18     2019-11-27 [1] CRAN (R 4.0.2)                   
#>  pillar            1.4.6      2020-07-10 [1] CRAN (R 4.0.2)                   
#>  pkgconfig         2.0.3      2019-09-22 [1] CRAN (R 4.0.0)                   
#>  purrr             0.3.4      2020-04-17 [1] CRAN (R 4.0.0)                   
#>  R6                2.5.0      2020-10-28 [1] CRAN (R 4.0.2)                   
#>  reprex            0.3.0.9001 2020-11-18 [1] Github (tidyverse/reprex@d3fc4b8)
#>  rlang             0.4.8      2020-10-08 [1] CRAN (R 4.0.2)                   
#>  rmarkdown         2.5        2020-10-21 [1] CRAN (R 4.0.2)                   
#>  rstudioapi        0.11       2020-02-07 [1] CRAN (R 4.0.0)                   
#>  sessioninfo       1.1.1      2018-11-05 [1] CRAN (R 4.0.0)                   
#>  stringi           1.5.3      2020-09-09 [1] CRAN (R 4.0.2)                   
#>  stringr           1.4.0      2019-02-10 [1] CRAN (R 4.0.0)                   
#>  tibble            3.0.4      2020-10-12 [1] CRAN (R 4.0.2)                   
#>  tidyselect        1.1.0      2020-05-11 [1] CRAN (R 4.0.0)                   
#>  vctrs             0.3.4      2020-08-29 [1] CRAN (R 4.0.2)                   
#>  withr             2.3.0      2020-09-22 [1] CRAN (R 4.0.2)                   
#>  xfun              0.19       2020-10-30 [1] CRAN (R 4.0.2)                   
#>  yaml              2.2.1      2020-02-01 [1] CRAN (R 4.0.0)                   
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

...but still doesn't work without doing a reprex. I'm guessing a dependency issue? Although I can't figure out what might be missing.

Apologies if this me being stupid... but I can't figure it out.

[csgillespie]

It's something to do with namespaces and parallel. I think if you explicitly load benchmarkme, you shouldn't get this issue. I'll investigate more when I get a minute

[csgillespie]

So I'm fairly sure this is caused when you don't call library(benchmarkme) first.

Unfortunately, I'm certain I don't know how to fix it and get the package on CRAN. In the parallel code, I'm calling non-exported functions, so adding a library(benchmarkme) call to clusterExport doesn't work.

I could export all private functions, but I really don't want to.

---

I've now added a vaguely helpful error message (in the next version). If anyone can think of a fix, feel free to comment.