make.prerun.object() cluster initialization issue on SLURM cluster

[Neves-P]

make.prerun.object() cluster initialization issue

Running make.prerun.object() from the instructions on Chapter 8 of the Light-level geolocation analyses manual on a single node job on a Linux HPC cluster running SLURM fails with:

making cluster
Error in socketAccept(socket = socket, blocking = TRUE, open = "a+b",  : 
  all connections are in use
Calls: make.prerun.object ... get.Phys.Mat.parallel -> <Anonymous> -> makePSOCKcluster -> socketAccept
Execution halted

Cause

This happens because parallel::makeCluster() is called via mycl <- parallel::makeCluster(threads) and the cluster type is left as default "PSOCK".

The workaround to this is to allow a user to choose a cluster type and to initialize it correctly:

hosts <- rep("localhost",threads)
mycl <- parallel::makeCluster(hosts, type=cluster.type)

Reprex and session info

─ Session info ───────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.2 (2023-10-31)
 os       Rocky Linux 8.9 (Green Obsidian)
 system   x86_64, linux-gnu
 ui       X11
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       Europe/Amsterdam
 date     2024-06-04
 pandoc   NA

─ Packages ───────────────────────────────────────────────────────────────────
 package          * version    date (UTC) lib source
 bit              * 4.0.5      2022-11-15 [1] CRAN (R 4.3.2)
 bitops             1.0-7      2021-04-24 [1] CRAN (R 4.3.2)
 boot             * 1.3-30     2024-02-26 [1] CRAN (R 4.3.2)
 cachem             1.0.8      2023-05-01 [3] CRAN (R 4.3.2)
 callr              3.7.3      2022-11-02 [3] CRAN (R 4.3.2)
 changepoint        2.2.4      2022-11-03 [1] CRAN (R 4.3.2)
 CircStats        * 0.2-6      2018-07-01 [1] CRAN (R 4.3.2)
 circular         * 0.5-0      2023-09-05 [1] CRAN (R 4.3.2)
 class              7.3-22     2023-05-03 [1] CRAN (R 4.3.2)
 classInt           0.4-10     2023-09-05 [1] CRAN (R 4.3.2)
 cli                3.6.2      2023-12-11 [1] CRAN (R 4.3.2)
 coda               0.19-4.1   2024-01-31 [1] CRAN (R 4.3.2)
 codetools          0.2-20     2024-03-31 [1] CRAN (R 4.3.2)
 colorspace         2.1-0      2023-01-23 [1] CRAN (R 4.3.2)
 crayon             1.5.2      2022-09-29 [3] CRAN (R 4.3.2)
 curl               5.2.1      2024-03-01 [1] CRAN (R 4.3.2)
 DBI                1.2.3      2024-06-02 [1] CRAN (R 4.3.2)
 devtools         * 2.4.5      2022-10-11 [3] CRAN (R 4.3.2)
 digest           * 0.6.35     2024-03-11 [1] CRAN (R 4.3.2)
 dotCall64          1.1-1      2023-11-28 [1] CRAN (R 4.3.2)
 dplyr              1.1.4      2023-11-17 [1] CRAN (R 4.3.2)
 e1071              1.7-14     2023-12-06 [1] CRAN (R 4.3.2)
 ellipsis           0.3.2      2021-04-29 [3] CRAN (R 4.3.2)
 fansi              1.0.6      2023-12-08 [1] CRAN (R 4.3.2)
 fastmap            1.1.1      2023-02-24 [3] CRAN (R 4.3.2)
 fields           * 15.2       2023-08-17 [1] CRAN (R 4.3.2)
 FLightR          * 0.5.5      2024-06-03 [1] Github (Neves-P/FLightR@7229a94)
 forecast         * 8.22.0     2024-03-04 [1] CRAN (R 4.3.2)
 foreign            0.8-86     2023-11-28 [2] CRAN (R 4.3.2)
 fracdiff           1.5-3      2024-02-01 [1] CRAN (R 4.3.2)
 fs                 1.6.4      2024-04-25 [1] CRAN (R 4.3.2)
 generics           0.1.3      2022-07-05 [1] CRAN (R 4.3.2)
 GeoLight         * 2.1.0      2024-05-28 [1] Github (SLisovski/GeoLight@360c909)
 GeoLocTools      * 0.1.0      2024-05-28 [1] Github (SLisovski/GeoLocTools@4455410)
 geosphere        * 1.5-18     2022-11-15 [1] CRAN (R 4.3.2)
 ggmap            * 4.0.0      2023-11-19 [1] CRAN (R 4.3.2)
 ggplot2          * 3.5.1      2024-04-23 [1] CRAN (R 4.3.2)
 glue               1.7.0      2024-01-09 [1] CRAN (R 4.3.2)
 gtable             0.3.5      2024-04-22 [1] CRAN (R 4.3.2)
 htmltools          0.5.8.1    2024-04-04 [1] CRAN (R 4.3.2)
 htmlwidgets        1.6.4      2023-12-06 [1] CRAN (R 4.3.2)
 httpuv             1.6.15     2024-03-26 [1] CRAN (R 4.3.2)
 httr               1.4.7      2023-08-15 [3] CRAN (R 4.3.2)
 jpeg               0.1-10     2022-11-29 [1] CRAN (R 4.3.2)
 jsonlite           1.8.8      2023-12-04 [1] CRAN (R 4.3.2)
 KernSmooth         2.23-24    2024-05-17 [1] CRAN (R 4.3.2)
 later              1.3.2      2023-12-06 [1] CRAN (R 4.3.2)
 lattice            0.22-6     2024-03-20 [1] CRAN (R 4.3.2)
 lifecycle          1.0.4      2023-11-07 [1] CRAN (R 4.3.2)
 lmtest             0.9-40     2022-03-21 [1] CRAN (R 4.3.2)
 magrittr           2.0.3      2022-03-30 [3] CRAN (R 4.3.2)
 mapdata          * 2.3.1      2022-11-01 [1] CRAN (R 4.3.2)
 maps             * 3.4.2      2023-12-15 [1] CRAN (R 4.3.2)
 maptools         * 1.1-8      2023-07-18 [2] CRAN (R 4.3.2)
 MASS             * 7.3-59     2023-04-21 [1] CRAN (R 4.3.2)
 memoise            2.0.1      2021-11-26 [3] CRAN (R 4.3.2)
 mime               0.12       2021-09-28 [3] CRAN (R 4.3.2)
 miniUI             0.1.1.1    2018-05-18 [3] CRAN (R 4.3.2)
 munsell            0.5.1      2024-04-01 [1] CRAN (R 4.3.2)
 mvtnorm            1.2-5      2024-05-21 [1] CRAN (R 4.3.2)
 ncdf4            * 1.22       2023-11-28 [2] CRAN (R 4.3.2)
 nlme               3.1-164    2023-11-27 [1] CRAN (R 4.3.2)
 nnet               7.3-19     2023-05-03 [1] CRAN (R 4.3.2)
 paletteer          1.6.0      2024-01-21 [1] CRAN (R 4.3.2)
 patchwork          1.2.0      2024-01-08 [1] CRAN (R 4.3.2)
 pillar             1.9.0      2023-03-22 [3] CRAN (R 4.3.2)
 pkgbuild           1.4.2      2023-06-26 [3] CRAN (R 4.3.2)
 pkgconfig          2.0.3      2019-09-22 [3] CRAN (R 4.3.2)
 pkgload            1.3.3      2023-09-22 [3] CRAN (R 4.3.2)
 plyr               1.8.9      2023-10-02 [1] CRAN (R 4.3.2)
 png                0.1-8      2022-11-29 [1] CRAN (R 4.3.2)
 PolarGeolocation * 0.1.0      2024-05-28 [1] Github (SLisovski/PolarGeolocation@6f321f0)
 prettyunits        1.2.0      2023-09-24 [3] CRAN (R 4.3.2)
 probGLS          * 0.9.7      2024-05-28 [1] Github (benjamin-merkel/probGLS@e92cd33)
 processx           3.8.2      2023-06-30 [3] CRAN (R 4.3.2)
 profvis            0.3.8      2023-05-02 [3] CRAN (R 4.3.2)
 promises           1.3.0      2024-04-05 [1] CRAN (R 4.3.2)
 proxy              0.4-27     2022-06-09 [1] CRAN (R 4.3.2)
 ps                 1.7.5      2023-04-18 [3] CRAN (R 4.3.2)
 purrr              1.0.2      2023-08-10 [3] CRAN (R 4.3.2)
 quadprog           1.5-8      2019-11-20 [1] CRAN (R 4.3.2)
 quantmod           0.4.26     2024-02-14 [1] CRAN (R 4.3.2)
 R6                 2.5.1      2021-08-19 [3] CRAN (R 4.3.2)
 raster           * 3.6-26     2023-10-14 [2] CRAN (R 4.3.2)
 Rcpp             * 1.0.12     2024-01-09 [1] CRAN (R 4.3.2)
 RcppArmadillo    * 0.12.8.4.0 2024-05-31 [1] CRAN (R 4.3.2)
 rematch2           2.1.2      2020-05-01 [3] CRAN (R 4.3.2)
 remotes            2.4.2.1    2023-07-18 [3] CRAN (R 4.3.2)
 rgdal            * 1.6-7      2023-05-31 [2] CRAN (R 4.3.2)
 rgeos            * 0.6-4      2023-07-18 [2] CRAN (R 4.3.2)
 rlang              1.1.3      2024-01-10 [1] CRAN (R 4.3.2)
 rnaturalearth      1.0.1      2023-12-15 [1] CRAN (R 4.3.2)
 scales             1.3.0      2023-11-28 [1] CRAN (R 4.3.2)
 sessioninfo        1.2.2      2021-12-06 [1] CRAN (R 4.3.2)
 sf               * 1.0-16     2024-03-24 [1] CRAN (R 4.3.2)
 SGAT             * 0.1.3      2024-05-28 [1] Github (SWotherspoon/SGAT@d089484)
 shiny              1.8.1.1    2024-04-02 [1] CRAN (R 4.3.2)
 sp               * 2.1-4      2024-04-30 [1] CRAN (R 4.3.2)
 spam             * 2.10-0     2023-10-23 [1] CRAN (R 4.3.2)
 stringi            1.8.4      2024-05-06 [1] CRAN (R 4.3.2)
 stringr            1.5.1      2023-11-14 [1] CRAN (R 4.3.2)
 svglite          * 2.1.3      2023-12-08 [1] CRAN (R 4.3.2)
 systemfonts        1.0.5      2023-10-09 [3] CRAN (R 4.3.2)
 terra            * 1.7-78     2024-05-22 [1] CRAN (R 4.3.2)
 tibble             3.2.1      2023-03-20 [3] CRAN (R 4.3.2)
 tidyr              1.3.1      2024-01-24 [1] CRAN (R 4.3.2)
 tidyselect         1.2.1      2024-03-11 [1] CRAN (R 4.3.2)
 timeDate           4032.109   2023-12-14 [1] CRAN (R 4.3.2)
 truncnorm        * 1.0-9      2023-03-20 [1] CRAN (R 4.3.2)
 tseries            0.10-56    2024-05-14 [1] CRAN (R 4.3.2)
 TTR                0.24.4     2023-11-28 [1] CRAN (R 4.3.2)
 TwGeos           * 0.1.2      2024-05-28 [1] Github (SLisovski/TwGeos@028f499)
 units              0.8-5      2023-11-28 [2] CRAN (R 4.3.2)
 urca               1.3-4      2024-05-27 [1] CRAN (R 4.3.2)
 urlchecker         1.0.1      2021-11-30 [3] CRAN (R 4.3.2)
 usethis          * 2.2.2      2023-07-06 [3] CRAN (R 4.3.2)
 utf8               1.2.4      2023-10-22 [3] CRAN (R 4.3.2)
 vctrs              0.6.5      2023-12-01 [1] CRAN (R 4.3.2)
 viridisLite      * 0.4.2      2023-05-02 [1] CRAN (R 4.3.2)
 withr              3.0.0      2024-01-16 [1] CRAN (R 4.3.2)
 xtable             1.8-4      2019-04-21 [3] CRAN (R 4.3.2)
 xts                0.13.2     2024-01-21 [1] CRAN (R 4.3.2)
 zoo              * 1.8-12     2023-04-13 [1] CRAN (R 4.3.2)

 [1] /home4/p282067/R/x86_64-pc-linux-gnu-library/4.3
 [2] /cvmfs/hpc.rug.nl/versions/2023.01/rocky8/x86_64/amd/zen3/software/R-bundle-CRAN/2023.12-foss-2023a
 [3] /cvmfs/hpc.rug.nl/versions/2023.01/rocky8/x86_64/amd/zen3/software/R/4.3.2-gfbf-2023a/lib64/R/library

──────────────────────────────────────────────────────────────────────────────

Running a SLURM job with settings:

#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=12GB

Rscript myscript.R

With myscript.R:

print(Sys.time())
url <- "https://github.com/slisovski/TheGeolocationManual/raw/master/download.zip"

temp <- tempfile()
download.file(url, temp)
unzip(temp, exdir = "data")

library(GeoLocTools)
setupGeolocation()

ID <- "M034"
wd <- "data"
Species <- "LanCol"

lon.calib <- 12.33
lat.calib <- 55.98

message("reading file")
raw <- readMTlux(paste0(wd, "/RawData/", Species, "/", ID, ".lux"))
names(raw) <- c("Date", "Light")
raw$Light  <- log(raw$Light+0.0001) + abs(min(log(raw$Light+0.0001)))
head(raw)

offset <- 12 # adjusts the y-axis to put night (dark shades) in the middle

### Looking at the raw data
message("Looking at the raw data")
lightImage( tagdata = raw,
            offset = offset,
            zlim = c(0, 10))

### Getting the twilight annotation
message("Getting the twilight annotation")
FLightR.data<-get.tags.data(paste0(wd, "/Results/", Species, "/", ID, "_twl.csv"))

#### Calibration ####
message("Calibration")
Calibration.periods<-data.frame(
  calibration.start=as.POSIXct(c(NA, '2015-05-15')),
  calibration.stop=as.POSIXct(c('2014-07-15', NA)),
  lon=lon.calib, lat=lat.calib)
# use c() also for the geographic coordinates,
# if you have more than one calibration location
# (e. g.,  lon=c(5.43, 6.00), lat=c(52.93,52.94))
print(Calibration.periods)

Calibration<-make.calibration(FLightR.data, Calibration.periods, model.ageing=TRUE, plot.final = T)

#### Spatial extent ####
message("Spatial extent")
Grid <- make.grid(left=5, bottom=-33, right=50, top=60,
                  distance.from.land.allowed.to.use=c(-Inf, 200),
                  distance.from.land.allowed.to.stay=c(-Inf, 50))

#### Preparing model run ####
# ~ 15 min run time
cpu <- as.numeric(Sys.getenv("SLURM_CPUS_ON_NODE", 1))
message("cpus: ", cpu)
all.in <- make.prerun.object(FLightR.data, Grid, start=c(lon.calib, lat.calib),
                             Calibration=Calibration, M.mean=750, threads = cpu)

On a compute node with two AMD 7763 (hence 128 cores total).

[eldarrak]

Thank you for your suggestions! Implemeted now.