<simpleError in lock("test.lock", exclusive = TRUE, timeout = Inf): Cannot lock file: '/path/to/file/test.lock': No locks available>

[bhoellbacher]

I ran into this error which shows up unpredictably when using filelock on parallel processes.

Rerunning the exact same code multiple times, the processes will finish without error sometimes, but throw an error around 50% of the times. I can't figure out what triggers it.

Here a "reproducible" example:

library(parallel)
library(filelock)

a <- function(){
  lck <- lock("test.lock", exclusive = TRUE, timeout = Inf)
  Sys.sleep(2) #simulate a process that takes a while
  on.exit( unlock(lck) )
  print('Done')
}

b <- function(){
  lck <- lock("test.lock", exclusive = TRUE, timeout = Inf) 
  on.exit( unlock(lck) )
  print("Done")
}


#Start the first process which will have a hold on file for 2 seconds
mcparallel(a())
#Start a second processes that is trying to access the same lock file
mcparallel(b())

mccollect()

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 10 (buster)

Matrix products: default
BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.5.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C              LC_PAPER=en_US.UTF-8      
 [8] LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] filelock_1.0.2

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.4.6                knitr_1.28                  XVector_0.26.0              GenomicRanges_1.38.0        BiocGenerics_0.32.0         zlibbioc_1.32.0             GenomicAlignments_1.22.1   
 [8] IRanges_2.20.2              BiocParallel_1.20.1         lattice_0.20-38             rlang_0.4.5                 jpeg_0.1-8.1                hwriter_1.3.2               GenomeInfoDb_1.22.1        
[15] tools_3.6.3                 SummarizedExperiment_1.16.1 grid_3.6.3                  Biobase_2.46.0              xfun_0.13                   png_0.1-7                   latticeExtra_0.6-29        
[22] htmltools_0.4.0             matrixStats_0.56.0          yaml_2.2.1                  digest_0.6.25               Matrix_1.2-18               GenomeInfoDbData_1.2.2      RColorBrewer_1.1-2         
[29] S4Vectors_0.24.4            bitops_1.0-6                RCurl_1.98-1.2              evaluate_0.14               rmarkdown_2.1               DelayedArray_0.12.3         compiler_3.6.3             
[36] Biostrings_2.54.0           Rsamtools_2.2.3             ShortRead_1.44.3            stats4_3.6.3

[gaborcsardi]

I cannot reproduce this with filelock 1.0.2 on R-devel or on R 3.6.3, just ran it 50 times on both.

Maybe you could try creating a reproducible example with reprex::reprex(), to make sure that all those packages that you have loaded don't influence this?

[gaborcsardi]

Are you running this on a shared file system, btw?

[bhoellbacher]

Are you running this on a shared file system, btw?

Yes, it is a shared file system.

[bhoellbacher]

I cannot reproduce this with filelock 1.0.2 on R-devel or on R 3.6.3, just ran it 50 times on both.

Maybe you could try creating a reproducible example with reprex::reprex(), to make sure that all those packages that you have loaded don't influence this?

Will do, as soon as I get reprex to work.

reprex::reprex(input = "filelock_example.R", outfile = "filelock_example.md")
Preparing reprex as .R file:
  * filelock_example_reprex.R
Rendering reprex...
Error in get_result(output = out, options) : 
  callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed

Calling rcall separately and setting the user_profile=FALSE (like suggested by others) doesn't work either.

> callr::r(function() var(iris[, 1:4]), user_profile = FALSE)
Error in get_result(output = out, options) : 
  callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed
Type .Last.error.trace to see where the error occured

Running the trace yields:

> .Last.error.trace

 Stack trace:

 1. callr::r(function() var(iris[, 1:4]), user_profile = FALSE)
 2. callr:::get_result(output = out, options)
 3. throw(new_callr_error(output, killmsg))

 x callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed 

Could it be something about my system that causes issues with both, the file locking and the rcall?

[gaborcsardi]

Yes, it is a shared file system.

Shared filesystems often do not support proper locking. I am afraid that you'll need to put the lock files somewhere else, e.g. a user lock directory or the temporary directory.

[bhoellbacher]

Yes, it is a shared file system.

Shared filesystems often do not support proper locking. I am afraid that you'll need to put the lock files somewhere else, e.g. a user lock directory or the temporary directory.

Saving the file on a local tmp directory instead of the shared filesystem did the trick.
Thank you for your help and the prompt replies!

[gaborcsardi]

For the record #1 would help with this, and so would documenting that there are issues with shared file systems.