pbapply hanging and when lapply succeeds

[dashaub]

I'm encountering a strange scenario where lapply succeeds but pblapply hangs with a mysterious Selection: message. Strangely this also occurs on only a very small subset of my data (50:60, but but lapply and pblapply work on other slices of the data).

Regular lapply works fine

lapply(cleaned[50:60], FUN = function(x) tsFeatures(x$x))

pblapply hangs

cl <- makeForkCluster(8)
features <- rbindlist(pblapply(X = cleaned[50:60], FUN = function(x) tsFeatures(x$x), cl = cl))
   |                                                  | 0 % ~calculating  Selection: 

tsFeatures() is complex function that I wrote that contains many package dependencies, and cleaned is similarly somewhat complex list, so apologies that I can't post on github a minimal reproducible example. I might be able to send these in a .RData file if you like.

[dashaub]

Thinking about it, the Selection: message looks like the CRAN mirror selection menu from install.packages() or update.packages().

> update.packages()
--- Please select a CRAN mirror for use in this session ---
Secure CRAN mirrors 

 1: 0-Cloud [https]                   2: Algeria [https]                
 3: Australia (Canberra) [https]      4: Australia (Melbourne 1) [https]
 5: Australia (Melbourne 2) [https]   6: Australia (Perth) [https]      
 7: Austria [https]                   8: Belgium (Ghent) [https]        
 9: Brazil (PR) [https]              10: Brazil (RJ) [https]            
11: Brazil (SP 1) [https]            12: Brazil (SP 2) [https]          
13: Bulgaria [https]                 14: Chile 1 [https]                
15: Chile 2 [https]                  16: China (Guangzhou) [https]      
17: China (Lanzhou) [https]          18: China (Shanghai) [https]       
19: Colombia (Cali) [https]          20: Czech Republic [https]         
21: Denmark [https]                  22: East Asia [https]              
23: Ecuador (Cuenca) [https]         24: Estonia [https]                
25: France (Lyon 1) [https]          26: France (Lyon 2) [https]        
27: France (Marseille) [https]       28: France (Montpellier) [https]   
29: France (Paris 2) [https]         30: Germany (Göttingen) [https]    
31: Germany (Münster) [https]        32: Greece [https]                 
33: Iceland [https]                  34: Indonesia (Jakarta) [https]    
35: Ireland [https]                  36: Italy (Padua) [https]          
37: Japan (Tokyo) [https]            38: Japan (Yonezawa) [https]       
39: Malaysia [https]                 40: Mexico (Mexico City) [https]   
41: Norway [https]                   42: Philippines [https]            
43: Serbia [https]                   44: Spain (A Coruña) [https]       
45: Spain (Madrid) [https]           46: Sweden [https]                 
47: Switzerland [https]              48: Turkey (Denizli) [https]       
49: Turkey (Mersin) [https]          50: UK (Bristol) [https]           
51: UK (Cambridge) [https]           52: UK (London 1) [https]          
53: USA (CA 1) [https]               54: USA (IA) [https]               
55: USA (KS) [https]                 56: USA (MI 1) [https]             
57: USA (NY) [https]                 58: USA (OR) [https]               
59: USA (TN) [https]                 60: USA (TX 1) [https]             
61: Vietnam [https]                  62: (other mirrors)                


Selection:

Not sure why this would need to be called in pblapply but not lappply. Any ideas?

[psolymos]

@dashaub I am not sure why the Selection: message without looking at your code (you can send it privately if you want me to look at it). It is likely related to install/update packages as you pointed out but not sure why, given that it only happens to some part of the jobs and not for others.

I tried Fork cluster with other examples and it worked. You can check if your code works with parallel::parLapply and parallel::mclapply. If you get a similar message, then you've got parallel to blame; but if it works fine, then I'd say it is a pbapply issue.

BTW, I wasn't really paying attention to what parallel::makeForkCluster is doing, but it is pretty sweet that it does forking without calling parallel::mclapply but much faster (it is a known limitation).

[dashaub]

Very helpful debug suggestion. Same problem with parLapply. But when I run it with mclapply I get a segfault and a useful diagnostic to follow.

res <- parallel::mclapply(X = cleaned[50:60], FUN = function(x) tsFeatures(x$x))

 *** caught segfault ***
address 0x110, cause 'memory not mapped'

Traceback:
 1: La.svd(x, nu, nv)
 2: svd(x, nu = 0, nv = k)
 3: prcomp.default(phantom, scale = TRUE)
 4: prcomp(phantom, scale = TRUE)
 5: as.matrix(prcomp(phantom, scale = TRUE)$x[, 2:(qstar + 1)])

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:

Thanks for your help and work on this great package.

[psolymos]

That's really helpful. Thanks for reporting back.