New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pbapply hanging and when lapply succeeds #30

Closed
dashaub opened this Issue Jan 23, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@dashaub

dashaub commented Jan 23, 2018

I'm encountering a strange scenario where lapply succeeds but pblapply hangs with a mysterious Selection: message. Strangely this also occurs on only a very small subset of my data (50:60, but but lapply and pblapply work on other slices of the data).

Regular lapply works fine

lapply(cleaned[50:60], FUN = function(x) tsFeatures(x$x))

pblapply hangs

cl <- makeForkCluster(8)
features <- rbindlist(pblapply(X = cleaned[50:60], FUN = function(x) tsFeatures(x$x), cl = cl))
   |                                                  | 0 % ~calculating  Selection: 

tsFeatures() is complex function that I wrote that contains many package dependencies, and cleaned is similarly somewhat complex list, so apologies that I can't post on github a minimal reproducible example. I might be able to send these in a .RData file if you like.

@dashaub

This comment has been minimized.

dashaub commented Jan 23, 2018

Thinking about it, the Selection: message looks like the CRAN mirror selection menu from install.packages() or update.packages().

> update.packages()
--- Please select a CRAN mirror for use in this session ---
Secure CRAN mirrors 

 1: 0-Cloud [https]                   2: Algeria [https]                
 3: Australia (Canberra) [https]      4: Australia (Melbourne 1) [https]
 5: Australia (Melbourne 2) [https]   6: Australia (Perth) [https]      
 7: Austria [https]                   8: Belgium (Ghent) [https]        
 9: Brazil (PR) [https]              10: Brazil (RJ) [https]            
11: Brazil (SP 1) [https]            12: Brazil (SP 2) [https]          
13: Bulgaria [https]                 14: Chile 1 [https]                
15: Chile 2 [https]                  16: China (Guangzhou) [https]      
17: China (Lanzhou) [https]          18: China (Shanghai) [https]       
19: Colombia (Cali) [https]          20: Czech Republic [https]         
21: Denmark [https]                  22: East Asia [https]              
23: Ecuador (Cuenca) [https]         24: Estonia [https]                
25: France (Lyon 1) [https]          26: France (Lyon 2) [https]        
27: France (Marseille) [https]       28: France (Montpellier) [https]   
29: France (Paris 2) [https]         30: Germany (Göttingen) [https]    
31: Germany (Münster) [https]        32: Greece [https]                 
33: Iceland [https]                  34: Indonesia (Jakarta) [https]    
35: Ireland [https]                  36: Italy (Padua) [https]          
37: Japan (Tokyo) [https]            38: Japan (Yonezawa) [https]       
39: Malaysia [https]                 40: Mexico (Mexico City) [https]   
41: Norway [https]                   42: Philippines [https]            
43: Serbia [https]                   44: Spain (A Coruña) [https]       
45: Spain (Madrid) [https]           46: Sweden [https]                 
47: Switzerland [https]              48: Turkey (Denizli) [https]       
49: Turkey (Mersin) [https]          50: UK (Bristol) [https]           
51: UK (Cambridge) [https]           52: UK (London 1) [https]          
53: USA (CA 1) [https]               54: USA (IA) [https]               
55: USA (KS) [https]                 56: USA (MI 1) [https]             
57: USA (NY) [https]                 58: USA (OR) [https]               
59: USA (TN) [https]                 60: USA (TX 1) [https]             
61: Vietnam [https]                  62: (other mirrors)                


Selection:

Not sure why this would need to be called in pblapply but not lappply. Any ideas?

@psolymos

This comment has been minimized.

Owner

psolymos commented Jan 23, 2018

@dashaub I am not sure why the Selection: message without looking at your code (you can send it privately if you want me to look at it). It is likely related to install/update packages as you pointed out but not sure why, given that it only happens to some part of the jobs and not for others.

I tried Fork cluster with other examples and it worked. You can check if your code works with parallel::parLapply and parallel::mclapply. If you get a similar message, then you've got parallel to blame; but if it works fine, then I'd say it is a pbapply issue.

BTW, I wasn't really paying attention to what parallel::makeForkCluster is doing, but it is pretty sweet that it does forking without calling parallel::mclapply but much faster (it is a known limitation).

@dashaub

This comment has been minimized.

dashaub commented Jan 23, 2018

Very helpful debug suggestion. Same problem with parLapply. But when I run it with mclapply I get a segfault and a useful diagnostic to follow.

res <- parallel::mclapply(X = cleaned[50:60], FUN = function(x) tsFeatures(x$x))

 *** caught segfault ***
address 0x110, cause 'memory not mapped'

Traceback:
 1: La.svd(x, nu, nv)
 2: svd(x, nu = 0, nv = k)
 3: prcomp.default(phantom, scale = TRUE)
 4: prcomp(phantom, scale = TRUE)
 5: as.matrix(prcomp(phantom, scale = TRUE)$x[, 2:(qstar + 1)])

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:

Thanks for your help and work on this great package.

@dashaub dashaub closed this Jan 23, 2018

@psolymos

This comment has been minimized.

Owner

psolymos commented Jan 23, 2018

That's really helpful. Thanks for reporting back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment