Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pbapply hanging and when lapply succeeds #30

Closed
dashaub opened this issue Jan 23, 2018 · 4 comments
Closed

pbapply hanging and when lapply succeeds #30

dashaub opened this issue Jan 23, 2018 · 4 comments

Comments

@dashaub
Copy link

@dashaub dashaub commented Jan 23, 2018

I'm encountering a strange scenario where lapply succeeds but pblapply hangs with a mysterious Selection: message. Strangely this also occurs on only a very small subset of my data (50:60, but but lapply and pblapply work on other slices of the data).

Regular lapply works fine

lapply(cleaned[50:60], FUN = function(x) tsFeatures(x$x))

pblapply hangs

cl <- makeForkCluster(8)
features <- rbindlist(pblapply(X = cleaned[50:60], FUN = function(x) tsFeatures(x$x), cl = cl))
   |                                                  | 0 % ~calculating  Selection: 

tsFeatures() is complex function that I wrote that contains many package dependencies, and cleaned is similarly somewhat complex list, so apologies that I can't post on github a minimal reproducible example. I might be able to send these in a .RData file if you like.

@dashaub
Copy link
Author

@dashaub dashaub commented Jan 23, 2018

Thinking about it, the Selection: message looks like the CRAN mirror selection menu from install.packages() or update.packages().

> update.packages()
--- Please select a CRAN mirror for use in this session ---
Secure CRAN mirrors 

 1: 0-Cloud [https]                   2: Algeria [https]                
 3: Australia (Canberra) [https]      4: Australia (Melbourne 1) [https]
 5: Australia (Melbourne 2) [https]   6: Australia (Perth) [https]      
 7: Austria [https]                   8: Belgium (Ghent) [https]        
 9: Brazil (PR) [https]              10: Brazil (RJ) [https]            
11: Brazil (SP 1) [https]            12: Brazil (SP 2) [https]          
13: Bulgaria [https]                 14: Chile 1 [https]                
15: Chile 2 [https]                  16: China (Guangzhou) [https]      
17: China (Lanzhou) [https]          18: China (Shanghai) [https]       
19: Colombia (Cali) [https]          20: Czech Republic [https]         
21: Denmark [https]                  22: East Asia [https]              
23: Ecuador (Cuenca) [https]         24: Estonia [https]                
25: France (Lyon 1) [https]          26: France (Lyon 2) [https]        
27: France (Marseille) [https]       28: France (Montpellier) [https]   
29: France (Paris 2) [https]         30: Germany (Göttingen) [https]    
31: Germany (Münster) [https]        32: Greece [https]                 
33: Iceland [https]                  34: Indonesia (Jakarta) [https]    
35: Ireland [https]                  36: Italy (Padua) [https]          
37: Japan (Tokyo) [https]            38: Japan (Yonezawa) [https]       
39: Malaysia [https]                 40: Mexico (Mexico City) [https]   
41: Norway [https]                   42: Philippines [https]            
43: Serbia [https]                   44: Spain (A Coruña) [https]       
45: Spain (Madrid) [https]           46: Sweden [https]                 
47: Switzerland [https]              48: Turkey (Denizli) [https]       
49: Turkey (Mersin) [https]          50: UK (Bristol) [https]           
51: UK (Cambridge) [https]           52: UK (London 1) [https]          
53: USA (CA 1) [https]               54: USA (IA) [https]               
55: USA (KS) [https]                 56: USA (MI 1) [https]             
57: USA (NY) [https]                 58: USA (OR) [https]               
59: USA (TN) [https]                 60: USA (TX 1) [https]             
61: Vietnam [https]                  62: (other mirrors)                


Selection:

Not sure why this would need to be called in pblapply but not lappply. Any ideas?

@psolymos
Copy link
Owner

@psolymos psolymos commented Jan 23, 2018

@dashaub I am not sure why the Selection: message without looking at your code (you can send it privately if you want me to look at it). It is likely related to install/update packages as you pointed out but not sure why, given that it only happens to some part of the jobs and not for others.

I tried Fork cluster with other examples and it worked. You can check if your code works with parallel::parLapply and parallel::mclapply. If you get a similar message, then you've got parallel to blame; but if it works fine, then I'd say it is a pbapply issue.

BTW, I wasn't really paying attention to what parallel::makeForkCluster is doing, but it is pretty sweet that it does forking without calling parallel::mclapply but much faster (it is a known limitation).

@dashaub
Copy link
Author

@dashaub dashaub commented Jan 23, 2018

Very helpful debug suggestion. Same problem with parLapply. But when I run it with mclapply I get a segfault and a useful diagnostic to follow.

res <- parallel::mclapply(X = cleaned[50:60], FUN = function(x) tsFeatures(x$x))

 *** caught segfault ***
address 0x110, cause 'memory not mapped'

Traceback:
 1: La.svd(x, nu, nv)
 2: svd(x, nu = 0, nv = k)
 3: prcomp.default(phantom, scale = TRUE)
 4: prcomp(phantom, scale = TRUE)
 5: as.matrix(prcomp(phantom, scale = TRUE)$x[, 2:(qstar + 1)])

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:

Thanks for your help and work on this great package.

@dashaub dashaub closed this Jan 23, 2018
@psolymos
Copy link
Owner

@psolymos psolymos commented Jan 23, 2018

That's really helpful. Thanks for reporting back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.