ncore problem

[subhomoyghosh]

Hi,

I am having some problem with the ncores option in biglasso() and cv.biglasso() function. I compiled biglasso on my mac enabling openMP (4 cores). However, when I am trying to reproduce the worked out example of the cv.biglasso() function usng ncores=4, I am getting an error

Error in checkForRemoteErrors(val) : 
  4 nodes produced errors; first error: could not find function "attach.big.matrix"
In addition: Warning messages:
1: In .Internal(gc(verbose, reset)) :
  closing unused connection 6 (<-localhost:11359)
2: In .Internal(gc(verbose, reset)) :
  closing unused connection 5 (<-localhost:11359)
3: In .Internal(gc(verbose, reset)) :
  closing unused connection 4 (<-localhost:11359)
4: In .Internal(gc(verbose, reset)) :
  closing unused connection 3 (<-localhost:11359)

I got the same error in our cluster where biglasso is compiled using intel MKL. Can anyone help?

Thanks,
Subhomoy

[privefl]

This is weird because each cluster should load package biglasso and then package bigmemory (because it's a depends).

As a new version of package bigmemory has just been released to CRAN it is worth reinstalling everything. Begin a new clean R session. Install package bigmemory then install package biglasso.

[subhomoyghosh]

Just reinstalled bigmemory and biglasso from cran. Having same problem Error in checkForRemoteErrors(val) : 2 nodes produced errors; first error: could not find function "attach.big.matrix"

[privefl]

Okay.
Then fork the repo.
Find the R/cv.biglasso.R file.
Edit it directly on your GitHub by adding bigmemory:: before attach.big.matrix.
Then install it via install_github of your fork.

Sorry, I have no computer to test/do this now.

[subhomoyghosh]

Thanks. Let me try.

[subhomoyghosh]

Now it is showing Error in checkForRemoteErrors(val) : 2 nodes produced errors; first error: there is no package called ‘bigmemory’
Strange!

[privefl]

See this

[subhomoyghosh]

Thanks, let me take a look at it.

Although I couldn't fixed the issue entirely. I kind of fixed it by changing parLapply part by mclapply as I figured that it was coming from passing functions. Forking shouldn't have this issues ideally. I committed the changes in my forked biglasso repo.

install_github('subhomoyghosh/biglasso')

This would only work on mac/linux based R. Could you possibly test once you have a computer?
Thanks much for the help.

[GabeAl]

Crashes outright on Windows 10. Installed from CRAN. R v3.5.3, biglasso 1.3-6.

I also saw mention of "parLapply" above. That doesn't work on Windows either. A lot of Big Data companies use Windows...

[privefl]

parLapply() does work on Windows.
What is your problem?

[GabeAl]

Thanks privelf,

Apparently I have another problem causing the crash, or I was confusing it with the other parallel apply that doesn't work (it just uses 1 thread instead of all of them). Maybe I was thinking mclapply (lot of parallel interfaces!).

What can I do to help debug?

[privefl]

I don't understand your problem.
Please look at https://stackoverflow.com/q/5963269/6103040.
If your problem is not related to this issue, please open another one.

[GabeAl]

My problem is a crash on simple data.

I have opened an issue about it here: #22

Thanks for your help.