FIX '[Error 5] Access is denied' under Windows

[ogrisel]

This should fix #354.

[lesteve]

Can you not use a else clause if the for loop for this?

[lesteve]

Should we try to get @krishnateja614 to test this? Maybe it is too much of a pain since it is happening on Azure ...

[krishnateja614]

Guys. I haven't had any problems after changing the timeout. If you want to test something out, I'll have access to the VM in the morning and can do so.

[lesteve]

Guys. I haven't had any problems after changing the timeout.

Patching the python stdlib is not really the nicest of solution though ;-).

If you could test this change and look whether you never get any warning, that would be great.

[ogrisel]

@krishnateja614 I would really appreciate it if you could undo the change you did to the standard library and instead test if this change in joblib can fix the crash you observe on your azure configuration.

[krishnateja614]

I'll test this once I get access to the VM. So basically,
reset the timeout to 0.1 in multiprocessing/forking.py
install joblib from the github repo, then run the code right?

[krishnateja614]

@ogrisel , I got the same error after trying with the updated joblib code. Also the joblib import error occurs only after this multiprocessing\forking.py terminateprocess function fails. This update on Joblib doesn't change that right or is there anything else I should be doing?. Again, the access denied error is not occurring for all the values of CV.

[lesteve]

The simplest is if you go there: https://raw.githubusercontent.com/ogrisel/joblib/71cc0d96967f5090894d7ae3347e04d0d60281c0/joblib/pool.py save it as a file (Control-S on most browsers) and then copy this file into C:\Anaconda2\lib\site-packages\sklearn\externals\joblib\pool.py.

[krishnateja614]

Got it.Trying it out. I tried with varying number of cv folds from 3-10,didn't run into errors at all for any of them. Got the warning "Failed to clean temporary folder" 2 out of 12 runs. Great job @ogrisel. Thank you for your help @lesteve and @ogrisel .

[ogrisel]

Interesting that you got the warning: it means that you still have some processes that are slow to die in the background. But at least it does not seem to hurt your program anymore. Will merge.

Thanks for testing the fix in your runtime environment!

[krishnateja614]

I don't know whether I should reopen this or not, but I have again ran into the same error as #354, the only difference is the dimensionality of the dataset. The error previously looked corrected when run on dataset of size 8,00,000 * 35 (I don't remember the number of columns I had exactly, but it's definitely less than 35) but now, I ran into the same error two days ago when running the model on 8,00,000 * 210 dataset. VM is same as before i.e Windows server 12 with 16 cores CPU & 116 GB RAM (AZURE D14_V2).