Running Minepy causes Python Jupyter Notebook to Crash

[hangyao]

Hi Team!

I tried to run a piece of Minepy code in Jupyter Notebook but it causes the notebook to crash.

from minepy import MINE

mine = MINE(alpha=0.6, c=15, est="mic_e")
scores = []
for col in X_train.T:
    mine.compute_score(col, y_train)
    scores.append(mine.mic())

The size of variable X_train is (66393, 79).

The system is:
Windows 7 x64
Python 3.6
Minepy 1.2.1

It seems that the data size of 66,393 causes the crash. I reduced the data rows to 20,000 and it worked fine, but 30,000 rows of data will crash the notebook. Is it caused by the limitation of physical memory or the algorithm? My system has 8G memory. Anyway I think some changes in the code may be required to resolve this issue, either asserting the memory or throwing an exception.

Thanks!

WERA24E.tmp.appcompat.txt
WER968A.tmp.WERInternalMetadata.xml.txt

[davidealbanese]

Dear @hangyao,
I don't think is a problem of the algorithm. Could you run the same code on the Python terminal and verify if the problem persists?

Thank you!

[davidealbanese]

Moreover, I suggest to use alpha=0.4 and c=5 or 1 for n>40.000 if you are using the MICe estimator (https://arxiv.org/pdf/1505.02214.pdf).

[hangyao]

@davidealbanese , thank you for the response! I tried to change the parameters and run on the Python terminal, and the crash problem persists. Down-sampling to around 21,020 - 21,030 rows of data can resolve the issue, but the exact number of rows leading to crash may change from time to time.

[davidealbanese]

Dear @hangyao,
what is the error message? Which compiler are you using for Python in Windows?

[hangyao]

It says An unhandled win32 exception occurred in python.exe [18936].

I'm not really sure about what compiler I'm using for Python in Windows. I think it is Microsoft Visual C++ 14.0 standalone: Build Tools for Visual Studio 2017 but I can be wrong. Could you please let me know how to find it out? Thank you!

[davidealbanese]

Dear @hangyao,
try to reinstall the Build Tools for Visual Studio 2017 from this page: https://www.visualstudio.com/downloads/. After that, reinstall the minepy package.
Let me know if it works...

[davidealbanese]

Dear @hangyao,
is there any news?

[hangyao]

@davidealbanese I reinstall the build tools for VS 2017, and reinstall minepy using pip. The issue is still there. Thanks!

[davidealbanese]

Probably related to #22.