MemoryError

[guhasayantan]

Hello:

I am using the unsupervised feature selection using Laplacian Score. But I am facing the below error message

Traceback (most recent call last):
File "fs.py", line 12, in
W = construct_W.construct_W(frame, *_kwargs_W)
File "/usr/local/lib/python2.7/dist-packages/skfeature/utility/construct_W.py", line 141, in construct_W
D = pairwise_distances(X)
File "/usr/local/lib/python2.7/dist-packages/sklearn/metrics/pairwise.py", line 1207, in pairwise_distances
return _parallel_pairwise(X, Y, func, n_jobs, *_kwds)
File "/usr/local/lib/python2.7/dist-packages/sklearn/metrics/pairwise.py", line 1054, in _parallel_pairwise
return func(X, Y, **kwds)
File "/usr/local/lib/python2.7/dist-packages/sklearn/metrics/pairwise.py", line 231, in euclidean_distances
distances = safe_sparse_dot(X, Y.T, dense_output=True)
File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/extmath.py", line 184, in safe_sparse_dot
return fast_dot(a, b)
MemoryError

I have updated the scikit-learn; but the issue persists. Any inputs will be helpful

Regards
Sayantan Guha

[jundongl]

It happens because your matrix is too large that cannot be fit into the memory.

[tiagofrepereira2012]

Hi, I'm facing the exactly the same issue.

But the issue is weird, because the same code with the same data crashes in my Ubuntu 16.04 LTS with 64GB of RAM, but it runs in my MacOSX with 16GB of RAM.

I have everything set using conda.
The only difference is the numpy version. In the macosx I have numpy 1.11 and in the ubuntu 16.04 I have numpy 1.12.

  File "./bin/plot_tsne_one_file.py", line 19, in <module>
    sys.exit(context_experiments.script.plot_tsne_one_file.main())
  File "/home/tiago-ttt/Documents/redmine/context_experiments/context_experiments/script/plot_tsne_one_file.py", line 126, in main
    fig = plot_per_attribute(file_name, seed, perplexity, learning_rate, 200, device, "id")
  File "/home/tiago-ttt/Documents/redmine/context_experiments/context_experiments/script/plot_tsne_one_file.py", line 83, in plot_per_attribute
    projected_data = model.fit_transform(data)
  File "/home/tiago-ttt/miniconda3/envs/context_py27/lib/python2.7/site-packages/sklearn/manifold/t_sne.py", line 884, in fit_transform
    embedding = self._fit(X)
  File "/home/tiago-ttt/miniconda3/envs/context_py27/lib/python2.7/site-packages/sklearn/manifold/t_sne.py", line 730, in _fit
    squared=True)
  File "/home/tiago-ttt/miniconda3/envs/context_py27/lib/python2.7/site-packages/sklearn/metrics/pairwise.py", line 1240, in pairwise_distances
    return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
  File "/home/tiago-ttt/miniconda3/envs/context_py27/lib/python2.7/site-packages/sklearn/metrics/pairwise.py", line 1083, in _parallel_pairwise
    return func(X, Y, **kwds)
  File "/home/tiago-ttt/miniconda3/envs/context_py27/lib/python2.7/site-packages/sklearn/metrics/pairwise.py", line 245, in euclidean_distances
    distances = safe_sparse_dot(X, Y.T, dense_output=True)
  File "/home/tiago-ttt/miniconda3/envs/context_py27/lib/python2.7/site-packages/sklearn/utils/extmath.py", line 189, in safe_sparse_dot
    return fast_dot(a, b)
MemoryError