Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

numpy.core._exceptions.MemoryError: Unable to allocate array with shape (24012, 10716) and data type float32 #19

Closed
Kai6662 opened this issue Nov 4, 2019 · 5 comments
Closed

numpy.core._exceptions.MemoryError: Unable to allocate array with shape (24012, 10716) and data type float32 #19

Kai6662 opened this issue Nov 4, 2019 · 5 comments

Comments

@Kai6662
Copy link

Kai6662 commented Nov 4, 2019

I am testing Scaden using the demo data. But when I run the training step, I got a error like this "Training on: ['data8k', 'donorA', 'donorC']
Training M256 Model ...
2019-11-04 17:59:27.410629: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-11-04 17:59:27.410714: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2019-11-04 17:59:27.410746: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2019-11-04 17:59:27.410766: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2019-11-04 17:59:27.410791: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Traceback (most recent call last):
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/bin/scaden", line 125, in
cli()
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/bin/scaden", line 63, in train
num_steps=steps)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/scaden/scaden_main.py", line 64, in training
cdn256.train(input_path=data_path, train_datasets=train_datasets)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/scaden/model/scaden.py", line 252, in train
self.build_model(input_path=input_path, train_datasets=train_datasets, mode="train")
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/scaden/model/scaden.py", line 213, in build_model
self.load_h5ad_file(input_path=input_path, batch_size=self.batch_size, datasets=train_datasets)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/scaden/model/scaden.py", line 158, in load_h5ad_file
raw_input = raw_input[raw_input.obs['ds'] != ds].copy()
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/anndata/core/anndata.py", line 1230, in getitem
return self._getitem_view(index)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/anndata/core/anndata.py", line 1234, in _getitem_view
return AnnData(self, oidx=oidx, vidx=vidx, asview=True)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/anndata/core/anndata.py", line 561, in init
self._init_as_view(X, oidx, vidx)
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/anndata/core/anndata.py", line 628, in _init_as_view
self._init_X_as_view()
File "/hpc/dhl_ec/kcui/tools/anaconda3/envs/myenv/lib/python3.6/site-packages/anndata/core/anndata.py", line 641, in _init_X_as_view
X = self._adata_ref._X[self._oidx, self._vidx]
numpy.core._exceptions.MemoryError: Unable to allocate array with shape (24012, 10716) and data type float32
"

I google the error and I am sure I have enough memory and I am using the cpu tensorflow. And I am not root ,I cannot revise like this "echo 1 > /proc/sys/vm/overcommit_memory". The problem is MemoryError really? The version of scaden is 0.9.0. If there is someone could help me? Thanks in advance.
@zeehio
Copy link
Collaborator

zeehio commented Nov 5, 2019

How much memory can you use?

In my RAM profiling, scaden was using up to 8GB of RAM during training. After some code review, we lowered the RAM usage in the training step, and now the RAM peak of the demo is at 3.2 GB (that happens on the preprocessing step). This RAM improvement is on the git repository (and not yet published to conda-forge as far as I know).

You may want to try the git master version. Here is a quick guide on how to build it based on the travis instructions:

# clone the repository, this will create the scaden directory:
git clone https://github.com/KevinMenden/scaden
# Enter the scaden directory:
cd scaden
# You may want to run this in a new conda environment instead
# Install conda-build and python-3.6
conda install -q -c defaults -c conda-forge python=3.6 conda-build conda-verify
# Build scaden
conda build --override-channels -c defaults -c conda-forge -c r -c bioconda --python 3.6 ./conda.recipe
# Install scaden in a new conda environment:
conda create \
      -n install-environment \
      --override-channels -c defaults -c conda-forge -c r -c local -c bioconda \
      python=3.6 \
      scaden
# Verify scaden is working:
conda activate install-environment
scaden --help

We will work on making it easier to install the development version of scaden. Meanwhile, this should work for you

@Kai6662
Copy link
Author

Kai6662 commented Nov 5, 2019 via email

@zeehio
Copy link
Collaborator

zeehio commented Nov 5, 2019

I don't use Mac so I can't say. You can try though.

@Kai6662
Copy link
Author

Kai6662 commented Nov 6, 2019 via email

@KevinMenden
Copy link
Owner

Closing this issue because this is rather a system problem and can't be solved with Scaden.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zeehio @KevinMenden @Kai6662