Issue getting Cellpose to use GPU

[nolsman]

Hi, I'm trying to test out Cellpose on data from a very large data set of bacterial images, and the CPU version seems to be struggling to process our images. I followed the instructions on your website regarding installing the GPU version of mxnet, but in the GUI the GPU setting remains grayed out.

When I try to run model = models.Cellpose(gpu=True, model_type='cyto'), I get the readout Using CPU.

My CUDA version is:
nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Fri_Feb__8_19:08:26_Pacific_Standard_Time_2019 Cuda compilation tools, release 10.1, V10.1.105

And my mxnet version is mxnet-cu101mkl version 1.5.0.

[lacan]

Hi!
I've had some issues with this. I've documented what worked for me here.
https://c4science.ch/w/bioimaging_and_optics_platform_biop/computers-servers/software/gpu-deep-learning/cellpose/
https://c4science.ch/w/bioimaging_and_optics_platform_biop/computers-servers/software/gpu-deep-learning/virtualenv/

The step that worked for me which was not clear was
You need to install mxnet-cu101
after you've uninstalled mxnet-mkl

Please let me know if any of this solved your issue.

EDIT: Changed the link to reflect our new documentation.

[carsen-stringer]

Thanks for the help @lacan, that's right, you have to uninstall mxnet-mkl first. You don't have to also install mxnet-cu101mkl unless you're going to be using the CPU version sometimes.

[carsen-stringer]

(mxnet-cu101 has a "CPU" version but it doesn't rely on mkl so it's slow)

[nolsman]

Thanks for the comments, I had tried both the instructions on the original site (including uninstalling mxnet-mkl) and the instructions @lacan proposed, each in a separate conda environment. I still am getting model = models.Cellpose(gpu=True, model_type='cyto') returning Using CPU.

I believe I followed all of the CUDA setup instructions, @carsen-stringer do you have an advice on tracing what might be causing this issue? I'm not sure if this is likely an issue, but I am trying to run this locally on my laptop which has both an integrated Intel graphics card (listed as GPU 0) and an NVIDIA GeForce GTX 1060 (listed as GPU 1). Is it possible that Cellpose is querying the first GPU and deciding it can't run CUDA?

Sorry if I am missing some obvious debugging steps, I have only really used CUDA/GPU computing through pre-packaged libraries like PyTorch and TensorFlow.

[glossdrop]

Hi nolsman,
I and a a colleague of mine were having the same problem (OS windows 10)
It was fixed by installing CUDA v9.0.
No idea why!

[nolsman]

Hmm, I'll try that next I guess. I just realized that I had a version of Cellpose installed in my base environment, I uninstalled it and updated conda, made sure to uninstall mxnet-mkl after the update, and now I'm getting the follow error when I tried to run model = models.Cellpose(gpu=True, model_type='cyto')

ImportError                               Traceback (most recent call last)
<ipython-input-5-197fb8558d18> in <module>
      1 # RUN CELLPOSE
      2 
----> 3 from cellpose import models
      4 
      5 # DEFINE CELLPOSE MODEL

~\Anaconda3\envs\cellpose\lib\site-packages\cellpose\models.py in <module>
      8 import cv2
      9 
---> 10 from mxnet import gluon, nd
     11 import mxnet as mx
     12 

ImportError: cannot import name 'gluon' from 'mxnet' (unknown location)

[glossdrop]

It is precisely the same error message I was getting with versions >= 10.0.

[nolsman]

@glossdrop When you installed 9.0, did you uninstall the previous version? I found some posts saying it isn't necessary, but when I run nvcc -V I still get 10.1 and I just tried pip installing mxnet-cu90, and now get new errors.

If I do need to uninstall 10.1, do I just uninstall through the windows uninstaller, or do I also need to delete environment variables/do anything else to tell nvcc to point to 9.0?

[carsen-stringer]

weird, I've also gotten that error when I didn't fully uninstall things.

to specify a certain gpu you can use the "device" input

import mxnet as mx

model.eval(data, device=mx.gpu(1))

[glossdrop]

Yes, I did uninstall CUDA 10 before installing a previous version. I would recommend you do it too.
Make sure your GPU driver is up to date.

In addition, in respect to what @carsen-stringer suggests, my computer also has an integrated and a dedicated gpu. Although when checking on the computer they respectively appear as device 0 and device 1, I can only use CUDA by setting device=mx.gpu(0).
Again I am not sure why, sorry!

[nolsman]

I tried uninstalling CUDA10 and now just have 9, getting new errors just from running from cellpose import utils. It seems like the error is with even importing mxnet. What is strange is if I run Python from the command line in the conda prompt, I can run import mxnet as nx with no errors. Tomorrow I guess I can just try to carefully remove everything related to mxnet/cellpose from the past few days and try to get a clean install to see if I broke some intermediate step, but after that I'm kind of out of ideas. The data sets we work with are on the order of terabytes, so I don't think I can use Cellpose without a GPU.

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-ff9fa486885a> in <module>
      7 get_ipython().run_line_magic('matplotlib', 'inline')
      8 # mpl.rcParams['figure.dpi'] = 300
----> 9 from cellpose import utils

~\Anaconda3\envs\cellpose\lib\site-packages\cellpose\utils.py in <module>
      7 import skimage.io
      8 import numpy as np
----> 9 import mxnet as mx
     10 
     11 

~\Anaconda3\envs\cellpose\lib\site-packages\mxnet\__init__.py in <module>
     22 from __future__ import absolute_import
     23 
---> 24 from .context import Context, current_context, cpu, gpu, cpu_pinned
     25 from . import engine
     26 from .base import MXNetError

~\Anaconda3\envs\cellpose\lib\site-packages\mxnet\context.py in <module>
     22 import warnings
     23 import ctypes
---> 24 from .base import classproperty, with_metaclass, _MXClassPropertyMetaClass
     25 from .base import _LIB
     26 from .base import check_call

~\Anaconda3\envs\cellpose\lib\site-packages\mxnet\base.py in <module>
    211 __version__ = libinfo.__version__
    212 # library instance of mxnet
--> 213 _LIB = _load_lib()
    214 
    215 # type definitions

~\Anaconda3\envs\cellpose\lib\site-packages\mxnet\base.py in _load_lib()
    202     """Load library by searching possible path."""
    203     lib_path = libinfo.find_lib_path()
--> 204     lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_LOCAL)
    205     # DMatrix functions
    206     lib.MXGetLastError.restype = ctypes.c_char_p

~\Anaconda3\envs\cellpose\lib\ctypes\__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
    354 
    355         if handle is None:
--> 356             self._handle = _dlopen(self._name, mode)
    357         else:
    358             self._handle = handle

OSError: [WinError 126] The specified module could not be found

[watsaig]

Hmm, I'll try that next I guess. I just realized that I had a version of Cellpose installed in my base environment, I uninstalled it and updated conda, made sure to uninstall mxnet-mkl after the update, and now I'm getting the follow error when I tried to run model = models.Cellpose(gpu=True, model_type='cyto')

ImportError                               Traceback (most recent call last)
<ipython-input-5-197fb8558d18> in <module>
      1 # RUN CELLPOSE
      2 
----> 3 from cellpose import models
      4 
      5 # DEFINE CELLPOSE MODEL

~\Anaconda3\envs\cellpose\lib\site-packages\cellpose\models.py in <module>
      8 import cv2
      9 
---> 10 from mxnet import gluon, nd
     11 import mxnet as mx
     12 

ImportError: cannot import name 'gluon' from 'mxnet' (unknown location)

I had this same error when setting up a Google Colab notebook (where CUDA 10.1 was installed) after uninstalling mxnet-mkl and installing mxnet-cu101, whereas everything worked fine on my PC (with CUDA 10.2 and mxnet-cu102, though I don't know if it's related to that).

Uninstalling mxnet-mkl and installing mxnet-cu101mkl did the trick it seems, the following code runs fine:

!pip install git+https://github.com/MouseLand/cellpose
!pip uninstall -y mxnet-mkl
!pip install mxnet-cu101mkl

import numpy as np
import time, os, sys
from urllib.parse import urlparse
import skimage.io
import matplotlib.pyplot as plt
import matplotlib as mpl
from cellpose import utils, models, plot

[carsen-stringer]

that's weird, those commands worked for me on google colab. For those of you who are happy running remotely, here is an example colab notebook with GPU enabled:

[nolsman]

Yeah I'm really confused, I tried installing everything in our compute cluster and the GPU works fine (but can't use GUI anymore). It isn't the end of the world because we can't practically do that much hand correction anyway. Apparently in the near future there will be CUDA support for Windows Subsystem for Linux, so I may try running locally again once that is out. Thanks for all the help!

[Jiburii]

Hi all. I am having the same issue as the above, and couldn't seem to solve it (ImportError: cannot import name 'gluon' from 'mxnet') when I attempt to import in the Jupyter Notebook.

In the cmd line (anaconda prompt), cellpose started running, but on the CPU, not the GPU.

As I was using CUDA 9.0, i was trying to install mxnet-cu90 and mxnet-cu90mkl and encountered the problems above.

As I need CUDA 9.0 for another application, I decided to try to get around this by installing CUDA 10.2, and switching to using that with mxnet-cu102 in my anaconda cellpose environment. I was attempting to do this by following the following guide:

https://stackoverflow.com/questions/40517083/multiple-cuda-versions-on-machine-nvcc-v-confusion

But when I checked my CUDA version in that environment, (nvcc -V) it still read 9.0. I think that could be because the above is a linux guide, and i am using windows 10.

Does anyone therefore have a windows guide for pointing the path to a particular CUDA version in the cellpose environment?

Alternatively, did anyone fix the original problem?

[Jiburii]

Update: Semi-solved.

I manually deleted mxnet and mxnet-mkl components in the lib/sitepackages section of the cellpose environment on my machine. I did this because pip uninstall could not find mxnet.

Then I used this: pip install https://repo.mxnet.io/dist/python/cu102/mxnet_cu102-1.6.0-py2.py3-none-win_amd64.whl to download and install mxnet-cu102 (I have CUDA 10.2). I did this because pip install mxnet-cu102 could not find mxnet.

Now, as you can see, the GPU is running. However, it is only running at 7 % capacity - does anyone know whether it is normal?

Thanks very much.

[lacan]

Hi @Jiburii

First off congrats on getting it to work!

Concerning your question, I have also experienced this low GPU usage on a GTX 1080. Thing is, these methods dont't use giant deep nets to work their magic. Usually what's useful is having RAM. So I would not worry about this.

@carsen-stringer thank you for the new GPU version of your notebook!

[tischi]

@carsen-stringer
We are setting up the environment for GPU like this:

```yaml
name: cellpose-gpu
dependencies:
  - python=3.7
  - pip
  - numpy
  - scikit-image
  - numba>=0.43.1
  - pyqt
  - scipy
  - matplotlib
  - jupyter
  - pip:
    # this needs to be adapted to your cuda version:
    # for 10.0 choose 100 etc.
    - mxnet-cu102
    - opencv_python
    - pyqtgraph==0.11.0rc0
    - natsort
    - google-cloud-storage
    - tqdm
    - cellpose

But it seems it may not be working.

>>> from cellpose import utils
>>> use_GPU = utils.use_gpu()
CUDA version not installed/working, will use CPU version.
>>> print('GPU activated? %d'%use_GPU)
GPU activated? 0

From the discussion above it seems that maybe one has to modify the mxnet packages after pip install cellpose?

[lacan]

Just an FYI,
Seeing as the actual example notebook that @carsen-stringer provided uses CUDA 10.1, what is the rationale for @tischi and @Jiburii for using 10.2? Nothing really changes between the two versions and if you are using anaconda, it can even manage the CUDA versions for you. Perhaps looking into
https://anaconda.org/anaconda/cudatoolkit
https://anaconda.org/anaconda/cudnn

@tischi, perhaps the paths to your CUDA are not exported properly? Seeing as your anaconda .yml file has no mention of the packages I mentionned, the issue most likely lies outside of your environment.

Similarily, if you install it the way you are doing it, you either need to make sure that either cellpose is installed without the dependecies OR that you uninstall mxnet before installing mxnet-cu10x as a last step. This is documented here:
https://github.com/MouseLand/cellpose#cuda-version

[tischi]

@constantinpape
Do you know why we are using mxnet-cu102?

@lacan

perhaps the paths to your CUDA are not exported properly?

Could you provide me with an example of how to do this? I am totally new to all of this!

[constantinpape]

Do you know why we are using mxnet-cu102?

Because the EMBL drivers support cuda 10.2 ;). In my experience it works best to use the most up-to-date version.
But I don't have any experience with mxnet, only with pytorch and tf.

@lacan

perhaps the paths to your CUDA are not exported properly?

Could you provide me with an example of how to do this? I am totally new to all of this!

This is a good point. I set up the env file a while ago and I assumed that mxnet-cu would include the cuda packages already, but that's probably not the case.

@tischi I can try including the cudatoolkit, and see if this helps.

[tischi]

@constantinpape
Thanks! If you would modify our yaml file (maybe also making cellpose not import its dependencies??), I could try it out (and post here for reference once we get it working).

[atarkowska]

@constantinpape
Thanks! If you would modify our yaml file (maybe also making cellpose not import its dependencies??), I could try it out (and post here for reference once we get it working).

@tischi did you manage to get that working?

[constantinpape]

@olatarkowska
Yes, we got a version with one conda environment file and an additional requirements.txt for pip only dependencies.
See here for details.

[carsen-stringer]

thanks for the help @constantinpape !