Kernel dies #121
Comments
|
Hi @malj390 , Hard to diagnose - does it happen always at cell 13? (which actually doesn't do anything on the GPU, so should not be a problem). |
|
Yes. I also try skipping that cell and go to the next one of Quick demo= True and there it also dies. |
|
Actually, the first time If you add this at the (very) beginning of the notebook, it shouldn't use the GPU and the kernel hopefully doesn't die: import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""Of course this isn't a solution, but your log messages don't help me to figure out where the problem is. |
|
Indeed, as Uwe mentioned it might be that already
directly after you created the model, and see whether this already makes the kernel die. |
|
nvidia-smi output: I tried both of your suggestions. Together and individually and still the kernel keeps dying. |
|
Ok. Could you try whether |
|
It doesn't work either. |
Is this really the entire output on the command line where you started the Jupyter notebook server? I would assume to see some more relevant messages to get a clue where the problem is. |
|
Hello, I do not know if this is related, but I have also encountered such issues on Mac. I try to install stardist, it first fails if I do not provide the proper CC, g++ flags. After I do, installation works fine but then when running prediction on 2D data the kernel dies. However if I run the same notebook on Colab where !pip install stardist appears in a cel before, everything works just fine. I could be h5py Mac related issue? Am not sure if this helps but maybe the issue is not 3D related but Mac/OS/h5py related? Maybe if you try to run your code on Colab and if that works then that would confirm the issue is not really a stardist bug but something else? |
|
@uschmidt83 No, I can only get that output through Spyder. Actually, Jupyter Notebook doesn't prompt any error, the kernel just dies. |
|
@kapoorlab I though to do so but I didn't think it was really going to give any extra answer. The code I'm running is directly from examples with their data, so if something goes wrong I know that for sure is my installation or my environment. The weird part is that I run the proper tests at the beginning to check if the installation is properly done (Tensorflow, CUDA, the graphic card is recognized, etc) and it says it is ok. I will follow your suggestion and I will run it in Colab. However at some point I need to run a training in my own computer. @uschmidt83 @maweigert @kapoorlab Thanks for the prompt reply |
|
So I can confirm the issue is with my environment as Colab works well. I'm not sure what it is and how to obtain more information to address and solve the problem. Do you have any extra advice @uschmidt83 to extract more relevant information? |
|
You can save/export the Jupyter notebook as a Python script and run it from the command line. Maybe that generates more helpful error messages. |
|
You can test whether basic inference works with the following
|
|
Just a related question, does stardist not work well with certain combinations of tensorflow/keras versions? Because I also had such a problem on my Mac and I switched to Colab with tensorflow1.x and keras 2.2.5 and it was all fine. |
Not to my knowledge.
I never use tensorflow (csbdeep, stardist, etc.) in Python on my Mac, because it doesn't have GPU support anyway. Maybe @maweigert can comment on this. |
|
I did pip install -U stardist and for me the kernel does not die now in the same virtual environment that it died before. So it is good from the Mac world side. Some users may want to have it on their Macs as they may not have access to a GPU server, just good to have it installed on slow computers as an option. |
Yes. Actually, the previous post where I mentioned this long output from below was thanks to the error raised by the notebook converted to .py and run in Spyder. I did not find a way to access the error from the Jupyter notebook. |
|
@maweigert It's still dying in jupyter notebook and in Spyder (giving the same error mentioned). I suspect is a problem with my system itself or the graphic card. As I'm not really into complex models I did this easy one and just to try tensorflow. It runs properly without dying, so now I'm not sure where the problem is. |
|
I found the error. Sorry guys, I didn't check the Anaconda prompt from Notebook which actually shows the error. It was this one: I did all the installation following this link (with updated compatible versions of CUDA, CudNN and CUDA Tool Kit ) so it seems in the tutorial that cudnn is not mentioned. So now, I copied the full folders from to Now the error in Notebook is: and in the Anaconda prompt: Thanks |
|
Ok, thanks for letting us know. Btw, I got TensorFlow with GPU support working (also on Windows) by simply installing this conda environment. It will automatically install the necessary CUDA and cuDNN libraries. After installing this environment, activate it, and then install StarDist via pip. |
|
Thanks a lot! I did it but still I get this error. Anaconda prompt: |
|
This looks like something else is occupying the GPU memory (another notebook?). |
|
No I just opened the one notebook. This is the complete error in the case brings up more information. I don't know if the warning log message that is referring to be printed above is this one: |
|
I really don't know what the problem is. One last try, run this at the very beginning of the notebook: from csbdeep.utils.tf import limit_gpu_memory
limit_gpu_memory(None, allow_growth=True) |
|
Yes it fixed! Thanks a lot |
Hello,
For some reason just running the notebook examples provided for 3D segmentation the kernel restart on the Training notebook for the cell 13th with
I'm running it on Windows 10, python=3.8.5, tensorflow=2.4.1 and gputools=0.2.9
I have done all the routinary checkings to check the GPU working and being recognized by TensorFlow.
I also tried in Spyder
and adding these lines to the notebook at the beginning but still.
Does someone know what could be the problem?
Thank you
The text was updated successfully, but these errors were encountered: