Error during Windows install and fine-tuning

[Eddymorphling]

Hi All,
I am trying to setup AxonDeepseg using Anaconda3 on a fresh env that runs on python 3.8. But, I come across this error during the install

File "/home/imn/conda/envs/ads/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command 'download_models' returned non-zero exit status 1.
    [end of output]

I followed the instructions that I found over here . Any advise on how I can fix this issue? Thanks!

[Eddymorphling]

Oh never mind. I was doing something silly. Its fixed now :)

Any tips on testing ADS is using the GPU when doing segmentations using the inbuilts? I did a quick test with python in the ads_venv and seems to detect the GPU.

>>> import torch
>>> torch.cuda.is_available()
True

But when I run axondeepseg_test, it seems to be using the CPU for predictions.

2024-02-16 10:38:52.803 | DEBUG    | ivadomed.inference:get_preds:74 - Evaluating model: /home/imn/axondeepseg/AxonDeepSeg/models/model_seg_rat_axon-myelin_sem/model_seg_rat_axon-myelin_sem.pt
2024-02-16 10:38:54.817 | DEBUG    | ivadomed.inference:get_preds:91 - Sending predictions to CPU

[Stoyan-I-A]

I think the Sending predictions to CPU message is always displayed. Even if the GPU is used. I guess you could open the task manager (Ctrl+Alt+Delete -> Task manager) and check if the total utilisation of your NVIDA GPU (in the Performance tab) goes up while you run a model.
Note that in the detailed view of the task manager, the percentage used by Cuda isn't shown by default.

That being said, I wonder if the default installation of AxonDeepSeg does not install the CPU-only version of PyTorch 🤔.
I remember a user did manage to make the GPU work on his Windows machine: #752

[Eddymorphling]

Thanks for the quick response! Let me check if I can monitor GPU usage. Also just an FYI, I have set this up to run on a Linux high-performance cluster. Let me see if this uses the GPU when running the inference.

Are the most up to date model downloaded with git clone https://github.com/neuropoly/axondeepseg.git

[Stoyan-I-A]

Are the most up to date model downloaded with git clone https://github.com/neuropoly/axondeepseg.git

Yes.
Also, the team is using GPU clusters to train the models, so perhaps some of them could help you if you have questions. Refer to the discussions page.
The package we use for both training and applying models is called Ivadomed.

[Eddymorphling]

Beautiful. Thank you very much. The base models work quiet well out of the box. I plan on improving model weights by retraining the TEM model with a few of my own labels/images.

[Eddymorphling]

One last question, are there any limits on file size that I can use for segmentation with the base models? My raw files are in its native resolution of 1.5nm/pixel. So I just update this in the CLI axondeepseg -t TEM -i $input_folder $input_folder2 -s 0.001745 --no-patch --overlap 100 during segmentation but end up with an error below

PIL.Image.DecompressionBombError: Image size (201326592 pixels) exceeds limit of 178956970 pixels, could be decompression bomb DOS attack.

Downsampling my images to 10nm/pixel has been the only way to resolve this. Any suggestions? Thank you.

[Stoyan-I-A]

are there any limits on file size that I can use for segmentation with the base models?

Kind of. The package used for loading the images has internal limits.

Perhaps if you remove the --no-patch option it might work but I'm not sure.

Another note: our software resamples the images/patches so that the resampled image has a resolution (distance per pixel) similar to the resolution of the model you are trying to use. So if your image has a resolution much higher than the model resolution, it likely won't lead to better results because the image will be resampled anyway.
The TEM model has a resolution of 0.01 micrometer per pixel.

[Eddymorphling]

Ah ok. Forgot that PIL has size limits. I did try without --no-patch but still came across size issues.

I did not know there was an internal resampling so I decided to resample my images to match the scaling that the model was trained on. I then included "resampled" pixel size in the CLI when running ADS.

[mathieuboudreau]

Beautiful. Thank you very much. The base models work quiet well out of the box. I plan on improving model weights by retraining the TEM model with a few of my own labels/images.

@Eddymorphling is training a new model with your dataset the goal of your project, or are the mainly interested in the segmentations/morphometrics for your research? If the latter, @hermancollin and @ArthurBoschet have a new deep learning model/method in development that's performing quite well with nearly any dataset/contrast you give it, if you're willing to share a sample image they could maybe try and see if they can get better results (and save you time).

[hermancollin]

Beautiful. Thank you very much. The base models work quiet well out of the box. I plan on improving model weights by retraining the TEM model with a few of my own labels/images.

@Eddymorphling is training a new model with your dataset the goal of your project, or are the mainly interested in the segmentations/morphometrics for your research? If the latter, @hermancollin and @ArthurBoschet have a new deep learning model/method in development that's performing quite well with nearly any dataset/contrast you give it, if you're willing to share a sample image they could maybe try and see if they can get better results (and save you time).

Indeed, I would advise against trying to fine-tune the TEM model yourself. The framework it was trained on is a little bit deprecated and maintenance is shaky. If you want us to try our new model, this could be a better option. Moreover, this new model will be publicly released very shortly, so if you really want to update the weights, I suggest waiting for this new version first, which would potentially give better results.

[Eddymorphling]

Beautiful. Thank you very much. The base models work quiet well out of the box. I plan on improving model weights by retraining the TEM model with a few of my own labels/images.

@Eddymorphling is training a new model with your dataset the goal of your project, or are the mainly interested in the segmentations/morphometrics for your research? If the latter, @hermancollin and @ArthurBoschet have a new deep learning model/method in development that's performing quite well with nearly any dataset/contrast you give it, if you're willing to share a sample image they could maybe try and see if they can get better results (and save you time).

Indeed, I would advise against trying to fine-tune the TEM model yourself. The framework it was trained on is a little bit deprecated and maintenance is shaky. If you want us to try our new model, this could be a better option. Moreover, this new model will be publicly released very shortly, so if you really want to update the weights, I suggest waiting for this new version first, which would potentially give better results.

Got it. Thank you for letting me know! I am happy to wait to the release of the new model. Will keep an eye out for it! Please do keep me updated if possible :)

[Eddymorphling]

this new model will be publicly released very shortly,

Out of curiosity, will this new model have training data that has "unmyelinated" or "remyelinated" axons? I am interested in segmenting these too and this was the rationale behind retraining the weight with some of my training data.

[hermancollin]

I can test the latest version of the unmyelinated axon segmentation model if you want. Maybe it already works well for your images? This model is still a WIP but it is usable.

[Eddymorphling]

Thanks @hermancollin. That's good to know. I would like to test this model. How do I setup and try it out? I am not sure if I am able to share datasets.

[Eddymorphling]

My bad. Found it

[hermancollin]

@Eddymorphling Yes! You should be able to use the model with this repo. Please let us know how it works I'm very curious.

The version you will find here https://github.com/axondeepseg/model_seg_unmyelinated_tem/releases/tag/v1.0.0 was a first iteration of the model. However, I only released 1/5 of the "full" model, so the performance will not be optimal (as I realized later on). I put it there and got busy with other stuff so I forgot about it, but I'll try to upload it next week if you want to test it.

[Eddymorphling]

Hi @hermancollin .I can try the new model if you have it ready to go. Is it already on the repo or would you need to upload it? I thought of setting this up and testing this today.