Guide to running Rootpainter on my local machine?

[TheaKristensen]

Hi there,

Is there a guide on how to run Rootpainter on my local machine (given I have enough GPU)? I don't mind command line if that's necessary, but a guide/instructions would be highly appreciated.

Mostly because I keep getting kicked out of co-lab because of resource limitations.

Thea

[Abe404]

Yes, it is on the front page:
https://github.com/Abe404/root_painter#server-setup

It assumes you already have python and pip installed. You should use a recent version of python i.e python3.8 or higher.

Please let me know how it goes.

Note: at the end it links to the lung tutorial but I think you would be better off ignoring that bit now as downloading that lung dataset takes ages these days for some reason. I suggest just to test your local setup with the biopores you downloaded as part of the colab tutorial.

[Abe404]

RootPainter uses PyTorch internally (the most popular deep learning library for researchers). PyTorch is designed to work with NVIDIA GPUs. There are ways to make PyTorch work with other GPUs but my understanding is that it is still fairly non-standard so it would likely be a hassle to get working and you would likely experience unusual bugs.

I think colab pro is your easiest option for now.

How are things going with the model training? Were you able to train a model for the biopores example yet? Or for your own images?

Kind regards,
Abraham

[TheaKristensen]

Oh I didn't realise - thanks a lot, that saved future Thea a lot of time!

It's going great, thanks. I'm training on my own data and today the model reached 30/60 epochs before co-lab kicked me out. I have tested the segmentation from an older model which already seemed pretty good. I'm excited to see how the improved model performs!

Thea

[Abe404]

Great to hear it is going well! I'm looking forward to seeing the results.

It is not usually a problem if the model doesn't reach 60 epochs without progress, assuming the model has been trained for long enough. Fig 8 from the paper (https://nph.onlinelibrary.wiley.com/doi/full/10.1111/nph.18387) is an investigation of this. It is not a huge amount of data but in Fig 8 you can see in some cases models trained for less than 90 minutes benefited substantially from waiting until 60 epochs without progress but in general it didn't make much difference. The caveat here is that we used hardware that is quite a bit faster than colab. So perhaps with colab it is not 90 minutes of (interactive) training but 180 minutes. I'm not sure exactly but the general idea is that the longer you have interactively trained your model the less you should expect waiting for 60 epochs without progress to provide benefit and in general I don't think it's something to worry about.

What you see is what you get with the segmentations. By this I mean, the quality you should expect is what you see in the interface during corrective annotation. The recently added metrics plot (available from the extras menu) also gives a measure of segmentation accuracy that doesn't take into account any potential benefit of waiting for the 60 epochs without progress.

I put a note to explain this in more detail when I expand the tutorial (#62)

[TheaKristensen]

Thanks for the elaboration.

I'm not training on roots but on fungal isolates which grow over time, so I'm expecting it to take more effort than for the root tutorials, as the fungi I'm measuring vary in appearance.

When I have been training, I'm afraid I have annotated ahead of time, while I am not sure the model training was keeping up. Not sure if it makes sense to you.

[Abe404]

Hi @TheaKristensen that does make sense. I assume training is going a little slowly due to colab being a little slow. But hopefully it will get there. Did you have any luck with your model training process? Were you able to arrive at a model suitable for your needs?

I have updated the documentation a little more today to try to cover some of the subsequent steps not included in the colab notebook.

To do this I have created a shorter mini guide including more concise instruction, that could be used as reference. I suggest the paper, videos and then colab tutorial to get an idea of how the software interface could be used and then this mini guide for reference to help remember each of the key steps to get from raw data to final measurements.

I'm aware there is still a need for more in-depth instruction on the subsequent data-processing steps so will leave the issue (#62) open for now until I have more time available to make a more detailed tutorial covering every step in the mini-guide.

[eberrigan]

I made an environment file to install the proper dependencies. @Abe404 it would be helpful to have a current environment file available with the correct versions of dependencies. Thank you!

[Abe404]

There’s requirements files for the painter (client) and trainer (server) in each of their folders. See https://github.com/Abe404/root_painter/blob/master/trainer/requirements.txt for example.

[eberrigan]

Yep, I used those! I guess I just like to use conda for package maintenance. Thank you for your help.

[Abe404]

I’m currently using venv with pip https://docs.python.org/3/library/venv.html

which seems to me to be the most default and simplest choice as it’s built into python now.

It seems to be working out ok so I haven’t felt the need to switch to third party solutions such as conda.

I guess maintaining conda specific environment files would take extra work with each release/update. I’m concerned it would quickly get out of date if I don’t use conda myself, causing extra problems, so for now I think the requirements files should be enough.

thanks for your input on this anyway! It’s useful to know peoples preferences

[lamewarden]

Hi, question about running the RootPainter server on the local machine. Should it necessarily be a Linux server? Won't it work on Win?

[Abe404]

The server works on windows too. You’ll need a suitable NVIDIA card.

You can also use a Mac if it has an M1 or M2 chip (currently requires using the 86-installing-root_trainer-on-an-m1-max-apple-silicon branch, but hopefully will be in an official release soon).

Kind regards,
Abraham