Maybe indicate how to install transformers dependency?

[FlorisCalkoen]

ping #openjournals/joss-reviews#6683

I'm particularly interested in CoastSeg' coastal DL model, so I'm currently installing everything on my PC. So far I have managed to install Tensorflow, but I see in your docs that I also need transformers. Since the installation requirements for all ML libs you use so far where more specific than the regular python libraries, I checked out the transformers docs. However, there I see that there are a whole lot of different ways of installing transformers, also depending on used TF version. Can you maybe provide some guidelines on which one to pick?

I'm a MacOS user, with this CoastSeg and Tensorflow version:

...
coastseg                  1.2.4              pyhd8ed1ab_0    conda-forge
tensorflow                2.16.1                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.37.0                   pypi_0    pypi

https://huggingface.co/docs/transformers/en/installation

[2320sharon]

Hi @FlorisCalkoen,

Thank you for your interest in the CoastSeg's coastal DL model and for detailing your setup. Regarding the installation of the transformers library, the general recommendation for most users is to use:

pip install transformers

This method has proved effective in most cases. Our team has explored a few alternative methods such as pip install transformers[tf-cpu], but this specific installation option has not been successful in our tests and might not be compatible with the TensorFlow version you are using.

We understand the importance of supporting various operating systems and installation methods. However, due to the complexity involved in maintaining compatibility between transformers and TensorFlow across different platforms, our resources are currently limited. Specifically, our team does not have access to Mac computers, which restricts our ability to test and provide tailored support for MacOS.

[2320sharon]

I want to provide some additional context as to why CoastSeg isn't using other versions of tensorflow or transformers.

Coastseg deliberately utilizes only the CPU versions of machine learning libraries like Tensorflow because CoastSeg doesn't rely on the GPU for model inference. We have attempted on multiple occasions to integrate GPU-accelerated workflows into CoastSeg. However, the complexity of setting up CUDA drivers and the inconsistent support for TensorFlow across various platforms have presented significant challenges. After careful consideration, we decided to prioritize a CPU-only approach to avoid these complexities and ensure that CoastSeg remains as compatible and user-friendly as possible across all supported platforms.

[dbuscombe-usgs]

Good points being discussed here, thanks.

@2320sharon I'm curious as to why transformers isn't listed in https://github.com/SatelliteShorelines/CoastSeg/blob/main/requirements.txt?

Floris, as you will have seen, our website Zoo page is still under development. I recommend the following environment if you planned to contribute new DL models to this project. Thanks for the offer! That would be great. We don't have the bandwidth to train new models at the moment.

Assuming you are on Linux (recommended for any DL model training workflow - dont mess around with trying to get things to work on Mac or Windows, you'll be there all week)

set up miniconda:

sudo apt-get update
sudo apt-get install wget
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh
bash Miniconda3-py39_4.12.0-Linux-x86_64.sh
bash

then set up mamba:

conda update -n base conda
conda clean --all -y
python -m pip install --upgrade pip 

conda install -n base conda-libmamba-solver
conda config --set solver libmamba

then configure your GPU dependencies

conda create -n gym_gpu python=3.10 -y
conda activate gym_gpu
conda install -c conda-forge cudatoolkit=11.8.0 -y
conda install -c nvidia cuda-nvcc -y

python3 -m pip install nvidia-cudnn-cu11 tensorflow[and-cuda]

then finally the rest of the dependencies

conda install -c conda-forge scikit-image ipython tqdm pandas natsort matplotlib -y
python -m pip install doodleverse_utils chardet

python -m pip install transformers

If you are contributing a new model, you'll be using Segmentation Gym, and these install instructions and more are detailed on the readme there