CoastSeg conda env is very slow. Abandon yml?

[dbuscombe-usgs]

The current approach to install the coastseg conda env, which is fairly complex as it contains large dependencies such as tensorflow, gdal, panel, ipywidgets, etc, (conda env create --file install/coastseg.yml) is quite slow

It may possibly be faster if we first installed mamba, then used mamba to find a solution and install the dependencies

[dbuscombe-usgs]

On a fresh Ubuntu 22.04 install, using the latest anaconda distro and with an updated conda, I killed the process of solving the CoastSeg conda environment for 14 hours ... this behaviour has also been reported previously by CoastSat users

I propose we test (and hopefully adopt) this conda recipe that works for me on Ubuntu 22.04, and Windows 10 (usgs laptop). It would be nice to know this works on other platforms

### create coastseg env
conda create -n coastseg python=3.8
conda activate coastseg

## coastsat dependencies
conda install -c conda-forge earthengine-api astropy
conda install gdal geopandas scikit-image notebook

## additional coastseg dependencies
conda install ipython cartopy  tqdm pip
conda install -c conda-forge simplekml leafmap
pip install area ipyfilechooser doodleverse_utils

then, depending on whether or not you have a CUDA-enabled GPU, either

pip install tensorflow-gpu

or (CPU-ony)

pip install tensorflow

[dbuscombe-usgs]

In the above, I'm not sure if python==3.8 is strictly necessary

[venuswku]

The recipe mostly works for me on my Windows 11 laptop. It took about 15 minutes for the environment to install, but I'm getting the following error when I run the first cell of custom_map.ipynb:

Version:  2.9.1
Eager mode:  True
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [1], in <cell line: 18>()
     16 import numpy as np
     17 # Local Imports 
---> 18 from CoastSeg import download_roi, file_functions, zoo_model_module, coastseg_map_class
     19 from CoastSeg.CoastSat.coastsat import SDS_shoreline,SDS_download
     20 # suppress tensorflow warnings

File ~\Projects\CoastSeg\CoastSeg\zoo_model_module.py:7, in <module>
      5 import json
      6 from tqdm import tqdm
----> 7 from doodleverse_utils.prediction_imports import do_seg
      8 from doodleverse_utils.imports import simple_resunet, simple_unet, simple_satunet, custom_resunet, custom_unet, mean_iou, dice_coef
      9 import tensorflow as tf

File ~\anaconda3\envs\coastsegenv\lib\site-packages\doodleverse_utils\prediction_imports.py:57, in <module>
     54 import tensorflow.keras.backend as K
     56 #crf
---> 57 import pydensecrf.densecrf as dcrf
     58 from pydensecrf.utils import create_pairwise_bilateral, unary_from_softmax
     59 # unary_from_labels, 

ModuleNotFoundError: No module named 'pydensecrf'

The command conda install -c conda-forge simplekml leafmap is just missing the pydensecrf package. After changing it to conda install -c conda-forge simplekml leafmap pydensecrf, I encountered this error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 18>()
     16 import numpy as np
     17 # Local Imports 
---> 18 from CoastSeg import download_roi, file_functions, zoo_model_module, coastseg_map_class
     19 from CoastSeg.CoastSat.coastsat import SDS_shoreline,SDS_download
     20 # suppress tensorflow warnings

File ~\Projects\CoastSeg\CoastSeg\coastseg_map_class.py:22, in <module>
     20 from pyproj import Proj, transform
     21 import matplotlib
---> 22 matplotlib.use("Qt5Agg")
     24 # Internal dependencies imports
     25 from .exceptions import DownloadError

File ~\anaconda3\envs\coastsegenv\lib\site-packages\matplotlib\__init__.py:1144, in use(backend, force)
   1139 if plt is not None:
   1140     try:
   1141         # we need this import check here to re-raise if the
   1142         # user does not have the libraries to support their
   1143         # chosen backend installed.
-> 1144         plt.switch_backend(name)
   1145     except ImportError:
   1146         if force:

File ~\anaconda3\envs\coastsegenv\lib\site-packages\matplotlib\pyplot.py:288, in switch_backend(newbackend)
    281 # Backends are implemented as modules, but "inherit" default method
    282 # implementations from backend_bases._Backend.  This is achieved by
    283 # creating a "class" that inherits from backend_bases._Backend and whose
    284 # body is filled with the module's globals.
    286 backend_name = cbook._backend_module_name(newbackend)
--> 288 class backend_mod(matplotlib.backend_bases._Backend):
    289     locals().update(vars(importlib.import_module(backend_name)))
    291 required_framework = _get_required_interactive_framework(backend_mod)

File ~\anaconda3\envs\coastsegenv\lib\site-packages\matplotlib\pyplot.py:289, in switch_backend.<locals>.backend_mod()
    288 class backend_mod(matplotlib.backend_bases._Backend):
--> 289     locals().update(vars(importlib.import_module(backend_name)))

File ~\anaconda3\envs\coastsegenv\lib\importlib\__init__.py:127, in import_module(name, package)
    125             break
    126         level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)

File ~\anaconda3\envs\coastsegenv\lib\site-packages\matplotlib\backends\backend_qt5agg.py:7, in <module>
      4 from .. import backends
      6 backends._QT_FORCE_QT5_BINDING = True
----> 7 from .backend_qtagg import (    # noqa: F401, E402 # pylint: disable=W0611
      8     _BackendQTAgg, FigureCanvasQTAgg, FigureManagerQT, NavigationToolbar2QT,
      9     backend_version,  FigureCanvasAgg,  FigureCanvasQT
     10 )
     13 @_BackendQTAgg.export
     14 class _BackendQT5Agg(_BackendQTAgg):
     15     pass

File ~\anaconda3\envs\coastsegenv\lib\site-packages\matplotlib\backends\backend_qtagg.py:9, in <module>
      5 import ctypes
      7 from matplotlib.transforms import Bbox
----> 9 from .qt_compat import QT_API, _enum, _setDevicePixelRatio
     10 from .. import cbook
     11 from .backend_agg import FigureCanvasAgg

File ~\anaconda3\envs\coastsegenv\lib\site-packages\matplotlib\backends\qt_compat.py:142, in <module>
    140         break
    141     else:
--> 142         raise ImportError("Failed to import any qt binding")
    143 else:  # We should not get there.
    144     raise AssertionError(f"Unexpected QT_API: {QT_API}")

ImportError: Failed to import any qt binding

I'm not sure how to solve this error though.

[dbuscombe-usgs]

thanks for testing --- pydensecrf of course!

The qt (graphics backend) error I think can be solved using conda install pyqt. Updated recipe below

### create coastseg env
conda create -n coastseg python=3.8
conda activate coastseg

## coastsat dependencies
conda install -c conda-forge earthengine-api astropy
conda install gdal geopandas scikit-image notebook

## additional coastseg dependencies
conda install ipython cartopy tqdm pip pyqt
conda install -c conda-forge simplekml leafmap pydensecrf
pip install area ipyfilechooser doodleverse_utils

Then, depending on whether or not you have a CUDA-enabled GPU, either

pip install tensorflow-gpu

or (CPU-ony)

pip install tensorflow

[venuswku]

Works now without any errors! Do you know how to get rid of the numpy warning (see above screenshots)? It says I have version 1.23.1 when it requires >=1.16.5 and <1.23.0. Also, can you include the commands for installing tensorflow in your updated recipe? I accidentally skipped that part at first. 😅

[dbuscombe-usgs]

Great! 🎉

I updated the comment above to include the tensorflow installation. It is my understanding that Tensorflow recommend installation via pip instead of conda-forge

As for numpy, I don't know if that will present a problem. I just attempted to install the latest compatible version (1.23.0) from conda-forge using conda install -c conda-forge numpy==1.23.0

Collecting package metadata (current_repodata.json): / 
done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/marda/anaconda3/envs/coastseg

  added / updated specs:
    - numpy==1.23.0


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    numpy-1.23.0               |   py38h3a7f9d9_0         7.1 MB  conda-forge
    scipy-1.9.1                |   py38hea3f02b_0        26.2 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        33.3 MB

The following packages will be UPDATED:

  certifi            pkgs/main/linux-64::certifi-2022.6.15~ --> conda-forge/noarch::certifi-2022.6.15-pyhd8ed1ab_1
  numpy                               1.22.3-py38h99721a1_2 --> 1.23.0-py38h3a7f9d9_0
  scipy               pkgs/main::scipy-1.7.3-py38hf838250_2 --> conda-forge::scipy-1.9.1-py38hea3f02b_0

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates    pkgs/main::ca-certificates-2022.07.19~ --> conda-forge::ca-certificates-2022.6.15-ha878542_0
  openssl              pkgs/main::openssl-1.1.1q-h7f8727e_0 --> conda-forge::openssl-1.1.1q-h166bdaf_0


Proceed ([y]/n)? 

Downloading and Extracting Packages
numpy-1.23.0         | 7.1 MB    | ####################################################################################### | 100% 
scipy-1.9.1          | 26.2 MB   | ####################################################################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Retrieving notices: ...working... done

I have not yet tested

[venuswku]

Sorry for the late response! I was trying all types of different ways to install a compatible version of numpy because it got stuck at solving the environment for a long time when I run the command conda install -c conda-forge numpy=1.23.0.

(coastsegenv) C:\Users\Venuxk\Projects\CoastSeg>conda install -c conda-forge numpy=1.23.0
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: |

I had to run conda install -c conda-forge “numpy>=1.16.5, <=1.23.0” in order to get rid of the warning. Looks like numpy-1.22.3 is compatible with scipy-1.7.1 for me. Not sure why you have scipy-1.9.1 while I have scipy-1.7.1 because we used the same set of commands.

(coastsegenv) C:\Users\Venuxk\Projects\CoastSeg>conda install -c conda-forge "numpy>=1.16.5, <=1.23.0"
Collecting package metadata (current_repodata.json): done
Solving environment: -
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - conda-forge/noarch::earthengine-api==0.1.321=pyhd8ed1ab_0
  - conda-forge/win-64::google-api-core==1.22.2=py38h32f6830_0
  - conda-forge/noarch::google-api-python-client==1.12.8=pyhd3deb0d_0
  - defaults/noarch::google-cloud-core==1.7.1=pyhd3eb1b0_0
  - conda-forge/noarch::google-cloud-storage==1.19.0=py_0
  - conda-forge/win-64::googleapis-common-protos==1.56.4=py38h5b57dd5_0
done

## Package Plan ##

  environment location: C:\Users\Venuxk\anaconda3\envs\coastsegenv

  added / updated specs:
    - numpy[version='>=1.16.5,<=1.23.0']


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    numpy-1.22.3               |   py38h1d2777f_2         6.1 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         6.1 MB

The following NEW packages will be INSTALLED:

  libblas            conda-forge/win-64::libblas-3.9.0-1_h8933c1f_netlib
  libcblas           conda-forge/win-64::libcblas-3.9.0-5_hd5c7e75_netlib
  liblapack          conda-forge/win-64::liblapack-3.9.0-5_hd5c7e75_netlib
  protobuf           conda-forge/win-64::protobuf-3.20.1-py38haa244fe_0

The following packages will be REMOVED:

  numpy-base-1.23.1-py38hca35cd5_0

The following packages will be SUPERSEDED by a higher-priority channel:

  numpy              pkgs/main::numpy-1.23.1-py38h7a0a035_0 --> conda-forge::numpy-1.22.3-py38h1d2777f_2


Proceed ([y]/n)? y


Downloading and Extracting Packages
numpy-1.22.3         | 6.1 MB    | ############################################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Retrieving notices: ...working... done

custom_map.ipynb runs fine with no warnings or errors now!
Here's a list of my installed packages and their versions: packages.txt

[dbuscombe-usgs]

Good input @venuswku

@2320sharon you could test this too when you have time, and if it works for all of us, let's get rid of the install\yml file (which I installed because I didn't read the new docs)

[2320sharon]

I'm working on trying out the new versions of the environment you guys have posted here.
@dbuscombe-usgs Just wondering if pydensecrf is compatible with python version 3.10? I believe it is but I want to double-check.

[2320sharon]

So I've been messing with the environment you both worked on as well as the environments I have had in development. I removed some of the old dependencies as well as installed dependencies that were missing from your guys' installation instructions. I found that the following instructions worked well.

conda create -n coastseg python=3.8
conda activate coastseg

## coastsat dependencies
conda install -c conda-forge earthengine-api astropy -y
conda install gdal geopandas scikit-image notebook pyqt -y #will installing gdal cause problems???
conda install -c conda-forge “numpy>=1.16.5, <=1.23.0" -y


## additional coastseg dependencies
conda install ipython cartopy  tqdm  -y    ## pip no!
conda install -c conda-forge simplekml leafmap pydensecrf h5py -y
pip install area doodleverse_utils tensorflow

I will remove the old yml file from the next update as well as update the installation instructions. For some reason I'm having difficulty getting python=3.10 to work. I'll update this thread more as I find out more.

[dbuscombe-usgs]

Great - I will try out this recipe and report back

[dbuscombe-usgs]

Installed ok (and fast).

When I run app.ipynb and execute the first cell, I notice:

GPU name:  []
Num GPUs Available:  0

the terminal output says

raries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

I removed tensorflow then pip install tensorflow-gpu, but I get the same error messages as above, even though I can confirm that tensorflow-gpu installed correctly.

I get the same problems when using the older coastseg env.

[dbuscombe-usgs]

A good way to test whether or not tensorflow is registering a GPU is to do

import tensorflow as tf
physical_devices = tf.config.experimental.list_physical_devices('GPU')
print(physical_devices)

[dbuscombe-usgs]

I did pip uninstall tensorflow-gpu followed by conda install -c conda-forge tensorflow-gpu. It took a little longer to solve, but successfully installs tensorflow-gpu-2.4.1, finds the GPU (see last comment).

and now when I run app.ipynb, I get expected behaviour

GPU name:  [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Num GPUs Available:  1

So, I think we should use conda tensorflow-gpu. I think that tensorflow-gpu reverts back to tensorflow if no CUDA-enabled gpu is present

[2320sharon]

@dbuscombe-usgs thank you for finding and reporting this bug. I'm happy you found the solution was as simple as running conda install -c conda-forge tensorflow-gpu will this solution still work if pip install tensorflow was installed first?

[dbuscombe-usgs]

I'm not sure. I removed the pip-installed version first

[2320sharon]

I don't have a GPU so I can't test this as well. Is it possible for you to uninstall both then try running pip install tensorflow followed by conda install -c conda-forge tensorflow-gpu

[dbuscombe-usgs]

I did as you requested. I removed coastseg_test env and rebuilt, including the pip tensorflow. It installs tensorflow-2.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

then I did conda install -c conda-forge tensorflow-gpu

The following NEW packages will be INSTALLED:

  _tflow_select      pkgs/main/linux-64::_tflow_select-2.1.0-gpu
  absl-py            conda-forge/noarch::absl-py-1.2.0-pyhd8ed1ab_0
  astor              conda-forge/noarch::astor-0.8.1-pyh9f0ad1d_0
  astunparse         conda-forge/noarch::astunparse-1.6.3-pyhd8ed1ab_0
  blinker            conda-forge/noarch::blinker-1.4-py_1
  cudatoolkit        conda-forge/linux-64::cudatoolkit-10.1.243-h8cb64d8_10
  cudnn              conda-forge/linux-64::cudnn-7.6.5.32-hc0a50b0_1
  cupti              pkgs/main/linux-64::cupti-10.1.168-0
  gast               conda-forge/noarch::gast-0.4.0-pyh9f0ad1d_0
  google-auth-oauth~ conda-forge/noarch::google-auth-oauthlib-0.4.6-pyhd8ed1ab_0
  google-pasta       conda-forge/noarch::google-pasta-0.2.0-pyh8c360ce_0
  keras-preprocessi~ conda-forge/noarch::keras-preprocessing-1.1.2-pyhd8ed1ab_0
  markdown           conda-forge/noarch::markdown-3.4.1-pyhd8ed1ab_0
  oauthlib           conda-forge/noarch::oauthlib-3.2.0-pyhd8ed1ab_0
  opt_einsum         conda-forge/noarch::opt_einsum-3.3.0-pyhd8ed1ab_1
  pyjwt              conda-forge/noarch::pyjwt-2.4.0-pyhd8ed1ab_0
  python-flatbuffers conda-forge/noarch::python-flatbuffers-2.0-pyhd8ed1ab_0
  requests-oauthlib  conda-forge/noarch::requests-oauthlib-1.3.1-pyhd8ed1ab_0
  tensorboard        conda-forge/noarch::tensorboard-2.10.0-pyhd8ed1ab_1
  tensorboard-data-~ conda-forge/linux-64::tensorboard-data-server-0.6.0-py38h2b5fc30_2
  tensorboard-plugi~ conda-forge/noarch::tensorboard-plugin-wit-1.8.1-pyhd8ed1ab_0
  tensorflow         pkgs/main/linux-64::tensorflow-2.4.1-gpu_py38h8a7d6ce_0
  tensorflow-base    pkgs/main/linux-64::tensorflow-base-2.4.1-gpu_py38h29c2da4_0
  tensorflow-estima~ conda-forge/linux-64::tensorflow-estimator-2.6.0-py38h709712a_0
  tensorflow-gpu     pkgs/main/linux-64::tensorflow-gpu-2.4.1-h30adc30_0
  termcolor          conda-forge/noarch::termcolor-1.1.0-pyhd8ed1ab_3
  werkzeug           conda-forge/noarch::werkzeug-2.2.2-pyhd8ed1ab_0
  wrapt              conda-forge/linux-64::wrapt-1.14.1-py38h0a891b7_0

The following packages will be DOWNGRADED:

  h5py                         3.1.0-nompi_py38hafa665b_100 --> 2.10.0-nompi_py38h9915d05_106

It doesn't really make sense to me to install pip-tensorflow alongside conda-tensorflow-gpu, but I can confirm that it does work

python -c "import tensorflow as tf; print(tf.__version__)"
2.4.1

and app.ipynb works normally

GPU name:  [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Num GPUs Available:  1

🎉

[2320sharon]

Thank you for trying this out 👍
I'll create an environment using conda-tensorflow-gpu without pip-tensorflow on a machine without a GPU and see how it goes.

[2320sharon]

I can confirm that using conda install -c conda-forge tensorflow-gpu works great and that pip install tensorflow is not necessary because conda install -c conda-forge tensorflow-gpu installs tensorflow

[2320sharon]

I'm going to close this issue since the current installation instructions have been updated in both the readme.md and in install.md.