Skip to content

Installation of the BasinMaker

Jump to bottom
dustming edited this page Feb 2, 2021 · 27 revisions

Table of content

  1. Install basinmaker to processing existing routing structure

    1.1 Installation procedure for QGIS working under anaconda environments

    1.2 Installation procedure for ArcGIS working under ArcGIS Pro environments

  2. Install basinmaker to delineate lake-river routing structure

    2.1. Overview

    2.2. Install QGIS and GRASS in different systems

    2.2.1 Install QGIS and GRASS in Windows systems

    2.2.2 Install QGIS and GRASS in Ubuntu systems

    2.2.3 Install QGIS and GRASS in Graham systems

    2.3. Install other dependent python packages

    2.4. Install BasinMaker

    2.5. Functions from QGIS and GRASS GIS

Install basinmaker to processing existing routing structure

Installation procedure for qgis working under anaconda environments

If basinmaker is only used to processing existing routing structure like the version 2 lake river routing product, the basinmake can be worked with in the anaconda environment. It's is the clean and simplest way to use basinmker.

Install anaconda

The installer of anaconda can be installed from here. Tools can be used without anaconda. If no anaconda is installed user has to manually setup QGIS python environment.

Create a empty python environment

Open conda command line, do following

conda create --name basinmaker
conda activate basinmaker

Install qgis to this environment

conda install -c conda-forge qgis

Install basinmaker to this environment

git clone https://github.com/dustming/basinmaker.git basinmaker 
conda develop ./basinmaker/basinmaker

Set up QGIS environment variable

# find path of QGIS_PREFIX_PATH
python
>>> import os
>>> os.environ
# alternatively, to look for QGIS_PREFIX_PATH specifically:
>>> print(os.environ.get('QGIS_PREFIX_PATH'))

>>> quit()
# check if QGIS_PREFIX_PATH is already exist in the environment 
# if yes move to next step 
# if not 
# find a environment variable name similar to QGIS_PREFIX_PATH, and use it to define 
# a new environment variable. Normally it should be a path to the environment library 
conda env config vars set QGIS_PREFIX_PATH='C:\\Users/dustm/anaconda3/envs/basinmaker/Library'
conda deactivate 
conda activate basinmaker

Install needed python packages and run following examples:

pip install pandas pytest scipy simpledbf netCDF4

Installation procedure for ArcGIS working under ArcGIS Pro environments

Create a new ArcGIS pro python work environment

First, a new ArcGIS pro python work environment needs to be created following the instructions in here

Install python packages

Then active python command Prompt, which can be found by search "python command Prompt". In the active python command prompt, active the new ArcGIS pro work environment you had just created and run following commands:

pip install simpledbf

Install basinmaker to this environment

Download basinmaker to a folder where you can easily find later. Let's say BasinMaker_Foder

git clone https://github.com/dustming/basinmaker.git basinmaker 
python setup.py develop

Here are some arcgis pro python examples

The basinmaker can also running within GUI of ArcGIS pro. which requires load basinmaker.tbx via Arc Catalog.

Install basinmaker to delineate lake-river routing structure

Overview

The BasinMaker is developed using Python3 languages and the installation of the BasinMaker itself is very simple, the hard part is to install QGIS and GRASS GIS under the same python environment since the BasinMaker using some functions from both QGIS and GRASS GIS. A list of dependent python packages can be found in the following Table 1. And List of GRASS addons needed by the toolbox is showed in Table 2.

The installation of BasinMaker can be divided into two steps. First, install the required packages and addons listed in Table 1 and Table 2. The installation procedure for the different operating systems can be found here. Please note python3.8, QGIS 3.1x, and GRASS 7.x are preferred. The second step is to install BasinMaker itself, which can be found in here

Table 1 List of python packages required by BasinMaker

Python packages Come from How to install
qgis QGIS Section 2
qgis.core QGIS Section 2
qgis.analysis QGIS Section 2
qgis.PyQt.QtCore QGIS Section 2
simpledbf Python pip
os Python pip
sys Python pip
numpy Python pip
shutil Python pip
distutils Python pip
tempfile Python pip
copy Python pip
pandas Python pip
sqlite3 Python pip
timeit Python pip
grass_session Python pip
grass.script GRASS Section 2
grass.pygrass.modules.shortcuts GRASS Section 2
grass.pygrass.modules GRASS Section 2
processing.core.Processing GRASS Section 2
processing.tools GRASS Section 2

Table 2 list of Grass addons required by BasinMaker

GRASS ADDONS How to install
r.accumulate Section 2
r.clip Section 2
r.stream.basins Section 2
r.stream.snap Section 2

Install QGIS and GRASS in different systems

Install QGIS and GRASS in Windows system

Installation of software

For the Windows system, QGIS and GRASS can be installed at the same by installing the QGIS with GRASS addon. The installer can be downloaded from here. Normally we will have both QGIS and GRASS installed in the system, and both of them are linked with python3 under the QGIS installation folder. The installation of GRASS addon for Windows system can be found here

Setup python environment for QGIS and GRASS GIS in Windows

The python environment for QGIS and GRASS GIS in Windows can be set up by modifying the following command or copy them into a .bat file

@echo off

SET OSGEO4W_ROOT=C:\QGIS310
SET GRASS_ROOT=C:\QGIS310\apps\grass\grass78
SET GISBASE=C:\QGIS310\apps\grass\grass78
SET QGISPrefixPath=C:\QGIS310\apps\qgis
SET GRASSBIN=C:\QGIS310\bin\grass78.bat

@echo off
call %OSGEO4W_ROOT%\bin\o4w_env.bat
call %OSGEO4W_ROOT%\apps\grass\grass78\etc\env.bat
call qt5_env.bat
call py3_env.bat
@echo off

path %PATH%;%OSGEO4W_ROOT%\apps\qgis\bin
path %PATH%;%OSGEO4W_ROOT%\apps\qgis\python\plugins
path %PATH%;%OSGEO4W_ROOT%\apps\Qt5\bin
path %PATH%;%OSGEO4W_ROOT%\apps\Python37\Scripts

path %PATH%;%GRASS_ROOT%\lib
path %PATH%;%GRASS_ROOT%\bin
path %PATH%;%GRASS_ROOT%\script

set GDAL_FILENAME_IS_UTF8=YES

SET PYTHONHOME=C:\QGIS310\apps\Python37

SET PYTHONPATH=%PYTHONPATH%;%OSGEO4W_ROOT%\apps\qgis\python;%OSGEO4W_ROOT%\apps\qgis\python\plugins;%OSGEO4W_ROOT%\apps\qgis\python\plugins\processing
SET PYTHONPATH=%PYTHONPATH%;%GRASS_ROOT%\etc\python\;%GRASS_ROOT%\etc\python\grass;%GRASS_ROOT%\etc\python\grass\script

rem Set VSI cache to be used as buffer, see #6448
set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000

set QT_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\qgis\qtplugins;%OSGEO4W_ROOT%\apps\qt5\plugins
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT:\=/%/apps/qgis

Validate the installation

First, open a command-line window and type, and copy the modified above command or run a .bat file that contains the above-modified commands. Check if the current default python is the python from the QGIS folder.

>where python 
>C:\QGIS310\apps\Python37\python.exe

Check if we can load the required QGIS and GRASS packages

>python
>>>from qgis.core import *
>>>import qgis
>>>from qgis.analysis import QgsNativeAlgorithms
>>>from qgis.PyQt.QtCore import *
>>>from qgis import processing
>>>from processing.core.Processing import Processing
>>>from processing.tools import dataobjects
>>>import grass.script as grass
>>>from grass.script import array as garray
>>>from grass.script import core as gcore
>>>import grass.script.setup as gsetup
>>>from grass.pygrass.modules.shortcuts import general as g
>>>from grass.pygrass.modules.shortcuts import raster as r
>>>from grass.pygrass.modules import Module

Install QGIS and GRASS in Ubuntu systems

Installation of software

Similar to Windows, Both QGIS and GRASS can be installed at the same time by installing the QGIS with GRASS addon. The installation procedure is the following:

  • Check system default python 3 path and it's version, better be python3.8.
which python3 
/user/bin/python3
python 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
  • Install QGIS with grass plugin Please following the instruction from here to install QGIS with the GRASS plugin.
sudo apt install gnupg software-properties-common

wget -qO - https://qgis.org/downloads/qgis-2020.gpg.key | sudo gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/qgis-archive.gpg --import

sudo chmod a+r /etc/apt/trusted.gpg.d/qgis-archive.gpg

sudo add-apt-repository "deb https://qgis.org/debian `lsb_release -c -s` main"

sudo apt update

sudo apt install qgis qgis-plugin-grass

Setup python environment for QGIS and GRASS GIS in Windows

In an ubuntu system, the QGIS will be installed with system default python3. we can directly import QGIS using system default python3.

python3 
>>>import qgis
>>>qgis.__file__  ### print qgis module path 
'/usr/lib/python3/dist-packages/qgis/__init__.py'

However, GRASS is not installed in the default as python3 site-packages. Need to setup a GRASS python environment by following steps.

a) install GRASS GUI and GRASS development package

sudo apt install grass-gui ### install grass GUI 
sudo apt install grass-dev ### install grass development package

b) load GRASS GUI and find path grass python modules

grass ## to load the grass 
### open the interactive python shell within grass GUI.
>>>  import os 
>>>  os.environ['PYTHONPATH']
/usr/lib/grass78/etc/python
>>>  os.environ['GISBASE']
'/usr/lib/grass78'

We need to add printed PYTHONPATH and GISBASE into the system and python environment variables as following:

export GISBASE='/usr/lib/grass78'
export QGISPrefixPath='/usr'

export PYTHONPATH=$PYTHONPATH:'/usr/lib/grass78/etc/python'  ### folder has a grass folder
export PYTHONPATH=$PYTHONPATH:'/usr/share/qgis/python/plugins' ## folder has db_manager and processing
export PYTHONPATH=$PYTHONPATH:'/usr/share/qgis/python' ## folder has plugin and console

Validate the installation

First, open a command line and then copy the modified above command line and then run the following commands.

>python
>>>from qgis.core import *
>>>import qgis
>>>from qgis.analysis import QgsNativeAlgorithms
>>>from qgis.PyQt.QtCore import *
>>>from qgis import processing
>>>from processing.core.Processing import Processing
>>>from processing.tools import dataobjects
>>>import grass.script as grass
>>>from grass.script import array as garray
>>>from grass.script import core as gcore
>>>import grass.script.setup as gsetup
>>>from grass.pygrass.modules.shortcuts import general as g
>>>from grass.pygrass.modules.shortcuts import raster as r
>>>from grass.pygrass.modules import Module

Install QGIS and GRASS in Graham systems

Installation of software

In the Graham system, the QGIS3.1x is preinstalled and can be load directly, but unfortunately, the preinstalled GRASS is complied with different compilers and thus can not be load with QGIS at the same time. So we have to load preinstalled QGIS and then manually compile GRASS.

  • Load pre-installed QGIS and setup python environment
module load python/3.8.2 
module load StdEnv/2020
module load gcc/9.3.0
module load qgis/3.10.6
export PYTHONPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.10.6/share/qgis/python:$PYTHONPATH
export PYTHONPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.10.6/share/qgis/python/plugins:$PYTHONPATH
export LD_LIBRARY_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.10.6/lib:$LD_LIBRARY_PATH

At this time we will find all QGIS functions can be load successfully in python

$which python
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.2/bin/python
$python
>>>from qgis.core import *
>>>import qgis
>>>from qgis.analysis import QgsNativeAlgorithms
>>>from qgis.PyQt.QtCore import *
>>>from qgis import processing
>>>from processing.core.Processing import Processing
>>>from processing.tools import dataobjects
  • Install GRASS and grass plugin
mkdir $HOME/src
mkdir $HOME/GRASS
cd $HOME/src/
git clone https://github.com/OSGeo/grass.git grass_master 
git clone https://github.com/OSGeo/grass-addons grass_addon
cd grass_master 
./configure  --prefix=$HOME/GRASS --with-cxx --with-sqlite --with-python --with-geos  --with-gdal --without-freetype --without-fftw --with-proj
make -j2
make install
##### install addons 
export MODULE_TOPDIR=$HOME/GRASS/grass79/
cd $HOME/src/grass_addon/grass7/raster/r.stream.basins
make MODULE_TOPDIR=$MODULE_TOPDIR

cd $HOME/src/grass_addon/grass7/raster/r.stream.snap
make MODULE_TOPDIR=$MODULE_TOPDIR

cd $HOME/src/grass_addon/grass7/raster/r.clip
make MODULE_TOPDIR=$MODULE_TOPDIR

cd $HOME/src/grass_addon/grass7/raster/r.accumulate
make MODULE_TOPDIR=$MODULE_TOPDIR

Setup python environment for QGIS and GRASS GIS in Graham

export PYTHONPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.10.6/share/qgis/python:$PYTHONPATH
export PYTHONPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.10.6/share/qgis/python/plugins:$PYTHONPATH
export LD_LIBRARY_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.10.6/lib:$LD_LIBRARY_PATH
export QGISPrefixPath=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.10.6/
export GISRC=$HOME/GRASS/grass79/demolocation/.grassrc79 
export GISBASE=$HOME/GRASS/grass79
export GRASSBIN=$HOME/GRASS/bin/grass79
export PATH=/home$HOME/grass79/bin:$PATH
export PATH=/home$HOME/grass79/scripts:$PATH 
export PYTHONPATH=$HOME/GRASS/grass79/etc/python:$PYTHONPATH 
export LD_LIBRARY_PATH=$HOME/GRASS/grass79/bin:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$HOME/GRASS/grass79/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$HOME/GRASS/grass79/scripts:$LD_LIBRARY_PATH

Validate the installation

First, open a command line and then copy the modified above command line and then run the following commands.

>python
>>>from qgis.core import *
>>>import qgis
>>>from qgis.analysis import QgsNativeAlgorithms
>>>from qgis.PyQt.QtCore import *
>>>from qgis import processing
>>>from processing.core.Processing import Processing
>>>from processing.tools import dataobjects
>>>import grass.script as grass
>>>from grass.script import array as garray
>>>from grass.script import core as gcore
>>>import grass.script.setup as gsetup
>>>from grass.pygrass.modules.shortcuts import general as g
>>>from grass.pygrass.modules.shortcuts import raster as r
>>>from grass.pygrass.modules import Module

Install other dependent python packages

For other dependent python packages, they can be installed via pip. Most of them included in python, only a few unavailable packages need to be installed. Before using pip, we need to set up the python environment for QGIS and GRASS as discussed in the previous section. Then using pip to install packages listed in Table 1.

Install BasinMaker

Download or clone BasinMaker repository by:

git clone https://github.com/dustming/RoutingTool.git GIS_Toolbox

Add the following path to system and python environment variables:

export PYTHONPATH=$Path_to_BasinMaker:$PYTHONPATH 
export PYTHONPATH=$Path_to_BasinMaker/Toolbox_Common_Functions:$PYTHONPATH 
export RoutingToolFolder=$Path_to_BasinMaker

Functions from QGIS and GRASS GIS

Function name QGIS/GRASS
g.region GRASS
garray.array GRASS
garray.array.write GRASS
gdal:translate GRASS
r.accumulate GRASS
r.cross GRASS
r.import GRASS
r.in.gdal GRASS
r.mapcalc GRASS
r.mask GRASS
r.null GRASS
r.out.gdal GRASS
r.pack GRASS
r.reclass GRASS
r.slope.aspect GRASS
r.stats.zonal GRASS
r.stream.basins GRASS
r.stream.extract GRASS
r.stream.snap GRASS
r.to.vect GRASS
r.unpack GRASS
r.water.outlet GRASS
r.watershed GRASS
v.buffer GRASS
v.db.addcolumn GRASS
v.db.dropcolumn GRASS
v.db.join GRASS
v.db.update GRASS
v.dissolve GRASS
v.import GRASS
v.out.ogr GRASS
v.overlay GRASS
v.pack GRASS
v.rast.stats GRASS
v.to.rast GRASS
v.vect.stats GRASS
v.what.rast GRASS
gdal:cliprasterbymasklayer QGIS
gdal:dissolve QGIS
gdal:polygonize QGIS
gdal_rasterize QGIS
native:addxyfields QGIS
native:buffer QGIS
native:centroids QGIS
native:extractbyexpression QGIS
native:extractbylocation QGIS
native:fixgeometries QGIS
native:joinattributestable QGIS
native:polygonstolines QGIS
native:reprojectlayer QGIS
native:reprojectlayer QGIS
QgsFeatureRequest().setFlags QGIS
QgsRasterLayer QGIS
QgsRasterLayer.crs().authid() QGIS
QgsRasterLayer.rasterUnitsPerPixelX() QGIS
QgsVectorLayer QGIS
QgsVectorLayer.getFeatures QGIS
QgsVectorLayer.select QGIS
QgsVectorLayer.writeAsVectorFormat QGIS
saga:cliprasterwithpolygon QGIS


© 2019-2021 - BasinMaker- ming.han@uwaterloo.ca

Clone this wiki locally