Virtual Environments - Install & Run HSP2 Python on Windows #55

jdkleiner · 2023-01-10T22:08:53Z

TL;DR This method is preferred for running HSP2 on windows because it keeps hsp2 and all of its dependencies installed in a virtual environment using its own python version and package versions. This method is also preferred because it prevents the user from having to make any tweaks to the file HSPsquared/HSP2/environment.yml in order to run hsp2. See the bad way to run hsp2 here: #48

Quick Reference for running HSP2:

bash: Start Up a virtual environment and cs to HSP2 directory

# start of path is the home directory where venvs were created
source /c/Users/faw18626/virtualenvs/hsp2dev_py10/Scripts/activate

Setup p10 on Windows in PowerShell

# install the necessary virtual environment modules
C:\usr\local\bin\python\python.exe  -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --user virtualenv --default-timeout=100
C:\usr\local\bin\python\python.exe  -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --default-timeout=100 pyreadline3

cd "C:\Users\faw18626\virtualenvs"
C:\usr\local\bin\python\python.exe -m virtualenv hsp2dev_py10
# Activate it
./hsp2dev_py10/Scripts/activate

cd c:/usr/local/home/git/HSPsquared
pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --default-timeout=100 -e .

Activate it for use in gitbash shell with py and hsp2

cd "C:\Users\faw18626\virtualenvs"

# Actviate in Bash
source ./hsp2dev_py10/Scripts/activate

cd c:/usr/local/home/git/HSPsquared

Activate it for use in PowerShell with py and hsp2

cd "C:\Users\faw18626\virtualenvs"

# Activate it in PS shell:
./hsp2dev_py10/Scripts/activate

cd c:/usr/local/home/git/HSPsquared

Running Full hsp2 executable

Open a Command Prompt and run the following
- Opening /HSPsquared locally in VSCode is optimal for development

cd /d C:/Users/nrf46657/virtualenvs
hsp2dev_venv\Scripts\activate
cd C:/Users/nrf46657/Desktop/GitHub/HSPsquared

# test10 
cd tests/test10/HSP2results
hsp2 import_uci test10.uci test10.h5
hsp2 run test10.h5

#testcbp
cd tests/testcbp/HSP2results
hsp2 import_uci PL3_5250_0001.uci PL3_5250_0001.h5
hsp2 run PL3_5250_0001.h5

#install the package once using "-e" for editable mode
#this ensures that you wont have to reinstall the package each time you make file changes during development!
pip install -e ../../../.

#note, the following is no longer necessary!
#reinstall hsp2 after making changes
# pip install ../../../.
# pip install --ignore-installed ../../../.
# cd C:\Users\nrf46657\Desktop\GitHub\HSPsquared
# pip install .

# deactivate the virtual environment when finished
deactivate

Running Command Line Tests in python shell

First time you clone git, or after a major refactor, you must reinstall (in a venv) pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --default-timeout=100 -e .

Detailed notes for running HSP2 in virtual environment:

install python version of interest
- installation executables of previous versions of python can be found here: https://www.python.org/ftp/python/
- I installed version 3.8.9 as HSP2 requires python =3.8*
  - For the installer, use python-3.8.9-amd64.exe
create a local folder for housing virtual environments, I went with: "C:\Users\nrf46657\virtualenvs"
open a PS terminal within that directory
PS C:\Users\nrf46657\virtualenvs>

create a virtual environment, utilizing a specific install of python (i.e. the 3.8 version installed above)

First ensure you have the python package virtualenv installed

PS C:\Users\nrf46657> C:\Users\nrf46657\AppData\Local\Programs\Python\Python38\python.exe -m pip install --user virtualenv

My virtual environment will be named: hsp2dev_venv

PS C:\Users\nrf46657\virtualenvs> C:\Users\nrf46657\AppData\Local\Programs\Python\Python38\python.exe -m virtualenv hsp2dev_venv
created virtual environment CPython3.8.9.final.0-64 in 1125ms
  creator CPython3Windows(dest=C:\Users\nrf46657\virtualenvs\hsp2dev_venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\nrf46657\AppData\Local\pypa\virtualenv)
    added seed packages: pip==22.3.1, setuptools==65.6.3, wheel==0.38.4
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Take a look at the packages in the virtual environment

PS C:\Users\nrf46657\virtualenvs> ls hsp2dev_venv\Lib\site-packages
    Directory: C:\Users\nrf46657\virtualenvs\hsp2dev_venv\Lib\site-packages


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         1/10/2023   4:04 PM                pip
d-----         1/10/2023   4:04 PM                pip-22.3.1.dist-info
d-----         1/10/2023   4:04 PM                pkg_resources
d-----         1/10/2023   4:04 PM                setuptools
d-----         1/10/2023   4:04 PM                setuptools-65.6.3.dist-info
d-----         1/10/2023   4:04 PM                wheel
d-----         1/10/2023   4:04 PM                wheel-0.38.4.dist-info
d-----         1/10/2023   4:04 PM                _distutils_hack
-a----         1/10/2023   4:04 PM            151 distutils-precedence.pth
-a----         1/10/2023   4:04 PM              0 pip-22.3.1.virtualenv
-a----         1/10/2023   4:04 PM              0 setuptools-65.6.3.virtualenv
-a----         1/10/2023   4:04 PM              0 wheel-0.38.4.virtualenv
-a----         1/10/2023   4:04 PM             18 _virtualenv.pth
-a----         1/10/2023   4:04 PM           5770 _virtualenv.py

Activate a virtual environment

First open a Command Prompt cmd
Navigate to directory housing your virtual environments

R:\>cd /d C:\Users\nrf46657\virtualenvs
C:\Users\nrf46657\virtualenvs>

hsp2dev_venv\Scripts\activate

C:\Users\nrf46657\virtualenvs>hsp2dev_venv\Scripts\activate

(hsp2dev_venv) C:\Users\nrf46657\virtualenvs>

You can verify the version of Python being used in the virtual environment

(hsp2dev_venv) C:\Users\nrf46657\Desktop\GitHub\HSPsquared>python -V
Python 3.8.9

install hsp2 within that virtual environment

still within cmd: change directory to HSPsquared

(hsp2dev_venv) C:\Users\nrf46657\virtualenvs>cd 
C:\Users\nrf46657\Desktop\GitHub\HSPsquared

(hsp2dev_venv) C:\Users\nrf46657\Desktop\GitHub\HSPsquared>

install hsp2

(hsp2dev_venv) C:\Users\nrf46657\Desktop\GitHub\HSPsquared>pip install .
Processing c:\users\nrf46657\desktop\github\hspsquared
  Preparing metadata (setup.py) ... done
Collecting scipy
  Using cached scipy-1.10.0-cp38-cp38-win_amd64.whl (42.2 MB)
Collecting pandas==1.2.*
  Using cached pandas-1.2.5-cp38-cp38-win_amd64.whl (9.3 MB)
Collecting numba==0.53.1
  Using cached numba-0.53.1-cp38-cp38-win_amd64.whl (2.3 MB)
Collecting numpy==1.20.*
  Using cached numpy-1.20.3-cp38-cp38-win_amd64.whl (13.7 MB)
...

This is super powerful because a) hsp2 gets installed within the virtual environment you created AND b) all of its hsp2 and all of its dependencies with the correct versions get installed within the virtual environment as well (i.e. all of the packages in the environment.yml get installed)

Verify success in PowerShell
- Just as before, take a look at the packages in the virtual environment (there should now be many more packages listed, including HSP2)
  - PS C:\Users\nrf46657\virtualenvs> ls hsp2dev_venv\Lib\site-packages

Verify you can now run hsp2

In cmd:

C:\Users\nrf46657\virtualenvs> 

hsp2dev_venv\Scripts\activate
cd tests/test10/HSP2results
hsp2 import_uci test10.uci test10.h5
hsp2 run test10.h5

The text was updated successfully, but these errors were encountered:

rburghol · 2023-09-20T14:28:41Z

Show version of python (example is inside vscode):

PS C:\usr\local\home\git\HSPsquared> py.exe -V  
Python 3.10.8
PS C:\usr\local\home\git\HSPsquared> py -0p
Installed Pythons found by C:\Users\faw18626\AppData\Local\Programs\Python\Launcher\py.exe Launcher for Windows
 -3.10-64       C:\usr\local\bin\python\python.exe *
 -3.8-64        C:\Users\faw18626\AppData\Local\Programs\Python\Python38\python.exe

PS C:\usr\local\home\git\HSPsquared>

Make my virtual environment directory and install virtualenv via pip:

mkdir $HOME\virtualenvs
# had to add stuff to allow ssl self-signed for pypi
C:\Users\faw18626\AppData\Local\Programs\Python\Python38\python.exe -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --user virtualenv

Verify virtualenv package:

C:\Users\faw18626\AppData\Local\Programs\Python\Python38\python.exe -m pip show virtualenv                                         
Name: virtualenv                                              
Version: 20.24.5
Summary: Virtual Python Environment builder
Home-page: None
Author: None
Author-email: None
License: None
Location: c:\users\faw18626\appdata\roaming\python\python38\site-packages
Requires: platformdirs, filelock, distlib
Required-by:

Create the environment

C:\Users\faw18626\AppData\Local\Programs\Python\Python38\python.exe -m virtualenv hsp2dev_venv 
created virtual environment CPython3.8.9.final.0-64 in 3105ms
  creator CPython3Windows(dest=C:\Users\faw18626\virtualenvs\hsp2dev_venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\faw18626\AppData\Local\pypa\virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.2.0, wheel==0.41.2
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Activate

# This was the original cd, but gave error when later invoked:
# cd /d C:\Users\faw18626\virtualenvs  
# instead, this worked fine
cd C:\Users\faw18626\virtualenvs

# activate the env
hsp2dev_venv\Scripts\activate
# verify the version of python
py.exe -V
# Python 3.8.9
# install, note: inside DEQ we may need to use the trusted-host things
# we also add default-timeout=1000 
# because the install repeatedly choked at installing "jedi" with a time out
pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --default-timeout=1000 -e .

rburghol · 2023-10-03T10:37:17Z

Inside VSCode

In VSCode type:

open the Command Palette (Ctrl+Shift+P)
Select Python: Select Interpreter
- A list of python.exe and venvs will appear
- If your desired venv shows up then:
  - select it, then proceed to step 3 below
- If your desired venv does NOT appear in this list:
  - then select + Enter Interpret path from the list
  - Enter the path to your virtual environment, ex: C:/Users/faw18626/virtualenvs/hsp2dev_venv
  - proceed to step 3 below
Run the python terminal
- Create a new powershell terminal (not a bash term, since the python executable doesn't have command history)
- Enter exec path to your Venv python executable, ex: C:/Users/faw18626/virtualenvs/hsp2dev_venv/Scripts/python.exe

Simple: Activate with gitbash (not powershell)

cd "C:\Users\faw18626\virtualenvs"
source hsp2dev_venv/Scripts/activate

# otherwise works as above with powershell
# install newest version of hsp2
# this is only needed if we've updated source code
python -m pip install . --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org

# go to directory with your model
cd /c/WorkSpace/modeling/hspf/hsp2/testcbp/
hsp2 import_uci PL3_5250_0001.uci PL3_5250_0001.h5
hsp2 run PL3_5250_0001.h5

rburghol · 2024-01-08T16:24:17Z

Run hsp2 in interactive session in vscode

install PyQt5 (to allow running interactive mode): python -m pip install PyQt5 --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org
open file HSP2_Driver.py
From Run menu, select Run -> Start Debugging
- A file dialog will open, Browse to the desired hdf5 file, select it
After running, try adding breakpoints and re-running
See below, example of model paused at breakpoint, variable values viewable from "Variables" in left column of vscode environment

rburghol · 2024-05-06T14:18:31Z

Set up python 13 venv

note: set a custom default-timeout parameter due to network latency
done with powershell this time

# install the necessary virtual environment modules
C:\Users\faw18626\AppData\Local\Programs\Python\Python313\python.exe  -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --user virtualenv --default-timeout=100

# create the virtual environment
cd "C:\Users\faw18626\virtualenvs"
C:\Users\faw18626\AppData\Local\Programs\Python\Python313\python.exe -m virtualenv hsp2dev_py313

# Activate the new env
./hsp2dev_py313/Scripts/activate # note, in bash use: source hsp2dev_py313/Scripts/activate

# install hsp2
cd C:\usr\local\home\git\HSPsquared
python -m pip install . --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --default-timeout=100

rburghol · 2024-05-06T20:38:19Z

Setup p10 on Windows in PowerShell

# install the necessary virtual environment modules
C:\usr\local\bin\python\python.exe  -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --user virtualenv --default-timeout=100
C:\usr\local\bin\python\python.exe  -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --default-timeout=100 pyreadline3

cd "C:\Users\faw18626\virtualenvs"
C:\usr\local\bin\python\python.exe -m virtualenv hsp2dev_py10
# Activate it
./hsp2dev_py10/Scripts/activate

cd c:/usr/local/home/git/HSPsquared
pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --default-timeout=100 -e .

Activate it for use in gitbash shell with py and hsp2

cd "C:\Users\faw18626\virtualenvs"

# Actviate in Bash
source ./hsp2dev_py10/Scripts/activate

cd c:/usr/local/home/git/HSPsquared

Activate it for use in PowerShell with py and hsp2

cd "C:\Users\faw18626\virtualenvs"

# Activate it in PS shell:
./hsp2dev_py10/Scripts/activate

cd c:/usr/local/home/git/HSPsquared

rburghol · 2024-05-09T21:30:07Z

Run and disable numba, which allows to do without a recompile:

NUMBA_DISABLE_JIT=1 hsp2 run test10specl.h5

rburghol · 2024-05-23T16:36:27Z

Get venv going on linux:

# install the venv package sitewide
sudo python3.9 -m pip install virtualenv
# create a 
cd /usr/local/share/
mkdir venv
sudo chgrp allmodelers venv
sudo chmod g+w venv
cd venv/
# create the python 3.10 env (actually ended up being 3.9+ but ok)
python3.9 -m virtualenv hsp2dev_py10
chgrp allmodelers hsp2dev_py10 -Rf
chmod g+w hsp2dev_py10/* -Rf
cd /opt/model/HSPsquared/
source /usr/local/share/venv/hsp2dev_py10/bin/activate
pip3.9 install .

# then run as per normal
rm PL3_5250_0001.h5
hsp2 import_uci PL3_5250_0001.uci PL3_5250_0001.h5
hsp2 run PL3_5250_0001.h5

jdkleiner mentioned this issue Jan 18, 2023

Set up dev environment and/or install paths and/or scripts for hsp2 tinkering #6

Closed

jdkleiner mentioned this issue Jul 10, 2023

Install & Run HSP2 Python on Windows #48

Closed

rburghol mentioned this issue Apr 19, 2024

Development Plan: Merges into develop-specact (1-2/2024) #80

Open

35 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Virtual Environments - Install & Run HSP2 Python on Windows #55

Virtual Environments - Install & Run HSP2 Python on Windows #55

jdkleiner commented Jan 10, 2023 •

edited by rburghol

Loading

rburghol commented Sep 20, 2023 •

edited

Loading

rburghol commented Oct 3, 2023 •

edited

Loading

rburghol commented Jan 8, 2024 •

edited

Loading

rburghol commented May 6, 2024 •

edited

Loading

rburghol commented May 6, 2024 •

edited

Loading

rburghol commented May 9, 2024

rburghol commented May 23, 2024 •

edited

Loading

Virtual Environments - Install & Run HSP2 Python on Windows #55

Virtual Environments - Install & Run HSP2 Python on Windows #55

Comments

jdkleiner commented Jan 10, 2023 • edited by rburghol Loading

Quick Reference for running HSP2:

bash: Start Up a virtual environment and cs to HSP2 directory

Setup p10 on Windows in PowerShell

Activate it for use in gitbash shell with py and hsp2

Activate it for use in PowerShell with py and hsp2

Running Full hsp2 executable

Running Command Line Tests in python shell

Detailed notes for running HSP2 in virtual environment:

rburghol commented Sep 20, 2023 • edited Loading

rburghol commented Oct 3, 2023 • edited Loading

Inside VSCode

Simple: Activate with gitbash (not powershell)

rburghol commented Jan 8, 2024 • edited Loading

rburghol commented May 6, 2024 • edited Loading

rburghol commented May 6, 2024 • edited Loading

Setup p10 on Windows in PowerShell

Activate it for use in gitbash shell with py and hsp2

Activate it for use in PowerShell with py and hsp2

rburghol commented May 9, 2024

rburghol commented May 23, 2024 • edited Loading

jdkleiner commented Jan 10, 2023 •

edited by rburghol

Loading

rburghol commented Sep 20, 2023 •

edited

Loading

rburghol commented Oct 3, 2023 •

edited

Loading

rburghol commented Jan 8, 2024 •

edited

Loading

rburghol commented May 6, 2024 •

edited

Loading

rburghol commented May 6, 2024 •

edited

Loading

rburghol commented May 23, 2024 •

edited

Loading