Trouble with swmm-toolkit install #79

karosc · 2021-07-20T14:42:15Z

I am getting various errors after installing on windows and linux. I also tried building and installing from source and still get the same errors. Not sure if I am missing something.

Windows

after pip install swmm-toolkit executing from swmm.toolkit import output, output_enum yields:

>>> from swmm.toolkit import output, output_enum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~\Miniconda\envs\swmm\lib\site-packages\swmm\toolkit\output.py", line 18, in <module>
    from . import _output
ImportError: DLL load failed while importing _output: The specified module could not be found.

Linux

after pip install swmm-toolkit executing from swmm.toolkit import output, output_enum yields:

>>> from swmm.toolkit import output, output_enum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'output_enum' from 'swmm.toolkit' (~/miniconda3/envs/swmm/lib/python3.9/site-packages/swmm/toolkit/__init__.py)

The text was updated successfully, but these errors were encountered:

michaeltryby · 2021-07-20T14:57:38Z

Did you activate your virtual environment prior to using it? If you don’t activate the search path for dlls won’t be set properly.

karosc · 2021-07-20T14:59:12Z

Does it only work with venv or virtualenv? I am using a conda env that is activated.

michaeltryby · 2021-07-20T15:00:32Z

It should work. I use both conda and venv in development.

michaeltryby · 2021-07-20T15:01:35Z

Are you installing from the wheels on pypi or from source?

karosc · 2021-07-20T15:07:47Z

I've tried both and am getting the same error. I'm not sure how to check by search path for dlls, is it just my $PATH env variable?

michaeltryby · 2021-07-20T15:12:46Z

In the latest version output_enum.py has been renamed shared_enum.py. So that would solve the error you are getting on Linux

jennwuu · 2021-07-20T15:16:57Z

Can you confirm aenum version?

karosc · 2021-07-20T15:19:26Z

Linux : aenum==2.2.6
Windows : aenum==2.2.6

michaeltryby · 2021-07-20T15:20:38Z

What version of swmm-toolkit do you have installed?

karosc · 2021-07-20T15:22:48Z

swmm-toolkit = 0.8.2

michaeltryby · 2021-07-20T15:32:23Z

When I install from pypi into a conda env everything works properly. Based on the info you have provided I can't isolate the problem. Can you take us through step by step what you are doing so we can recreate the problem on our side?

karosc · 2021-07-20T15:32:54Z

ok, so changing output_enum to shared_enum fixed the issue on linux.

On windows I made a fresh conda env, ran pip install swmm-toolkit am still getting:

>>> from swmm.toolkit import output, shared_enum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\KAROSC\Miniconda\envs\swmmtk\lib\site-packages\swmm\toolkit\output.py", line 18, in <module>
    from . import _output
ImportError: DLL load failed while importing _output: The specified module could not be found.

popped open my site-packages directory and see swmm and swmm_toolkit folders, which matches what I find on linux. Below are the dlls in the swmm_toolkit folder:

Directory: C:\Users\KAROSC\Miniconda\envs\swmmtk\Lib\site-packages\swmm_toolkit

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         7/20/2021  11:28 AM         332568 concrt140.dll
-a----         7/20/2021  11:28 AM         627992 msvcp140.dll
-a----         7/20/2021  11:28 AM          31512 msvcp140_1.dll
-a----         7/20/2021  11:28 AM         206104 msvcp140_2.dll
-a----         7/20/2021  11:28 AM          19456 swmm-output.dll
-a----         7/20/2021  11:28 AM         470016 swmm5.dll
-a----         7/20/2021  11:28 AM         154392 vcomp140.dll
-a----         7/20/2021  11:28 AM          85784 vcruntime140.dll

michaeltryby · 2021-07-20T15:34:29Z

That's good info. Whats in the swmm.toolkit folder? Cause its not able to load the swmm/toolkit/_output*.dll for some reason

karosc · 2021-07-20T15:35:18Z

    Directory: C:\Users\KAROSC\Miniconda\envs\swmmtk\Lib\site-packages\swmm\toolkit


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         7/20/2021  11:28 AM                __pycache__
-a----         7/20/2021  11:28 AM          14300 output.py
-a----         7/20/2021  11:28 AM          12803 output_metadata.py
-a----         7/20/2021  11:28 AM          16097 shared_enum.py
-a----         7/20/2021  11:28 AM          43481 solver.py
-a----         7/20/2021  11:28 AM          52736 _output.cp39-win_amd64.pyd
-a----         7/20/2021  11:28 AM         161792 _solver.cp39-win_amd64.pyd
-a----         7/20/2021  11:28 AM            893 __init__.py

michaeltryby · 2021-07-20T15:36:51Z

The library is in there. Why can't it find it?

michaeltryby · 2021-07-20T15:38:21Z

What shell are you using? Powershell?

michaeltryby · 2021-07-20T15:39:36Z

Could it be a permissions issue? You don't appear to have execution rights for the library.

jennwuu · 2021-07-20T15:42:10Z

We have seen this error before with conda installation.

Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated.  Libraries may fail to load.  To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from swmm.toolkit import solver
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\Anaconda3\lib\site-packages\swmm\toolkit\solver.py", line 18, in <module>
    from . import _solver
ImportError: DLL load failed while importing _solver: The specified module could not be found.

It happens when I don't activate my conda environment using:
conda activate

karosc · 2021-07-20T15:44:36Z

I am activating for sure:


(jupyter) PS C:\CKAROS_PROJECTS> conda activate
(base) PS C:\CKAROS_PROJECTS> conda activate swmmtk
(swmmtk) PS C:\CKAROS_PROJECTS> python
Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:37:25) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from swmm.toolkit import output, shared_enum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\KAROSC\Miniconda\envs\swmmtk\lib\site-packages\swmm\toolkit\output.py", line 18, in <module>
    from . import _output
ImportError: DLL load failed while importing _output: The specified module could not be found.
>>>

jennwuu · 2021-07-20T15:46:38Z

@karosc Can you try adding your miniconda path (and miniconda\Library\bin) to your system environment path?

Edit: Maybe C:\Users\KAROSC\Miniconda\Library\bin\ ? I'm not familiar with miniconda

karosc · 2021-07-20T15:51:19Z

I ran the following on both my working linux and not working windows systems:

import sys
[print(i) for i in sys.path

Linux output

~/miniconda3/envs/swmm/lib/python39.zip
~/miniconda3/envs/swmm/lib/python3.9
~/miniconda3/envs/swmm/lib/python3.9/lib-dynload
~/miniconda3/envs/swmm/lib/python3.9/site-packages

Windows output

C:\Users\KAROSC\Miniconda\envs\swmmtk\python39.zip
C:\Users\KAROSC\Miniconda\envs\swmmtk\DLLs
C:\Users\KAROSC\Miniconda\envs\swmmtk\lib
C:\Users\KAROSC\Miniconda\envs\swmmtk
C:\Users\KAROSC\Miniconda\envs\swmmtk\lib\site-packages

karosc · 2021-07-20T15:54:45Z

Another note, I can fix the windows issue by copying all dlls from site-packages\swmm_toolkit to site-packages\swmm\toolkit. So it definitely feels like a path issue....

michaeltryby · 2021-07-20T15:55:11Z

I did the Windows dev work using cmd prompt. So I suppose it's possible this is a PowerShell related issue. Could you humor me by trying a windows command prompt?

karosc · 2021-07-20T15:58:07Z

Still same error.

(base) C:\Users\KAROSC>conda create -n swmmtk_cmd python
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.10.1
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: C:\Users\KAROSC\Miniconda\envs\swmmtk_cmd

  added / updated specs:
    - python


The following NEW packages will be INSTALLED:

  ca-certificates    conda-forge/win-64::ca-certificates-2021.5.30-h5b45459_0
  certifi            conda-forge/win-64::certifi-2021.5.30-py39hcbf5309_0
  openssl            conda-forge/win-64::openssl-1.1.1k-h8ffe710_0
  pip                conda-forge/noarch::pip-21.1.3-pyhd8ed1ab_0
  python             conda-forge/win-64::python-3.9.6-h7840368_1_cpython
  python_abi         conda-forge/win-64::python_abi-3.9-2_cp39
  setuptools         conda-forge/win-64::setuptools-49.6.0-py39hcbf5309_3
  sqlite             conda-forge/win-64::sqlite-3.36.0-h8ffe710_0
  tzdata             conda-forge/noarch::tzdata-2021a-he74cb21_1
  ucrt               conda-forge/win-64::ucrt-10.0.20348.0-h57928b3_0
  vc                 conda-forge/win-64::vc-14.2-hb210afc_5
  vs2015_runtime     conda-forge/win-64::vs2015_runtime-14.29.30037-h902a5da_5
  wheel              conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0
  wincertstore       conda-forge/win-64::wincertstore-0.2-py39hcbf5309_1006


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate swmmtk_cmd
#
# To deactivate an active environment, use
#
#     $ conda deactivate


(base) C:\Users\KAROSC>conda activate swmmtk_cmd

(swmmtk_cmd) C:\Users\KAROSC>python -m pip install swmm-toolkit
Collecting swmm-toolkit
  Using cached swmm_toolkit-0.8.2-cp39-cp39-win_amd64.whl (905 kB)
Collecting aenum==2.2.6
  Using cached aenum-2.2.6-py3-none-any.whl (43 kB)
Installing collected packages: aenum, swmm-toolkit
Successfully installed aenum-2.2.6 swmm-toolkit-0.8.2

(swmmtk_cmd) C:\Users\KAROSC>python
Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:37:25) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from swmm.toolkit import output, shared_enum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\KAROSC\Miniconda\envs\swmmtk_cmd\lib\site-packages\swmm\toolkit\output.py", line 18, in <module>
    from . import _output
ImportError: DLL load failed while importing _output: The specified module could not be found.
>>>

michaeltryby · 2021-07-20T15:59:41Z

Well the swmm\toolkit_init_.py file should add swmm_toolkit to the path automatically.This doesn't appear to be happening on your machine for some reason. We have experienced init.py not getting called when the conda environment is not activated properly. I haven't used conda in powershell. What command do you use to activate an conda env in powershell?

michaeltryby · 2021-07-20T16:09:01Z

Ok. I am able to recreate the problem. It's related to python3.9. I will investigate.

karosc · 2021-07-20T16:10:41Z

Ok I think I may have found the cause. I changed the end of the toolkit init.py to include a print call:

# Adds directory containing swmm libraries to path
if platform.system() == "Windows":
    libdir = os.path.join(os.path.dirname(__file__), "../../swmm_toolkit")
    print(libdir)
    
    if hasattr(os, 'add_dll_directory'):
        os.add_dll_directory(libdir)
    else:
        os.environ["PATH"] = libdir + ";" + os.environ["PATH"]

printing libdir yields:

C:\Users\KAROSC\Miniconda\envs\swmmtk\lib\site-packages\swmm\toolkit\../../swmm_toolkit

that doesn't look quite right...

jennwuu · 2021-07-20T16:15:24Z

looks like it might be caused by os path separator

karosc · 2021-07-20T16:19:17Z

I did some testing and found that commenting out the following in __init__.py fixes the issue:

# Adds directory containing swmm libraries to path
if platform.system() == "Windows":
    libdir = os.path.join(os.path.dirname(__file__), "../../swmm_toolkit")
    print(libdir)
    
    #if hasattr(os, 'add_dll_directory'):
    #   os.add_dll_directory(libdir)
    #else:
    os.environ["PATH"] = libdir + ";" + os.environ["PATH"]

This is very weird, because python docs suggest os.add_dll_directory is the correct way to do this.

michaeltryby · 2021-07-20T16:20:32Z

Nice catch!

michaeltryby · 2021-07-20T16:22:25Z

A quick google search reveals this is a known issue.

karosc · 2021-07-20T16:22:39Z

Ok, even further, this is conda issue. We need to set another environment variable if using conda. If I set CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1, then the whole thing works with the original source code.

Not sure if we want to check if using conda before setting the variable, but the following is a fix

# Adds directory containing swmm libraries to path
if platform.system() == "Windows":
    libdir = os.path.join(os.path.dirname(__file__), "../../swmm_toolkit")
    print(libdir)
    
    if hasattr(os, 'add_dll_directory'):
       os.environ['CONDA_DLL_SEARCH_MODIFICATION_ENABLE']="1"
       os.add_dll_directory(libdir)
       
    else:
        os.environ["PATH"] = libdir + ";" + os.environ["PATH"]

if we want to check for conda, we could do something like this:

import sys

# Adds directory containing swmm libraries to path
if platform.system() == "Windows":
    libdir = os.path.join(os.path.dirname(__file__), "../../swmm_toolkit")
    print(libdir)
    
    if hasattr(os, 'add_dll_directory'):
       if 'conda' in sys.version:
           os.environ['CONDA_DLL_SEARCH_MODIFICATION_ENABLE']="1"
       os.add_dll_directory(libdir)
       
    else:
        os.environ["PATH"] = libdir + ";" + os.environ["PATH"]

michaeltryby · 2021-07-20T16:28:30Z

Awesome! Thanks so much!

jennwuu · 2021-07-20T16:50:11Z

This looks great. Do you want to create a PR to close this issue?

karosc · 2021-07-20T17:03:28Z

on it!

jennwuu · 2021-07-20T20:59:16Z

Addressed in #80

karosc mentioned this issue Jul 20, 2021

fix to get add_dll_directory working on conda-built python distributions #80

Merged

jennwuu closed this as completed Jul 20, 2021

This was referenced Sep 15, 2021

Pyswmm simulation error: ImportError: DLL load failed while importing _solver: %1 is not a valid Win32 application. pyswmm/pyswmm#296

Closed

ImportError: DLL load failed while importing _solver: The specified module could not be found. pyswmm/pyswmm#298

Closed

Trouble with swmm-toolkit install #79

Trouble with swmm-toolkit install #79

Comments

karosc commented Jul 20, 2021

Windows

Linux

michaeltryby commented Jul 20, 2021

karosc commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

michaeltryby commented Jul 20, 2021 • edited

karosc commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

jennwuu commented Jul 20, 2021

karosc commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

karosc commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

karosc commented Jul 20, 2021

michaeltryby commented Jul 20, 2021 • edited

karosc commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

jennwuu commented Jul 20, 2021 • edited

karosc commented Jul 20, 2021

jennwuu commented Jul 20, 2021 • edited

karosc commented Jul 20, 2021

Linux output

Windows output

karosc commented Jul 20, 2021

michaeltryby commented Jul 20, 2021 • edited

karosc commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

karosc commented Jul 20, 2021

jennwuu commented Jul 20, 2021

karosc commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

michaeltryby commented Jul 20, 2021

karosc commented Jul 20, 2021 • edited

michaeltryby commented Jul 20, 2021

jennwuu commented Jul 20, 2021

karosc commented Jul 20, 2021

jennwuu commented Jul 20, 2021

michaeltryby commented Jul 20, 2021 •

edited

michaeltryby commented Jul 20, 2021 •

edited

jennwuu commented Jul 20, 2021 •

edited

jennwuu commented Jul 20, 2021 •

edited

michaeltryby commented Jul 20, 2021 •

edited

karosc commented Jul 20, 2021 •

edited