Environments.yml requires gssha - not available for Windows

[kcpevey]

The env cannot be set up on Windows as-is since the gssha package is only available for Linux and OSX (https://anaconda.org/gbrener/gssha).

There are other gssha-related Windows issues as well. Namely the fact that earthsim is on py3.5, requires gsshapy, and gsshapy requires pangaea which is not available for Windows py3.5. (https://github.com/ContinuumIO/EarthSim/issues/124)

[philippjfr]

I can only suggest disabling those dependencies for now. If these issues can't be addressed in the near future it might make sense to have separate environments for building the full documentation which includes GSSHA examples and one for users to try out the other content.

[dharhas]

@kcpevey Amanda Hines is working a Windows GSSHA conda package. Getting pangaea available for Python3/Windows so that GSSHApy will install is on us (ERDC).

[dharhas]

Actually, now that I think about it no she isn't. She's working on merging some GSSHA branches so we can start building GSSHA packages. Either way the Windows GSSHA package is on us.

[kcpevey]

I just looked through the earthsim dependencies and it looks like the only current GSSHA-related dependency is gsshapy. I suspect the gssha dependency in the yml (https://anaconda.org/gbrener/gssha) might have been from before gsshapy existed? I can't test it this theory on my Windows machine, but maybe someone else will remember or be able to test?

If that is true, then this issue boils down to updating Pangaea (https://anaconda.org/conda-forge/pangaea).

[dharhas]

Ok so this is a casualty of Alan Snow (@snowman2) moving on to greener pastures. He was the primary developer of gsshapy for a while and he added some dependencies to a couple of projects he created. Pangaea is one of his projects. We either need to update gsshapy to drop the pangaea dependency, update pangaea to be windows & python 3 compatible or investigate if pangaea is an optional dependency and update the conda-forge build recipe package to drop pangaea. All three options require one of us investigating, making the changes and then either doing a pull request on Alan's repos or asking him to make one of us a maintainer on those repos.

[kcpevey]

Pangaea is already available for py3 for OSX and Linux. I don't know enough about cross-platform development, but could it be that Pangaea was simply not built for Windows?

I did some testing on my Windows machine. First, I cloned the Pangaea repo and tried to python setup.py install into a py35 env. This failed on wrf-python and mapkit because it was grabbing the dependency eggs from pypi (I guess?). I did conda install for mapkit and wrf-python and those succeeded. Then I retried python setup.py install on Pangaea and that worked. I can import pangaea without errors.

Is that helpful? What else can I check? Should I just try to build the conda package? (Not sure how to do that...)

[dharhas]

I added Win64/Python3 packages for pangaea and gsshapy to the conda-forge channel. @kcpevey can you test that you can create the environment. We are still missing a gssha conda package for windows but that should be a simple workaround, just use the official exe download located at https://gsshawiki.com/GSSHA_Download

[kcpevey]

Just to be clear, the yml needs the following modifications:

1. remove the gssha package
2. change the quest repo location
3. switch from the ioam channel to ioam/label/dev and then I can switch from a pip install of holoviews and geoviews to conda.

Is all that correct?

[dharhas]

Actually which yml file are you looking at?

This one already has the correct quest repo:

https://github.com/pyviz/EarthSim/blob/master/environment.yml

[kcpevey]

Thats the one I'm looking at. I knew quest moved, but stupidly didn't realize the yml was already updated. As for holoviews and geoviews, I thought @philippjfr told me to pull from ioam/label/dev? Or did I misunderstand that?

I guess I also don't understand why holoviews and geoviews are imbedded in the pip command instead of pulling from conda?

[dharhas]

So the pip commands pull directly from the latest github branches. I think now holoviews and geoviews automatically build conda packages every night and put them in the dev channel which is why @philippjfr said you can pull them in from the ioam/label/dev channel.

[philippjfr]

I guess I also don't understand why holoviews and geoviews are imbedded in the pip command instead of pulling from conda?

That's not yet the case as there is still some churn around our packaging/release procedures, which should all be solved in the coming weeks. I don't remember when I originally gave that advice, currently the way to get the latest versions would be to install holoviews from ioam/conda-forge/main channels and geoviews from ioam/label/dev channel, but there will be a geoviews release later today, so ioam should be sufficient for both.

That said I think for development purposes we will probably want to track master (using pip), and separately have semi-regular EarthSim/HoloViews/GeoViews dev releases which pin specific versions and guarantee a bit more stability.

[kcpevey]

Ok. The only change I made to the yml was removing gssha.

First, I'll say that I had multiple connection errors when it was getting timezonefinder. So I'm not sure if that is causing these problems.

At any rate, it did finally download and I got past the preparing transaction. Now my env creation is failing after verifying transaction: failed. I'm getting maybe a hundred CondaVerificationError and ClobberError for a variety of different packages. I tried using --force but that didn't help.

The beginning of a thousands of lines of error:

C:\PROJECTS\ERS\gitHub\earthsim_pyviz\newmaster\EarthSim>conda env create -n ers environment.yml --force
Using Anaconda API: https://api.anaconda.org
Solving environment: done
Preparing transaction: done
Verifying transaction: failed

CondaVerificationError: The package for wheel located at C:\ProgramData\Anaconda3\pkgs\wheel-0.31.0-py35_0
appears to be corrupted. The path 'Scripts/wheel.exe'
specified in the package manifest cannot be found.

CondaVerificationError: The package for pip located at C:\ProgramData\Anaconda3\pkgs\pip-9.0.3-py35_0
appears to be corrupted. The path 'Lib/site-packages/pip/_vendor/distlib/_backport/sysconfig.cfg'
specified in the package manifest cannot be found.

CondaVerificationError: The package for pip located at C:\ProgramData\Anaconda3\pkgs\pip-9.0.3-py35_0
appears to be corrupted. The path 'Lib/site-packages/pip/_vendor/distlib/t32.exe'
specified in the package manifest cannot be found.

CondaVerificationError: The package for pip located at C:\ProgramData\Anaconda3\pkgs\pip-9.0.3-py35_0
appears to be corrupted. The path 'Lib/site-packages/pip/_vendor/distlib/t64.exe'
specified in the package manifest cannot be found.

CondaVerificationError: The package for pip located at C:\ProgramData\Anaconda3\pkgs\pip-9.0.3-py35_0
appears to be corrupted. The path 'Lib/site-packages/pip/_vendor/distlib/w32.exe'
specified in the package manifest cannot be found.

CondaVerificationError: The package for pip located at C:\ProgramData\Anaconda3\pkgs\pip-9.0.3-py35_0
appears to be corrupted. The path 'Lib/site-packages/pip/_vendor/distlib/w64.exe'
specified in the package manifest cannot be found.

CondaVerificationError: The package for pip located at C:\ProgramData\Anaconda3\pkgs\pip-9.0.3-py35_0
appears to be corrupted. The path 'Scripts/pip.exe'
specified in the package manifest cannot be found.

ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::python-3.5.5-1, conda-forge::cligj-0.4.0-py35_0
  path: 'lib/__pycache__/__future__.cpython-35.pyc'


ClobberError: This transaction has incompatible packages due to a shared path.
  packages: conda-forge::python-3.5.5-1, conda-forge::heapdict-1.0.0-py35_0, conda-forge::sip-4.18-py35_1, conda-forge::cligj-0.4.0-py35_0
  path: 'lib/__pycache__/_bootlocale.cpython-35.pyc'

[dharhas]

So I've had this issue on Windows and it is something to do with conda-forge and default channels packages overwriting some common files. I fixed it by change my conda settings to warn and not clobber

conda config --set path-conflict warn

I also had to first create a basic conda config file since I didn't have a user editable one in my path:

conda config --write-default

[kcpevey]

@dharhas I set it warn but I'm getting the same verification errors and now the ClobberErrors are ClobberWarnings.

I noticed that the verification errors are focusing on pip. I've been getting warnings that my pip needs to be upgraded to 10.0.1, but my update is failing with an Attribute Error deep inside the script. I'm thinking the best approach would be to just get rid of pip and reinstall, but I'm not clear on how intrinsic that is to my python installation. If I delete it on my own, will I destroy my python installation?

[dharhas]

conda clean --all should delete any old downloaded packages lying around. You should probably try that first.

[kcpevey]

I did conda clean --all but that didn't change anything.

I decided to delete my pip and wheel directories. My env create downloaded new ones that got me past the VerficiationErrors. I still have a lot of ClobberWarnings. This is the last warning and then the actual failure.

ClobberWarning: Conda was asked to clobber an existing path.
  source path: C:\ProgramData\Anaconda3\pkgs\pip-9.0.3-py35_0\Lib\site-packages\pip\vcs\__pycache__\subversion.cpython-35.pyc
  target path: C:\ProgramData\Anaconda3\envs\ers8\Lib\site-packages\pip\vcs\__pycache__\subversion.cpython-35.pyc



failed
2018-04-23 12:51:57,309 - conda.core.link - ERROR - An error occurred while installing package 'conda-forge::ipykernel-4.8.2-py35_0'.
LinkError: post-link script failed for package conda-forge::ipykernel-4.8.2-py35_0
running your command again with `-v` will provide additional information
location of failed script: C:\ProgramData\Anaconda3\envs\ers8\Scripts\.ipykernel-post-link.bat
==> script messages <==
<None>

Attempting to roll back.

ERROR conda.core.link:_execute(502): An error occurred while installing package 'conda-forge::ipykernel-4.8.2-py35_0'.
LinkError: post-link script failed for package conda-forge::ipykernel-4.8.2-py35_0
running your command again with `-v` will provide additional information
location of failed script: C:\ProgramData\Anaconda3\envs\ers8\Scripts\.ipykernel-post-link.bat
==> script messages <==
<None>

Attempting to roll back.

Rolling back transaction: done

LinkError: post-link script failed for package conda-forge::ipykernel-4.8.2-py35_0
running your command again with `-v` will provide additional information
location of failed script: C:\ProgramData\Anaconda3\envs\ers8\Scripts\.ipykernel-post-link.bat
==> script messages <==
<None>

I'm starting to think that I'm going to have to delete python/anaconda altogether and start over from scratch....

[dharhas]

That might be the simplest. Also unless you use the other stuff that comes with Anaconda, i.e. like the navigator GUI etc, I would just install miniconda.

[kcpevey]

Ok. Will do...

[kcpevey]

Whew. I completely removed python from my machine and reinstalled it. I removed gssha from the yml then used it to create a conda env. I still got ClobberWarnings but everything seems to have installed correctly.

I ran python setup.py install to install earthsim itself. And then just tested to ensure that import earthsim didn't throw errors. No problems, and everything looks good!

[kcpevey]

Just for the record... I think my issues had something to do with pip not playing nice with conda.

Last week pip started warning me to update from 9.0.3 to 10.0.1. I went ahead and updated pip outside of conda (10.0 is not available via conda yet). That pip --upgrade process failed somewhere in the middle. I think the combination of pip failing on the upgrade and/or upgrading pip outside of conda is what ended up breaking my root python (i.e. all of it).

[dharhas]

ok as a rule of thumb never mix conda with non conda stuff. Pip can be installed inside conda packages but using a system pip can cause issues as you saw. There are some breaking changes in pip 10 that are causing issues all over the python ecosystem which might be why conda is not upgrading pip yet.

[dharhas]

Lets leave this issue open until we have the windows gssha package available.

[kcpevey]

I'm finally getting around to testing this more extensively. import earthsim works, but some of the other dependencies are broken, starting with datashader. @philippjfr Is this a versioning issue?:

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-2-a634c8ae9cc8> in <module>()
     19 import earthsim
     20 import numpy as np
---> 21 import datashader as ds
     22 import holoviews as hv
     23 import geoviews as gv

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\datashader\__init__.py in <module>()
      8                          mean, std, var, count_cat, summary)
      9 from .glyphs import Point                                # noqa (API import)
---> 10 from .pipeline import Pipeline                           # noqa (API import)
     11 from . import transfer_functions as tf                   # noqa (API import)
     12 

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\datashader\pipeline.py in <module>()
      3 from toolz import identity
      4 
----> 5 from . import transfer_functions as tf
      6 from . import reductions
      7 from . import core

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\datashader\transfer_functions.py in <module>()
     12 
     13 from .colors import rgb, Sets1to3
---> 14 from .composite import composite_op_lookup, over
     15 from .utils import ngjit, orient_array
     16 

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\datashader\composite.py in <module>()
      8 
      9 
---> 10 @nb.jit('(uint32,)', nopython=True, nogil=True, cache=True)
     11 def extract_scaled(x):
     12     """Extract components as float64 values in [0.0, 1.0]"""

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\numba\decorators.py in wrapper(func)
    190             with typeinfer.register_dispatcher(disp):
    191                 for sig in sigs:
--> 192                     disp.compile(sig)
    193                 disp.disable_compile()
    194         return disp

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\numba\dispatcher.py in compile(self, sig)
    566 
    567                 # Try to load from disk cache
--> 568                 cres = self._cache.load_overload(sig, self.targetctx)
    569                 if cres is not None:
    570                     self._cache_hits[sig] += 1

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\numba\caching.py in load_overload(self, sig, target_context)
    644         """
    645         # Refresh the context to ensure it is initialized
--> 646         target_context.refresh()
    647         with self._guard_against_spurious_io_errors():
    648             return self._load_overload(sig, target_context)

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\numba\targets\base.py in refresh(self)
    266         # Also refresh typing context, since @overload declarations can
    267         # affect it.
--> 268         self.typing_context.refresh()
    269 
    270     def load_additional_registries(self):

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\numba\typing\context.py in refresh(self)
    140         Useful for third-party extensions.
    141         """
--> 142         self.load_additional_registries()
    143         # Some extensions may have augmented the builtin registry
    144         self._load_builtins()

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\numba\typing\context.py in load_additional_registries(self)
    596 
    597     def load_additional_registries(self):
--> 598         from . import (cffi_utils, cmathdecl, enumdecl, listdecl, mathdecl,
    599                        npydecl, operatordecl, randomdecl, setdecl)
    600         self.install_registry(cffi_utils.registry)

C:\ProgramData\Anaconda3\envs\ers\lib\site-packages\numba\typing\cffi_utils.py in <module>()
     15 try:
     16     import cffi
---> 17     ffi = cffi.FFI()
     18 except ImportError:
     19     ffi = None

C:\ProgramData\Miniconda3\Lib\site-packages\cffi\api.py in __init__(self, backend)
     52                     raise Exception("Version mismatch: this is the 'cffi' package version %s, located in %r.  When we import the top-level '_cffi_backend' extension module, we get version %s, located in %r.  The two versions should be equal; check your installation." % (
     53                         __version__, __file__,
---> 54                         backend.__version__, backend.__file__))
     55                 else:
     56                     # PyPy

Exception: Version mismatch: this is the 'cffi' package version 1.11.4, located in 'C:\\ProgramData\\Miniconda3\\Lib\\site-packages\\cffi\\api.py'.  When we import the top-level '_cffi_backend' extension module, we get version 1.11.5, located in 'C:\\ProgramData\\Anaconda3\\envs\\ers\\lib\\site-packages\\_cffi_backend.cp35-win_amd64.pyd'.  The two versions should be equal; check your installation.

[dharhas]

I wonder if this is due to the new environment stacking feature in conda. Try conda deactivate twice to deactivate both the 'ers' and 'base' environment and then only activate the ers env with conda activate ers

[kcpevey]

the conda deactivate didn't work. I don't think it was a problem with stacking envs.

It appeared to have been a problem with my paths (Windows). I had my root conda directory (and the libraries, site-packages, etc) in BOTH my system path AND a PYTHONPATH environment variable. It was the SAME set of directories so I don't understand why that caused problems. I deleted everything out of PYTHONPATH and now my workflow notebooks are running smoothly.

[dharhas]

Yeah conda recommends not setting PYTHONPATH or PYTHONHOME -> https://conda.io/docs/user-guide/troubleshooting.html

[dharhas]

gssha 7.12 should now be available on all three platforms (OSX, Win, Linux) as a conda package in the erdc channel. I installed and checked that it ran and gave me usage instructions. I haven't tested by run any models, so please let me know if you hit issues.

[ceball]

I came here to post that I just packaged the gssha 7.0 binary for win-64, and that it seems to work but will be testing it properly tomorrow in the EarthSim notebooks. But now I'll test yours instead :)

[ceball]

On a windows 7 machine, I did conda install -c erdc gssha into a fresh python 3.6 environment. It installed fine, but when I type gssha I get: The program can't start because VCOMP90.DLL is missing from your computer. Try reinstalling the program to fix this problem.

[dharhas]

I built on win10. Did not try win7

…

On Tue, Apr 24, 2018, 4:35 PM Chris B ***@***.***> wrote: On a windows 7 machine, I did conda install -c erdc gssha into a fresh python 3.6 environment. It installed fine, but when I type gssha I get: The program can't start because VCOMP90.DLL is missing from your computer. Try reinstalling the program to fix this problem. [image: gssha] <https://user-images.githubusercontent.com/1929/39215449-c309728e-480f-11e8-9ac8-5ef754cfcb74.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAiQlW4mDyMw9uheyxzEap1roth5XPtiks5tr5qhgaJpZM4SCXZc> .

[ceball]

I'm stopping for today, but can test out anything new tomorrow (on win7 and win10). But if you don't get chance, it won't hold me up as I can use the package I made (and we can replace it later on with yours).

[dharhas]

Ok l think it is because I did not make openmp a runtime dependency. Can you install openmp from conda forge and see if that fixes it. I'll try and fix the packagre later. I'm out and about right now.

[dharhas]

Ok its something to do the vs runtime. The package I built is dependent on the vs2008_runtime but the vs2015_runtime is being installed. Working on a fix.

[dharhas]

@ceball @kcpevey Try now. you should be installing

#Name                    Version                   Build  Channel
gssha                     7.12                h0510ff6_25    erdc

[ceball]

When I installed, python was downgraded from 3.6 to 2.7:

>conda install -y -c erdc gssha
Solving environment: done

## Package Plan ##

  environment location: [...]

  added / updated specs:
    - gssha


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    vc-9                       |       h7299396_1           3 KB
    gssha-7.12                 |      h0510ff6_25         670 KB  erdc
    certifi-2018.4.16          |           py27_0         143 KB
    ------------------------------------------------------------
                                           Total:         815 KB

The following NEW packages will be INSTALLED:

    gssha:          7.12-h0510ff6_25        erdc
    vs2008_runtime: 9.00.30729.1-hfaea7d5_1

The following packages will be UPDATED:

    certifi:        2018.4.16-py36_0             --> 2018.4.16-py27_0
    pip:            9.0.3-py36_0                 --> 9.0.3-py27_0
    setuptools:     39.0.1-py36_0                --> 39.0.1-py27_0
    wheel:          0.31.0-py36_0                --> 0.31.0-py27_0
    wincertstore:   0.2-py36h7fe50ca_0           --> 0.2-py27hf04cefb_0

The following packages will be DOWNGRADED:

    python:         3.6.5-h0c2934d_0             --> 2.7.14-h4a10d90_31
    vc:             14-h0510ff6_3                --> 9-h7299396_1
[...]

But it runs :)

>conda list gssha
# packages in environment at [...]:
#
# Name                    Version                   Build  Channel
gssha                     7.12                h0510ff6_25    erdc

>gssha
Parallel version of GSSHA (7.12) with SCE (OpenMP), 4 total threads.

Usage: gssha [-option] [inputfile]
[...]

I can try to inspect the recipe later (presumably it's available?). But just guessing, if you depended on vc=9 at run time, I think that would tie it to python 2.7.

[dharhas]

Boo. So I can't get it to compile with VS2015. I think there is some C++ constructs that are not compatible... Its a pure C++ code with no python dependencies so I was hoping I could get away with vc9

Earlier version. I was accidentally setting the meta.yaml to vc14 but it was compiling with vc9 because it was in the path and it built a package that was usable as long as you installed vs2008_runtime into your conda environment.

For now lets stick with your repackaged version, I don't have cycles to spare on this right now...

[dharhas]

@ceball, @kcpevey

Try again. I've uploaded a new build that is compiled with vc9 but only dependent on the vs2008_runtime. This allows me to install it in a python 3 environment. The following works for me now:

conda install -c erdc gssha python=3

You should be getting build 26:

>conda list gssha
# packages in environment at C:\Users\IEUser\Miniconda3:
#
# Name                    Version                   Build  Channel
gssha                     7.12                         26    erdc

[kcpevey]

@dharhas It found the correct package and installed without errors. The gssha package (with correct version and build) is showing up in my conda list. I wrote a little test script that just imports gssha, but its failing to import.

$ conda install -c erdc gssha python=3
Solving environment: ...working... done

## Package Plan ##
  environment location: C:\ProgramData\Anaconda3\envs\ers_dev
  added / updated specs:
    - gssha
    - python=3

The following packages will be downloaded:
    package                    |            build
    ---------------------------|-----------------
    vs2008_runtime-9.0.30729.6161|                0         1.0 MB  conda-forge
    gssha-7.12                 |               26         670 KB  erdc
    ------------------------------------------------------------
                                           Total:         1.7 MB

The following NEW packages will be INSTALLED:
    gssha:          7.12-26          erdc
    vs2008_runtime: 9.0.30729.6161-0 conda-forge

Proceed ([y]/n)? y

vs2008_runtime 9.0.30729.6161########## | 100%
gssha 7.12########## | 100%
Downloading and Extracting Packages
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

(ers_dev)
RDCHLKCP@CHLKCP-WN-S8470 MINGW64 /c/PROJECTS/ERS/sheep
$ python test.py
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import gssha
ImportError: No module named 'gssha'

[dharhas]

@kcpevey you also need to install gsshapy.

gssha only provides the GSSHA executable, gsshapy is the python wrapper. They are not currently explicitly dependent on each other. So you would need to do:

conda install -c conda-forge -c erdc gssha gsshapy python=3

[kcpevey]

@dharhas gsshapy is already installed
gsshapy 2.1.1 py35_2 conda-forge

I also ran conda update gsshapy but its up to date.

My env is py 3.5 for earthsim, is that ok?

[kcpevey]

OHHH I think I see what you're saying... You can't import gssha

I'll find the example notebooks to test.

[dharhas]

yes. you would import gsshapy, you will have to look at the GSSHA workflow example for more details. Alan and Scott are more familiar with gsshapy than I am.

@kcpevey @ceball can we close this ultra long issue now and open a new issues if we have further problems.

[kcpevey]

@dharhas gsshapy imports without issues. Completely agree with closing.

[ceball]

I'll close this, although I've seen longer issues ;) If I have trouble with the example notebooks, I'll open a new issue.

Meanwhile, @kcpevey, I have an unrelated question. Are you using the Anaconda Prompt on windows, or something else (e.g. git bash)? Or do you have bash/similar installed some other way (e.g. via conda, or as part of windows 10 maybe)? I noticed when you pasted your session, it looked like some kind of bash.

[kcpevey]

@ceball I usually use gitBash since its generally my favorite. However, I'll say that when I use gitBash to add/remove packages from a conda env, it does the task successfully, but ends up breaking the path to my conda tools. So if I add a package to an env via conda install, then immediately try another conda install it can't find conda because it has stripped all of the \ out of the path to the conda executable. I have to reopen gitBash and activate my env again to add another package (just an example, I know I can add multiple at one time). Does that make sense?

Because of that, I've added the unix commands to the native Windows Command Prompt and occasionally I'll use that when I get annoyed with gitBash.

All that to say, I'm fairly certain that all the above discussions were run with gitBash.