Cannot install pyrosm

[bramson]

I've been trying to install this pyrosm on Windows10 and Python 3.6 because it looks potentially useful (although inadequately documented... like, which elements are included in the "walking" network?) for isolating parts of OSM data within a polygon before converting the remaining "highways" into a NetworkX graph.

However, I tried to use pip installl pyrosm, but I cannot install the package because I get the infamous "A GDAL API version must be specified" error.

` Collecting fiona
Using cached Fiona-1.8.17.tar.gz (1.3 MB)
ERROR: Command errored out with exit status 1:
command: 'c:\winpython\python-3.6.5.amd64\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = ...
Complete output (1 lines):
A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.`

The thing is, I already have the newest version of GDAL and Fiona 1.8.17 installed, and I already added GDAL to my path variables, and I can already use geopandas, fiona, shapely, rtree, etc. normally.

I couldn't find a list of requirements and/or dependencies in the documentation aside from Py3.6, so maybe the real problem is something else.

I hope to be able to use this package, and so I appreciate any assistance in properly installing it.

[HTenkanen]

Hi @bramson,

the recommended way to install pyrosm is by using conda as mentioned in the docs:
https://pyrosm.readthedocs.io/en/latest/installation.html

If you have not use conda before, I very much recommend start using it. There is more information about this e.g. in here:
https://autogis-site.readthedocs.io/en/latest/course-info/installing-miniconda.html

The issue you face is not actually related to Pyrosm as such, but installing the dependencies (geopandas) which rely on external C-libraries (which do not install well on Windows unfortunately).

[bramson]

My python installation is WinPython, rather than Anaconda, and is not a(as far as I know) compatible with conda installation.

So, In order to install GeoPandas I had to do things manually with whl files, but it's not that bad once you know the sequence.
I believe I have already installed all the packages of the same versions that pyrosm depends on, but for some reason the installation of pyrosm with pip on Windows doesn't recognize my previous installations, and tries to collect and reinstall them, which it cannot do. But it shouldn't be trying to do that (GeoPandas installation doesn't try to reinstall GDAL after my manual installation of GDAL).

I don't really know much about dependencies, installation, or pip, but what I though would work (installing all the dependencies, then installing the package) is not working in this case.

Do you have any ideas for a work around?

[HTenkanen]

@bramson Is there a specific reason why you need to stick with WinPython instead of using conda which is nowadays more or less the "industry standard" dealing with package installations for Python ecosystem?

What comes to your installation issue under Windows, I cannot say much with this information, as I would need to understand what versions of Geopandas, Fiona, etc. packages you have installed. Most likely there is a dependency for newer version of specific package why pyrosm attempts to install fiona etc. again with pip instead of using your already installed versions. But to make your life easier also in the long run, I still highly recommend to start using conda and do all your Python installations with that, because conda was developed exactly to handle tricky installation situations like this. Hence, it will help make your life MUCH easier also with many other packages as well. With conda you do not need to worry about any environment settings or installing packages separately from wheels etc. but everything (just) works with a single command i.e. conda install -c conda-forge pyrosm.

There is a good Twitter thread about these things, and how to solve them in here:
https://twitter.com/ocefpaf/status/1321814088662568965

[dominikbucher]

Hi all! Thanks for the library, very awesome! To (hopefully) shed some light on this issue (because I was struggling with it too - somehow didn't see the conda-forge installation process and tried with pip):

It seems that setup_requires=requirements is set in setup.py. As far as I understood, this tries to install geopandas (with its dependencies fiona etc.) for the setup process, and not just for running it later. I managed to resolve the issue by cloning the repository, replacing the line with setup_requires=['setuptools>=18.0'] and installing with pip install -e ..

In case the setup_requires is not required I'll gladly make a pull request.

I had some other issues due to pygeos geometries not being compatible with geopandas but I think they're not related to the problem at hand but to my geopandas installation.

Cheers

[bramson]

@HTenkanen The only reason I'm sticking with WinPython is that up to now I've only faced minor inconveniences with it...much smaller inconveniences than installing a new python distribution from scratch and making sure all my code still works on it. Obviously I like the idea of installing packages with no effort, but that's what pip is already supposed to do, and usually does. I looked at the twitter thread you pointed to in order to convince myself to make the sideways leap to anaconda, but it immediately descended to technical jargon that I could not follow, so unfortunately I could not understand what the proposed benefits or solutions to problems actually are. So far the only thing I've had to install "manually" is geopandas. And the only thing that's ever failed to install is this package.

@dominikbucher This sounds reasonable to me. Like I said, based on the docs, I already have the appropriate versions of all the required packages installed. But the installation of this package seems to be trying to reinstall them, and it can't via pip. I'm not currently proficient in cloning and altering repositories, so it would be great (at least for me and other non-conda users) to make this change part of a pip-able package.

[HTenkanen]

Hi guys! And thanks @dominikbucher for sharing how you tackled this issue! 👍 Your solution indeed seems like it could resolve this issue with pip, and definitely the goal is to support installation using pip only also in Windows (assuming all the requirements have been installed). So I will test if this trick works, and update the package for the next release. This should then hopefully resolve @bramson's installation issue.

@bramson About conda vs pip/WinPython: it is definitely up to you what package manager you want to use. Just from experience I can say that pip installations in Windows can sometimes be tricky especially with GIS packages that often rely on non-Python libraries (such as GDAL), hence recommended conda.

[bramson]

@HTenkanen Thank you for incorporating this fix in the next version.

If I could go back, then I'd go the Anaconda route, as I've advised new people in the lab to do. But from where I am, it's probably a few days of package version compatibility frustration just to get back to exactly where I am. Actually, if there is a function in conda to read a requirements file and get all those versions of all those packages, then it may not be so bad. I'll look into it. But anyway, as you said, being to install via pip in Windows will only strengthen the package and broaden/accelerate its use.

[HTenkanen]

FYI: @bramson With conda, there is this fileformat called environment.yaml (or .yml which is the same thing) which does same things as requirements.txt (it is formatted a bit differently but should be quite easy). Typically the safest way to do the installations is to use these things called environments that isolate the installations e.g. for different projects that you might have.

To get a quick overview about this, you can take a look some docs that we have written about it here: https://autogis-site.readthedocs.io/en/latest/course-info/create-python-gis-env.html#create-an-environment-from-an-yaml-file

Also to understand how the environment.yml files typically look like, you can take a look e.g. how the environment files for different versions of Python have been written for this pyrosm package, such as this for Python 3.8: 38-conda.yaml

[HTenkanen]

@bramson : I now merged the installation fix to the master, and hopefully it should work now. Could you please test and verify if you can now successfully install pyrosm with pip by running the following command (which installs the dev version directly from Github):
pip install git+https://github.com/HTenkanen/pyrosm.git@master

[bramson]

@HTenkanen : Thank you for your message. I was able to successfully install the dev version with your command.
When I run the command from pyrosm import get_data (in order to work through some examples in the docs), I get:

C:\WinPython\python-3.6.5.amd64\lib\site-packages\geopandas\_compat.py:88: UserWarning: The Shapely GEOS version (3.8.0-CAPI-1.13.1 ) is incompatible with the GEOS version PyGEOS was compiled with (3.8.1-CAPI-1.13.3). Conversions between both will be slow.
  shapely_geos_version, geos_capi_version_string

I don't know if this is the same problem @dominikbucher mentioned, but my geopandas and shapely versions seem to match the pyrosm requirements. I don't know if ignoring this will cause problems (I mean, how slow, and will I ever do that?), or what exactly the fix would be. But it seems to be installed and working, so I'll try it on my real problem on Tuesday and let you know if anything explodes.

[HTenkanen]

@bramson Excellent, now everything seems to work. 👍 The warning that you get actually comes from geopandas (used under the hood in pyrosm). It is nothing dangerous and you can ignore it. Will add a note about this to the documentation as well.

Technical details but: It just warns that in couple of different libraries that are used under the hood of geopandas are using different version of GEOS library. This is something that will eventually be fixed once Shapely library is updated to version 2.0 (should happen sometime next year hopefully).