Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.7 & 3.11 support #35

Merged
merged 3 commits into from
Feb 8, 2023
Merged

Python 3.7 & 3.11 support #35

merged 3 commits into from
Feb 8, 2023

Conversation

val-ismaili
Copy link
Contributor

@val-ismaili val-ismaili commented Jan 19, 2023

Closes #32

Based on @mfitz issue here, these changes to the requirements.txt seem to enable osmox to work in both a Python 3.7 and 3.11 in virtual environments on my machine. I've had seperate issues relating to me running this on an M1, so would be useful to see if others are able to run this ok with both python versions.

I originally just left the option as >= for both packages as below and this worked

diff requirements.txt requirements-3.11.txt
4c4
< osmium==3.1.0
---
> osmium>=3.4.0
6c6
< pyproj==3.1.0
---
> pyproj>=3.1.0

Assuming that it would be better to pin the osmium & pyproj to specific versions (?) I've found that osmium==3.4.0 is the crossover for support on both so I've changed it to that for now. The pyproj seems to need to stay with this >= flexibility though.

@val-ismaili val-ismaili linked an issue Jan 19, 2023 that may be closed by this pull request
Copy link
Contributor

@mfitz mfitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes and explanation seem fine, but I'm guessing you did a bunch of verification of OSMOX with these modified dependencies in both a 3.7 and a 3.11 environment, and I need to see the details and outcome of that. With a change like this, the assurance that we've not broken anything is 95% of what I'm looking for.

Can you describe what you did and how? Include copy/pasted shell sessions, or screenshots, that kind of thing. Maybe you set up a clean virtual env for each Python version, installed with the new dependencies, then ran all the tests successfully? That would be a great thing to see. Showing me exactly what you did and saw beats telling me.

As far as the question about pinning to a specific version versus using looser version constraints, we would generally pin everything for an application, (because it's fair to assume that it will be running in a deterministic environment - think a dedicated virtual env), but take a looser approach with a library (because our library will be installed into the user's environment alongside a bunch of other libraries that we know nothing about, making it a non-deterministic environment and thus more prone to "dependency hell" type version clashes). You can see that with the geopandas>=0.9.0 dependency.

The downside is that too loose a set of version constraints can make pip install run unusably slowly as it investigates every permutation of versions. In this case, I don't think it matters whether we pin Osmium or not; if we can get away with >=, let's do that.

Finally, I've added a closes comment to the PR. When your PR addresses a particular GitHub issue, you can link the two so that the issue will be closed automatically when the PR is merged.

@val-ismaili
Copy link
Contributor Author

val-ismaili commented Jan 20, 2023

Thanks Michael - that makes sense about the application vs. library version constraints. I've unpinned osmium now to osmium>=3.4.0.

Steps were:

  • Build two clean python virtual environments
    (osmox-3.7) val.ismaili@LDNML6TDXNNY osmox % python --version Python 3.7.13
    (osmox-3.11) val.ismaili@LDNML6TDXNNY osmox % python --version Python 3.11.0rc2

  • Install as per readme steps

(osmox-3.7) val.ismaili@LDNML6TDXNNY osmox % pip install -e .
Obtaining file:///Users/val.ismaili/CML/cml-osmox/osmox
  Preparing metadata (setup.py) ... done

...

Installing collected packages: pytz, charset-normalizer, zipp, urllib3, typing-extensions, toml, six, Shapely, pyflakes, pycodestyle, py, packaging, numpy, mccabe, iniconfig, idna, coverage, Click, certifi, attrs, Rtree, requests, python-dateutil, pyproj, munch, importlib-metadata, cligj, click-plugins, pluggy, pandas, osmium, flake8, fiona, pytest, geopandas, pytest-mock, pytest-cov, osmox
  Running setup.py develop for osmox
Successfully installed Click-7.1.2 Rtree-1.0.1 Shapely-1.8.0 attrs-22.2.0 certifi-2022.12.7 charset-normalizer-3.0.1 click-plugins-1.1.1 cligj-0.7.2 coverage-7.0.5 fiona-1.8.22 flake8-5.0.4 geopandas-0.10.2 idna-3.4 importlib-metadata-4.2.0 iniconfig-2.0.0 mccabe-0.7.0 munch-2.5.0 numpy-1.21.6 osmium-3.5.0 osmox-0.0.0 packaging-23.0 pandas-1.3.5 pluggy-0.13.1 py-1.11.0 pycodestyle-2.9.1 pyflakes-2.5.0 pyproj-3.2.1 pytest-6.2.4 pytest-cov-2.8.1 pytest-mock-3.1.0 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.28.2 six-1.16.0 toml-0.10.2 typing-extensions-4.4.0 urllib3-1.26.14 zipp-3.11.0
(osmox-3.11) val.ismaili@LDNML6TDXNNY osmox % pip install -e .
Obtaining file:///Users/val.ismaili/CML/cml-osmox/osmox
  Preparing metadata (setup.py) ... done
...
Building wheels for collected packages: Shapely, osmium, fiona
  Building wheel for Shapely (setup.py) ... done
  Created wheel for Shapely: filename=Shapely-1.8.0-py3-none-any.whl size=177363 sha256=7eed0a585951b5caa67e0888323a4922884552a96d8726d595fdbbb768ab0cc0
  Stored in directory: /Users/val.ismaili/Library/Caches/pip/wheels/33/75/31/1a0a0b89ddeac210bd7de8b5b4d02981888ed177049e6f53a1
  Building wheel for osmium (setup.py) ... done
  Created wheel for osmium: filename=osmium-3.5.0-cp311-cp311-macosx_13_0_arm64.whl size=917724 sha256=6ba1fb56252929b6896fdf6ac411829cfd58c6ab56ae5fd647f1c04183f26ae5
  Stored in directory: /Users/val.ismaili/Library/Caches/pip/wheels/52/a7/83/22e058d33e2bd5bfe9ec04ca5b00699a39703b1560ab621e51
  Building wheel for fiona (setup.py) ... done
  Created wheel for fiona: filename=Fiona-1.8.22-cp311-cp311-macosx_13_0_arm64.whl size=566494 sha256=d7cfccdb380293d17c8e657dcc406bf13be05ad7386db57b5b333132cccf86db
  Stored in directory: /Users/val.ismaili/Library/Caches/pip/wheels/01/e4/f8/401e10af41c62ea487a9d372fe2b5cc6e4c77417bedff20fbb
Successfully built Shapely osmium fiona
Installing collected packages: pytz, charset-normalizer, urllib3, toml, six, Shapely, Rtree, pyflakes, pycodestyle, py, pluggy, packaging, numpy, mccabe, iniconfig, idna, coverage, Click, certifi, attrs, requests, python-dateutil, pytest, pyproj, munch, flake8, cligj, click-plugins, pytest-mock, pytest-cov, pandas, osmium, fiona, geopandas, osmox
  Running setup.py develop for osmox
Successfully installed Click-7.1.2 Rtree-1.0.1 Shapely-1.8.0 attrs-22.2.0 certifi-2022.12.7 charset-normalizer-3.0.1 click-plugins-1.1.1 cligj-0.7.2 coverage-7.0.5 fiona-1.8.22 flake8-5.0.4 geopandas-0.12.2 idna-3.4 iniconfig-2.0.0 mccabe-0.7.0 munch-2.5.0 numpy-1.24.1 osmium-3.5.0 osmox-0.0.0 packaging-23.0 pandas-1.5.3 pluggy-0.13.1 py-1.11.0 pycodestyle-2.9.1 pyflakes-2.5.0 pyproj-3.4.1 pytest-6.2.4 pytest-cov-2.8.1 pytest-mock-3.1.0 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.28.2 six-1.16.0 toml-0.10.2 urllib3-1.26.14
  • Run tests. 3.11 environment passes quicker and without any warnings
(osmox-3.7) val.ismaili@LDNML6TDXNNY osmox % python -m pytest
========================================= test session starts ==========================================
platform darwin -- Python 3.7.13, pytest-6.2.4, py-1.11.0, pluggy-0.13.1
rootdir: /Users/val.ismaili/CML/cml-osmox/osmox
plugins: mock-3.1.0, cov-2.8.1
collected 48 items                                                                                     

tests/test_activityhandler.py ...........                                                        [ 22%]
tests/test_autotree.py .....                                                                     [ 33%]
tests/test_config.py ..........                                                                  [ 54%]
tests/test_helpers.py ......................                                                     [100%]

=========================================== warnings summary ===========================================
../osmox-3.7/lib/python3.7/site-packages/geopandas/_compat.py:18
  /Users/val.ismaili/CML/cml-osmox/osmox-3.7/lib/python3.7/site-packages/geopandas/_compat.py:18: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_10 = str(pd.__version__) >= LooseVersion("1.0.0")

...

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=================================== 48 passed, 18 warnings in 11.63s ===================================

(osmox-3.11) val.ismaili@LDNML6TDXNNY osmox % python -m pytest
========================================= test session starts ==========================================
platform darwin -- Python 3.11.0rc2, pytest-6.2.4, py-1.11.0, pluggy-0.13.1
rootdir: /Users/val.ismaili/CML/cml-osmox/osmox
plugins: mock-3.1.0, cov-2.8.1
collected 48 items                                                                                     

tests/test_activityhandler.py ...........                                                        [ 22%]
tests/test_autotree.py .....                                                                     [ 33%]
tests/test_config.py ..........                                                                  [ 54%]
tests/test_helpers.py ......................                                                     [100%]

========================================== 48 passed in 9.73s ==========================================

  • Finally did a manual check by running the Isle of Man example. Didn't notice any speed difference here (don't have a time counter here - something to add maybe). Outputs are as expected and are identical.

3.7
Screenshot 2023-01-20 at 09 35 50

3.11
Screenshot 2023-01-20 at 09 36 16

@mfitz mfitz self-requested a review January 20, 2023 12:16
@mfitz
Copy link
Contributor

mfitz commented Feb 6, 2023

Are you deliberately not merging this @val-ismaili , or did you forget about it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Python 3.11
2 participants