Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

gdal: Wrap executables in environment containing numpy #285391

Merged
merged 1 commit into from Feb 14, 2024

Conversation

mpickering
Copy link
Contributor

Since gdal 3.8.1 there is a numpy dependency on the python library it seems. This resulted in broken python scrips such as gdal2tiles.py

[nix-shell:~/map-scraper/shake-rg]$ gdal2tiles.py --help
ModuleNotFoundError: No module named 'numpy'

Therefore it's important to wrap the executables in an environment which contains numpy.

Description of changes

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 馃憤 reaction to pull requests you find important.

@imincik
Copy link
Contributor

imincik commented Feb 1, 2024

I confirm, I can reproduce the problem.

@imincik
Copy link
Contributor

imincik commented Feb 1, 2024

I am wondering how gdal2tiles tests are passing if we are missing some dependency.

@imincik
Copy link
Contributor

imincik commented Feb 1, 2024

We are adding packages on PYTHONPATH for tests -

export PYTHONPATH="$out/${python3.sitePackages}:$PYTHONPATH"
. This might explain the fact that tests are passing.

Content of that PYTHONPATH is following (numpy is at the end):

/nix/store/dyl7f7v3cfi7j9d4z42pn2ss0bgw0r33-python3.11-setuptools-69.0.2/lib/python3.11/site-packages:/nix/store/w4fvvhkzb0ssv0fw5j34pw09f0qw84w8-python3-3.11.7/lib/python3.11/site-packages:/nix/store/yjwcmiq2nv1a36h9g02lmj5zpyrdcwks-python3.11-pytest-7.4.3/lib/python3.11/site-packages:/nix/store/3pc2r6g5sikxlz2nn7lpidfn8b6zvvs3-python3.11-iniconfig-2.0.0/lib/python3.11/site-packages:/nix/store/2f44311bksw58c0cnzgsrvh9ha3gvv7g-python3.11-packaging-23.2/lib/python3.11/site-packages:/nix/store/h0bkymwqrc7zdf0n1jd1asw2mqrqy5kp-python3.11-pluggy-1.3.0/lib/python3.11/site-packages:/nix/store/4x4vyj14siki84i7hfpivv1iapw18cps-python3.11-pytest-env-1.1.3/lib/python3.11/site-packages:/nix/store/app3589pn4wdk1dbyji4lzxbfpm5i3ng-python3.11-filelock-3.13.1/lib/python3.11/site-packages:/nix/store/21w8irygqj3q4rh3n8iip9n8zd3lw8sw-python3.11-lxml-4.9.4/lib/python3.11/site-packages:/nix/store/vglknnlh6wjnpy9zmdmp8lphv0515339-python3.11-numpy-1.26.2/lib/python3.11/site-packages

@imincik
Copy link
Contributor

imincik commented Feb 1, 2024

Would you please avoid starting change description with capital letter ( change gdal: Wrap executables ... to gdal: wrap executables). See https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#commit-conventions .

Thank you very much for your contribution.

@mpickering
Copy link
Contributor Author

@imincik I have updated with your suggested changes, but for the record I find this style of nitpicking review very annoying and I think maintainers should be more lenient when accepting patches from casual contributors.

@imincik
Copy link
Contributor

imincik commented Feb 1, 2024

I find this style of nitpicking review very annoying and I think maintainers should be more lenient when accepting patches from casual contributors.

I very much appreciate your contribution. I am not happy with nitpicking either and I think this was the most I ever made. Sorry again. I hope we can have some automatic linter doing it for use automatically.

@ofborg ofborg bot requested a review from imincik February 1, 2024 11:13
@imincik
Copy link
Contributor

imincik commented Feb 1, 2024

Result of nixpkgs-review pr 285391 run on x86_64-linux 1

9 packages marked as broken and skipped:
  • python311Packages.worldengine
  • python311Packages.worldengine.dist
  • python312Packages.fiona
  • python312Packages.fiona.dist
  • python312Packages.geopandas
  • python312Packages.geopandas.dist
  • python312Packages.worldengine
  • python312Packages.worldengine.dist
  • t-rex
3 packages failed to build:
  • apacheHttpdPackages.mod_tile
  • cloudcompare
  • qgis-ltr
77 packages built:
  • entwine
  • flatcam
  • gdal (python311Packages.gdal)
  • gdalMinimal
  • gmt
  • gplates
  • grass
  • mapcache
  • mapnik
  • mapproxy
  • mapproxy.dist
  • mapserver
  • merkaartor
  • mysql-workbench
  • octavePackages.mapping
  • openorienteering-mapper
  • paraview
  • pdal
  • perl536Packages.Tirex
  • perl536Packages.Tirex.devdoc
  • perl538Packages.Tirex
  • perl538Packages.Tirex.devdoc
  • postgresql12JitPackages.postgis
  • postgresql12JitPackages.postgis.doc
  • postgresql12Packages.postgis
  • postgresql12Packages.postgis.doc
  • postgresql13JitPackages.postgis
  • postgresql13JitPackages.postgis.doc
  • postgresql13Packages.postgis
  • postgresql13Packages.postgis.doc
  • postgresql14JitPackages.postgis
  • postgresql14JitPackages.postgis.doc
  • postgresql14Packages.postgis
  • postgresql14Packages.postgis.doc
  • postgresqlJitPackages.postgis (postgresql15JitPackages.postgis)
  • postgresqlJitPackages.postgis.doc (postgresql15JitPackages.postgis.doc)
  • postgresql15Packages.postgis
  • postgresql15Packages.postgis.doc
  • postgresql16JitPackages.postgis
  • postgresql16JitPackages.postgis.doc
  • postgresql16Packages.postgis
  • postgresql16Packages.postgis.doc
  • python311Packages.bsuite
  • python311Packages.bsuite.dist
  • python311Packages.cartopy
  • python311Packages.cartopy.dist
  • python311Packages.django-bootstrap4
  • python311Packages.django-bootstrap4.dist
  • python311Packages.fiona
  • python311Packages.fiona.dist
  • python311Packages.folium
  • python311Packages.folium.dist
  • python311Packages.geopandas
  • python311Packages.geopandas.dist
  • python311Packages.osmnx
  • python311Packages.osmnx.dist
  • python311Packages.plotnine
  • python311Packages.plotnine.dist
  • python311Packages.pygmt
  • python311Packages.pygmt.dist
  • python311Packages.python-mapnik
  • python311Packages.python-mapnik.dist
  • python311Packages.rasterio
  • python311Packages.rasterio.dist
  • python311Packages.wktutils
  • python311Packages.wktutils.dist
  • python312Packages.gdal
  • python312Packages.pygmt
  • python312Packages.pygmt.dist
  • pytrainer
  • pytrainer.dist
  • qgis
  • qmapshack
  • saga
  • sumo
  • udig
  • vpv

Since gdal 3.8.1 there is a numpy dependency on the python library it
seems. This resulted in broken python scrips such as gdal2tiles.py

```
[nix-shell:~/map-scraper/shake-rg]$ gdal2tiles.py --help
ModuleNotFoundError: No module named 'numpy'
```

Therefore it's important to wrap the executables in an environment which
contains numpy.
@ofborg ofborg bot requested a review from imincik February 14, 2024 11:37
@imincik imincik merged commit 2881b6b into NixOS:master Feb 14, 2024
21 of 23 checks passed
imincik added a commit to imincik/geospatial-nix that referenced this pull request Feb 16, 2024
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/week-in-geospatial-team/37035/8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants