Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update gsi-addon-dev to include Python modules #942

Open
DavidHuber-NOAA opened this issue Jan 11, 2024 · 5 comments
Open

Update gsi-addon-dev to include Python modules #942

DavidHuber-NOAA opened this issue Jan 11, 2024 · 5 comments
Assignees
Labels
NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center

Comments

@DavidHuber-NOAA
Copy link
Collaborator

DavidHuber-NOAA commented Jan 11, 2024

Is your feature request related to a problem? Please describe.
The EMC_verif-global package generates plots that require the matplotlib and cartopy modules that are not built by the gsi-addon-dev template.

Describe the solution you'd like
These modules should be added to gsi-addon-dev.

Additional context
This is only needed in the standalone, postprocessing portion of verif-global and there is a workaround available by using local or older system/contributed installs of Python. Additionally, the gsi-addon environment may not be necessary much longer as the BUFR and Python issues that it was needed for are being resolved. Since there is a workaround, there isn't a rush to do this work, but it should be completed if it looks like BUFR and Python will not be resolved by the rollout of spack-stack in 2Q2024.

@DavidHuber-NOAA DavidHuber-NOAA self-assigned this Jan 11, 2024
@climbfuji climbfuji added OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center NOAA-EMC labels Jan 13, 2024
DavidHuber-NOAA added a commit to DavidHuber-NOAA/spack-stack that referenced this issue Mar 4, 2024
@DavidHuber-NOAA
Copy link
Collaborator Author

@AlexanderRichert-NOAA I am having some trouble adding new Python packages to the gsi-addon environment. To try and do this, I revived the emc-verif-global-env package, added the needed Python packages to it, and added emc-verif-global-env to gsi-addon-dev. However, when concretizing, both Python versions 3.11.6 and 3.10.13 are slated for installation, which is problematic. Am I missing something in my branch?

A sample environment was concretized here: /work/noaa/global/dhuber/SPACK/spack-stack-gsi-py/envs/gsi-addon (log file log.concretize) with the upstream /work/noaa/global/dhuber/SPACK/spack-stack-gsi-py/envs/unified-env/install.

@AlexanderRichert-NOAA
Copy link
Collaborator

You could try concretizer:unify:true if it will let you (and/or require an exact python version, which it looks like you're doing). In that case it should give you a concretization error if one of the additional packages has incompatible Python version constraints.

@DavidHuber-NOAA
Copy link
Collaborator Author

@AlexanderRichert-NOAA Thanks for the suggestion. I gave it a try, but the concretizer is still attempting to install both 3.10.13 and 3.11.6. It seems that cartopy is the problematic package based on this portion of the output of spack concretize:

==> Concretized emc-verif-global-env%intel ^bufr@11.7.0 ^met@9.1.3 ^metplus@3.1.1
...
 -   b465e5l      ^py-cartopy@0.21.1%intel@2021.9.0~ows+plotting build_system=python_pip arch=linux-rocky9-icelake
 -   syccxv4          ^gdal@3.8.3%intel@2021.9.0~archive~armadillo~arrow~basisu~blosc~brunsli~cfitsio~crnlib~cryptopp~csharp+curl~deflate~ecw+expat~filegdb~freexl~fyba+geos~gif~gta~hdf4~hdf5~hdfs~heif~iconv~idb~ipo~java+jpeg~jxl~kdu~kea~lerc~libaec~libcsf~libkml~liblzma~libqb3~libxml2~luratech~lz4~mongocxx~mrsid~mssql_ncli~mssql_odbc~mysql~netcdf~odbc~odbccpp~ogdi~opencad~opencl~openexr~openjpeg~openssl~oracle~parquet~pcidsk~pcre2~pdfium~pic+png~podofo~poppler~postgresql+python~qhull~rasterlite2~rdb~sfcgal+shared~spatialite+sqlite3~teigha~tiledb~webp~xercesc~zstd build_system=cmake build_type=Release generator=ninja patches=52459dc arch=linux-rocky9-icelake
[^]  e27r5y7              ^json-c@0.16%intel@2021.9.0~ipo build_system=cmake build_type=Release generator=make arch=linux-rocky9-icelake
[^]  vuiqv37              ^libgeotiff@1.7.1%intel@2021.9.0+jpeg+proj+zlib build_system=autotools arch=linux-rocky9-icelake
[^]  427zpsn              ^libtiff@4.5.1%intel@2021.9.0+ccitt~ipo~jbig+jpeg~jpeg12~lerc~libdeflate+logluv~lzma+lzw+next~old-jpeg+packbits~pic~pixarlog+shared+thunder~webp+zlib~zstd build_system=cmake build_type=Release generator=make arch=linux-rocky9-icelake
[^]  qole5nx              ^ninja@1.11.1%intel@2021.9.0+re2c build_system=generic arch=linux-rocky9-icelake
[^]  57u4gya                  ^python@3.10.13%intel@2021.9.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,ebdca64,f2fd060 arch=linux-rocky9-icelake

However, I have gone through the emc-verif-global-env dependent package.pys and I do not see any concretized versions that require Python :3.10, so I'm not sure where this is coming from.

@AlexanderRichert-NOAA
Copy link
Collaborator

It's coming from the upstream environment, which uses 3.10.13. If the idea is to use the non-Python packages from the upstream environment, I'm not sure there's a clean way to do that... You might try something like:

packages:
  all:
    require:
      - any_of: ["^python@3.11.6", "@:"]

which in theory should make it so that any packages that can use python@3.11.6 will do so.

@DavidHuber-NOAA
Copy link
Collaborator Author

Unfortunately, that's not working for me, either. I suspect that means this environment would need to be made standalone, rather than a downstream one. I'm not sure that we want to do this just for the additional Python packages needed for offline verification plotting. I will bring this up at the spack-stack meeting tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center
Projects
Development

No branches or pull requests

3 participants