Tutorial Issue: running recipe_python.yml #1923
Comments
|
Hi @almerrifield thanks for reporting this. In the debug log one of the last lines is: Perhaps this is similar to the recent issue we've seen here: #1894
|
|
I'll give it a try, thank you! |
|
Unfortunately, upon running: There seems to be a conflict between esmvaltool-python and cartopy version >= 0.18. UnsatisfiableError: The following specifications were found to be incompatible with each other: |
|
The log is not super clear, but it appears to be a very similar plotting issue indeed. I just tried and the command above does work on my system without conflicts. Did you remove the existing
|
|
I did remove the esmvaltool environment using: Even with a new environment name, the same conflict occurs... |
|
I had the same issue with this recipe, relating to: Following the suggestion above, and installing a new environment, it switches to a different issue: I'm not sure of exactly why, but I notice that when creating the environment as above: this produces an install with: whereas if I do not specify the cartopy version, I end up with version this allows it to function for me at least. |
|
@swartn that's very helpful information, thanks! @bouweandela @valeriupredoi any thoughts on these version conflicts? |
|
@swartn Thank you for the recommendation! We are currently triaging the situation in-house with the following updates: -On our servers, we as users cannot update the conda environment, but have access to conda 4.9.2 The following works to create the environment, but not run the recipe:
The following cannot create the environment, due to package conflicts:
The following cannot create the environment, due to more package conflicts: @ruthlorenz has had a look too. She has found an additional issue associated with the config file. When uncommenting the rootpath and drs to input data, the tab structure needs to be maintained. In the example, it is correct. However, the tabs are missing in the solution for "set the correct rootpath". The recipe also will not run without the correct tab structure... |
|
there is no need to pass all those dependency constraints at env creation stage - |
|
another thing - when we made the 2.1 release we checked if the simple install from conda via |
|
@valeriupredoi Thank you! Unfortunately re: |
I would not recommend that on our servers since the base environment is mutual between all users (and I do not think the general user can install anything into it). We would need to ask our IT person to do it, and if I remember correctly we have talked about it before and decided against it. Regarding Is this mostly an issue we have at ETH or is this/could this be more common? Thanks for helping though :-) |
|
ah my apologies, I didn't know this a case of not enough permissions to use the base environment; I would recommend having your own conda installation - in all cases this is the preferred solution because it becomes hairy when certain packages need to be written to the |
@almerrifield Thanks for reporting this, I have opened an issue about it in the tutorial repository: ESMValGroup/ESMValTool_Tutorial#170 |
|
I think the problem is that cartopy 0.17 is not compatible with matplotlib 3.3. So if installing cartopy 0.18 is not possible, you could also try installing matplotlib 3.2 instead? Another possibility would be to try mamba instead of conda to install the package. It has a much better dependency solver. |
|
good point @bouweandela - that's why I suggested recreating the environment since for a while now (more than a month) |
|
@valeriupredoi We do have cartopy 0.17.0 in our base environment so I tried to install cartopy 0.18.0 using
After a lengthy report of conflicts, cartopy 0.18 did not install :( If it helps, here is the list of packages in the esmvaltool conda environment I was able to create: |
|
@almerrifield cheers for the detailed diagnosis! I have replicated your chain of commands and indeed, You should then have a solid working environment with A few notes:
Cheers 🍺 |
|
@valeriupredoi Thank you for following up, I was very much looking forward to completing the cheers 🍻. All went smoothly until the pip installs:
and as anticipated: I think you are right that central installation may be necessary at this point.... Thanks to all again! |
yeah I could see that coming, my base env had it and conda finally had the decency to look in the base env prefix path too @bouweandela and myself we'll release a bugfix version of Core with pinning cartopy to 0.18 (we'll try do that tomorrow, Bouwe had the idea to do it and I think it's welcome for people struggling to install in standard mode like yourself, but I do encourage you to ask the ETHZ sys admins to look into a maintenable central installation 👍 ) |
|
Interesting, for the old version ( conda create --name esmvaltool_conda -c esmvalgroup -c conda-forge esmvaltool=2.1.0 esmvaltool-python=2.1.0
mamba create --name esmvaltool_mamba -c esmvalgroup -c conda-forge esmvaltool=2.1.0 esmvaltool-python=2.1.0Sometimes the incompatibilities are difficult to know and can be very indirect (e.g. both cdo and cartopy pin proj). But the conflict conda shows do not really make sense and I also made this experience before... So thanks for pointing out mamba! I did not know this tool. The new version of esmvaltool ( conda create --name esmvaltool -c esmvalgroup -c conda-forge esmvaltoolFeel free to ask or tag me directly when you have problems with setting up an environment. We can also discuss setting up a global esmval environment. @valeriupredoi please don't recommend installing esmvaltool in the base environment. The base environment should basically only contain conda and its dependencies. See also in the conda docs:
That's what environments are for & when you do |
|
Oh I just found out - mamba does not enforce mamba create --name esmvaltool_mamba -c esmvalgroup -c conda-forge esmvaltool=2.1.0 esmvaltool-python=2.1.0mamba wants to install cartopy from |
|
Sorry, couldn't leave it: your actual problem is pynio. pynio is no longer maintained (see conda-forge/pynio-feedstock#90). It conflicts with the newest version of netCDF, but also with cartopy mamba create --name test_pynio --override-channels --strict-channel-priority -c conda-forge pynio python cartopy=0.18I see only one mention of |
|
we have just released 2.1.1 today and I wanted to test carefully its full functionality before we posted here that's ready for use and it'd solve this issue, cheers @mathause for trying it out and posting your results!
yes and no: conda is a package manager first and a virt environment manager second (and it's doing a relatively poor job as a package manager) so there is no problem to have an installation in
Careful here - if all you share are paths to executables and some lib dirs, it's perfectly fine to do it (this is how a module is constructed anyway). No other permissions, no possibility to install new packages. If you start from a 100% working environment you will be sure your users will have it working as well (provided they don't mix paths, that'd be havoc 😁 )
Yes, well spotted, I've already told @bouweandela today, after some env solving headaches the previous days, we should look into removing that! 🍺 Cheers for the comments @mathause - good points overall, and good detective work! Hopefully @almerrifield will be able to get the installation working! @mathause if you have a plan for a central install at ETHZ that'd be great, I can share the module I built at JASMIN for that central install with you 👍 |
|
oh and apparently |
|
@bouweandela @mathause @valeriupredoi Success!! 🍻🍻🍻
threw an r-base SafetyError and a series of conda-forge/linux-64 ClobberErrors, but ultimately worked and the recipe ran! Thank you all for continuing to follow up! |
|
excellent, cheers @almerrifield for the heads up and sticking with it! I created an issue for |
Describe the bug
A clear and concise description of what the bug is. If you are developing a new diagnostic script, please provide a link to the code/branch on GitHub that you are working in.
I am running the tutorial using the complete esmval package on an ETHZ server:
(conda create -n esmvaltool -c conda-forge -c esmvalgroup esmvaltool)
The installed version is:
ESMValCore: 2.1.0
ESMValTool: 2.1.0
When running the first recipe, I get this error:
Diagnostic script examples/diagnostic.py failed with return code 1. In the log.
In the log, there is an AttributeError: 'NoneType' object has no attribute 'lower'.
Please attach
rundirectory in the output directorymain_log_debug.txtfile, this can also be found in therundirectory in the output directoryRe: Attaching the recipe. The .yml file type is not supported as an attachment so I've attached it as a .txt.
recipe_python.txt
main_log_debug.txt
The text was updated successfully, but these errors were encountered: