Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix provenance of Julia example diagnostic #2289

Merged
merged 3 commits into from
Nov 1, 2021
Merged

Conversation

bouweandela
Copy link
Member

@bouweandela bouweandela commented Sep 7, 2021

Description

The Julia diagnostic was recording provenance in a way that used to be recommended until a year ago, but did not actually work. This pull request updates the diagnostic so it uses the latest recommendations from https://docs.esmvaltool.org/en/latest/community/diagnostic.html#recording-provenance


Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 馃洜 Technical or 馃И Scientific review.

New or updated recipe/diagnostic


To help with the number of pull requests:

@bouweandela
Copy link
Member Author

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Sep 7, 2021

Since @bouweandela asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@esmvalbot
Copy link

esmvalbot bot commented Sep 7, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@valeriupredoi
Copy link
Contributor

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Oct 20, 2021

Since @valeriupredoi asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@esmvalbot
Copy link

esmvalbot bot commented Oct 20, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@remi-kazeroni
Copy link
Contributor

I have just killed the ESMValBot run on the VM and did some maintainance there. Problems so far have been unsolvable conda environments (done as in the doc, missing -c conda-forge) and failing installation (fixed in #2344).

I'm starting a new run now. The logs for the enviroment creation and software installation should be recorded and displayed on the output page. So in theory, there should not be any "404 error" any more. If the bot still does not run, I'll open a specific draft PR to test things a bit more...

@remi-kazeroni
Copy link
Contributor

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Oct 20, 2021

Since @remi-kazeroni asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@valeriupredoi
Copy link
Contributor

@remi-kazeroni excellent! 馃嵑

@esmvalbot
Copy link

esmvalbot bot commented Oct 20, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@valeriupredoi
Copy link
Contributor

the bot is still installing pip conda-forge/noarch::pip-21.3-pyhd8ed1ab_0 so no wonder it runs into Build backend does not support editables, falling back to setup.py egg_info. - just merged main - now it should work!

@valeriupredoi
Copy link
Contributor

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Oct 20, 2021

Since @valeriupredoi asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@esmvalbot
Copy link

esmvalbot bot commented Oct 20, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@remi-kazeroni
Copy link
Contributor

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Oct 20, 2021

Since @remi-kazeroni asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@esmvalbot
Copy link

esmvalbot bot commented Oct 20, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@remi-kazeroni
Copy link
Contributor

Not sure the Julia installation with the bot is fine. See the log of the diagnostics:

ERROR: LoadError: LoadError: could not load library "/mnt/esmvaltool_disk1/esmvalbot-work/esmvalbotwork-fix-example-julia-prov-kta6hhlq/conda/envs/esmvaltool/lib/libpython3.9.so.1.0"
/mnt/esmvaltool_disk1/esmvalbot-work/esmvalbotwork-fix-example-julia-prov-kta6hhlq/conda/envs/esmvaltool/lib/libpython3.9.so.1.0: cannot open shared object file: No such file or directory. Please run `Pkg.build("Py\
Call")` if your Python build has changed
Stacktrace:
 [1] top-level scope
   @ ~/.julia/packages/PyCall/BD546/src/startup.jl:51
 [2] include(x::String)
   @ PyCall ~/.julia/packages/PyCall/BD546/src/PyCall.jl:1
 [3] top-level scope
   @ ~/.julia/packages/PyCall/BD546/src/PyCall.jl:38
 [4] top-level scope (repeats 2 times)
   @ none:1
in expression starting at /home/esmvalbot/.julia/packages/PyCall/BD546/src/startup.jl:41
in expression starting at /home/esmvalbot/.julia/packages/PyCall/BD546/src/PyCall.jl:1

caused by: could not load library "/mnt/esmvaltool_disk1/esmvalbot-work/esmvalbotwork-fix-example-julia-prov-kta6hhlq/conda/envs/esmvaltool/lib/libpython3.9.so.1.0"
/mnt/esmvaltool_disk1/esmvalbot-work/esmvalbotwork-fix-example-julia-prov-kta6hhlq/conda/envs/esmvaltool/lib/libpython3.9.so.1.0: cannot open shared object file: No such file or directory
Stacktrace:
 [1] top-level scope
   @ ~/.julia/packages/PyCall/BD546/src/startup.jl:48
 [2] include(x::String)
   @ PyCall ~/.julia/packages/PyCall/BD546/src/PyCall.jl:1
 [3] top-level scope
   @ ~/.julia/packages/PyCall/BD546/src/PyCall.jl:38
 [4] top-level scope (repeats 2 times)
   @ none:1
ERROR: LoadError: Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to /home/esmvalbot/.julia/compiled/v1.6/PyCall/jl_nuxnLp.
Stacktrace:
 [1] top-level scope (repeats 2 times)
   @ none:1
in expression starting at /home/esmvalbot/.julia/packages/PyPlot/XaELc/src/PyPlot.jl:2
ERROR: LoadError: Failed to precompile PyPlot [d330b81b-6aea-500a-939a-2ce795aea3ee] to /home/esmvalbot/.julia/compiled/v1.6/PyPlot/jl_rQ3H5h.
in expression starting at /mnt/esmvaltool_disk1/esmvalbot-work/esmvalbotwork-fix-example-julia-prov-kuzmhlm1/ESMValTool/esmvaltool/diag_scripts/examples/diagnostic.jl:20

@valeriupredoi
Copy link
Contributor

@remi-kazeroni how is the Julia package being installed, is it using esmvaltool install Julia?

@zklaus
Copy link
Contributor

zklaus commented Oct 20, 2021

I am fairly certain this is because Julia itself is installed by the OS package manager and doesn't play nicely with the Python coming from conda-forge. I will suggest a PR to update the install script of the bot to improve the Julia situation, but let's get the general mamba goodness in there first.

@valeriupredoi
Copy link
Contributor

...and I, after a fresh env creation and install in this branch, am getting a different error while running the recipe_julia.yml:

example: starting routine
example: creating work and plot directories
example: working on file /home/valeriu/esmvaltool_output/recipe_julia_20211020_153614/preproc/example/tas/CMIP5_CanESM2_Amon_historical_r1i1p1_tas_1997-1997.nc
example: calling diagnostic with following parameters
CanESM2 CanESM2 1997 1997 historical r1i1p11
example: I am your Julia diagnostic
Base.StackTraces.StackFrame[close(::NcFile) at NetCDF.jl:871, write_netcdf2d(::String, ::Array{Float32,3}, ::Array{Float64,1}, ::Array{Float64,1}, ::String, ::String) at write_netcdf2d.jl:50, compute_diagnostic(::Dict{Any,Any}, ::String, ::String, ::Int64, ::String, ::String) at diagnostic.jl:77, main(::Dict{Any,Any}) at diagnostic.jl:115, top-level scope at diagnostic.jl:128, include(::Function, ::Module, ::String) at Base.jl:380, include(::Module, ::String) at Base.jl:368, exec_options(::Base.JLOptions) at client.jl:296, _start() at client.jl:506]
鈹 Warning: NetCDF.close is deprecated, because closing files is done with finalizers from now on.
鈹 @ NetCDF ~/.julia/packages/NetCDF/0sRUL/src/NetCDF.jl:872
ERROR: LoadError: PyError (PyImport_ImportModule

The Python package cartopy.crs could not be imported by pyimport. Usually this means
that you did not install cartopy.crs in the Python version being used by PyCall.

with a julia: julia version 1.5.1 - I'll try reinstall Julia see if this goes away

@valeriupredoi
Copy link
Contributor

OK same problem with julia=1.6.3 installed via jill (as our instructions say - btw @bouweandela we really need to add there the last step, the one that ln -s the executable, since that user installation remains in .local and doesn't become system-wide). But by adding the Conda package and installing cartopy from setup.jl, like:

@info "Load conda packages"
Pkg.add("Conda")
using Conda
Conda.add("Cartopy")

all goes well and the recipe runs fine. few points:

  • we should check the setup.jl since it looks to be antiquated
  • we should enable Conda support for Julia and download all the deps from Conda
  • we should probably allow the user to install their own needed Julia deps by making the setup.jl (or whatever deps controller from ESMValTool) visible and editable by the user; we should not support any more deps than we need (ie what we have in our grand total of 2 Julia recipes)

I am happy for this PR to be merged (again, heh) since all works fine with it, it's just the Julia infrastructure is a bit rusty in our tool

@valeriupredoi
Copy link
Contributor

@remi-kazeroni if you enable the Conda pkgs in setup.jl for esmvalbot (like above) then the poor thing will not be complaining about missing modules; but we should have this done at top level, not as special case for the bot 馃

@zklaus
Copy link
Contributor

zklaus commented Oct 20, 2021

No no no. Just install Julia via conda and forget about jill and the rest.

@valeriupredoi
Copy link
Contributor

ah mucho better then! I'll test tomorrow, had too much of Julia's capricious nature today 馃榿

@esmvalbot
Copy link

esmvalbot bot commented Oct 29, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@valeriupredoi
Copy link
Contributor

valeriupredoi commented Oct 29, 2021

why is the bot performing the julia installation outside the main esmvaltool environment creation and why is it installing a whole lot of packages while performing (well, right before) the diagnostic? Something's not right here

@valeriupredoi
Copy link
Contributor

valeriupredoi commented Oct 29, 2021

OK I had a similar issue with the example recipe (run locally, fresh env created, fresh install of this branch and main too), but missing Cartopy (seen this before!), but no rogue conda pkgs installs inside the diagnostic, I added the following in setup.jl:

import Pkg
Pkg.add("Conda")
using Conda
Conda.add("Cartopy")

and the recipe went and finished fine! I think we really are missing cartopy from Julia's environment, no buts no ifs 馃嵑

@zklaus
Copy link
Contributor

zklaus commented Nov 1, 2021

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

Since @zklaus asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@zklaus
Copy link
Contributor

zklaus commented Nov 1, 2021

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

Since @zklaus asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@zklaus
Copy link
Contributor

zklaus commented Nov 1, 2021

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

Since @zklaus asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@valeriupredoi
Copy link
Contributor

I am getting a 404 when trying to look at the bot's output, am I the only one?

@zklaus
Copy link
Contributor

zklaus commented Nov 1, 2021

@valeriupredoi, no you are not the only one. I'm working on it...

@valeriupredoi
Copy link
Contributor

good man! Give the bot a couple strong coffees, it seems he's a bit hungover 馃榿

@zklaus
Copy link
Contributor

zklaus commented Nov 1, 2021

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

Since @zklaus asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

ESMValBot is sorry to report it failed to run recipe examples/recipe_julia.yml: exit is 1, output has been generated here

@zklaus
Copy link
Contributor

zklaus commented Nov 1, 2021

@esmvalbot Please run examples/recipe_julia.yml

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

Since @zklaus asked, ESMValBot will run recipe examples/recipe_julia.yml as soon as possible, output will be generated here

@esmvalbot
Copy link

esmvalbot bot commented Nov 1, 2021

ESMValBot is happy to report recipe examples/recipe_julia.yml ran OK, output has been generated here

@zklaus
Copy link
Contributor

zklaus commented Nov 1, 2021

Ok, so the issue was a combination of Julia always setting up its own environment in $HOME/.julia and the being set up in a specific way. I made some local changes that allow the bot to run and will follow up with a PR on the bot repo.

In any case, this PR is therefore ok 馃帀

@zklaus zklaus self-assigned this Nov 1, 2021
@zklaus zklaus self-requested a review November 1, 2021 12:59
@zklaus zklaus merged commit 091617c into main Nov 1, 2021
@zklaus zklaus deleted the fix-example-julia-prov branch November 1, 2021 12:59
@valeriupredoi
Copy link
Contributor

excellent stuff! I am still confused as to why my local run of the recipe still needs cartopy though...am investigating now too

@zklaus
Copy link
Contributor

zklaus commented Nov 1, 2021

@valeriupredoi, not sure about that, but make sure to start from a clean environment and I suggest you nix your $HOME/.julia dir as well.

@valeriupredoi
Copy link
Contributor

happy camper here! Did exactly that - fresh env, nuked ~/.julia, installed deps with esmvaltool install Julia then recipe ran smooth like a baby's arse 馃嵑

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants