Repository navigation
Proposal: A shared python environment for running E3SM #8906
Replies: 14 comments 52 replies
|
Just a quick clarification:
If running with a py venv loaded, eamxx installs in the py venv. This may or may not work, depending on whether the venv folder has That said, I have some questions:
Tbc: I do like the idea of components not having to install common py pkgs manually, so I welcome a more centric approach (while still allow components to run pip install at runtime, so they can experiment with new pkgs before we add them to e3sm env). |
|
This would replace the cime-env conda that some of the test scripts use? I don't think we load cime-env everywhere, just places that will run the non-BFB suite. But this would definitely be loaded by every test script. |
|
Two (more) comments:
|
|
paradoxically, I think this will be a regression in usability. I think we ought to support a builtin solution to source/build what's needed (just like how we do with shared libs, and to an extent the pragmatic solution in eamxx). Adding more obstactles to the obstacle course of E3SM usability is a regression imho I don't like the eamxx solution, but as you said, it is pragmatic. I think ultimately, a rethinking of how CIME works is needed. Beyond that, I think E3SM will also need to transition to shipping the executable/binaries as a release product rather than the source code, but we are far from that. I don't have concrete suggestions for alternatives, but I think this is a dangerous route to go down, wnich I think is a usability regression for E3SM. |
|
I would put that source command in my bashrc and forget about it. How might that bite me if I'm doing other python things in my session? |
|
Those interested in CIME refactor should follow ESMCI/cime#4950 and related issues linked therein. |
|
Clarification: this will be needed for production runs as well, right? Not just testing and development; case.setup is used by production runs as well, not just testing and development |
|
Another issue with this proposal is the fact that sometimes people need some packages only for very narrow cases. For example, the MVK test needs a specific environment because its so-called CIME's SystemTest has specific dependencies that nobody else needs. This proposal calls for a shared/unified solution
which means the union of everything needed for running, testing, and development. This will grow into a bit of a nightmare. Take the example of EAMxx, the standalone suite of tests requires cupy, torch, amongst several heavy hitters (see here, which btw, I plan to take out of the default containers to make them optional only for eamxx, E3SM-Project/containers#108). In the future, hypothetically, when people develop more pieces of software, and if they decide to link to the growing selection of jax ESM codes, this may become difficult to manage. Less hypothetically, E3SM groups in phase 4 (ocean, polar, atm, etc.) are supposed to develop in-component process emulators (say to replace a parameterization somewhere). A shared env will either have to contain all the software they need and will force them to use the same packages, which again, will likely grow too complex and hard to resolve, increasing maintenance burden on both side of development (the people developing these codes and the people maintaining the shared env(s)). The likely software solution for process emulation will be in the form of a python bridge (i.e., C++/F90 (-> C) -> Python). Two alternatives:
Or ofc both, i.e., give people the python project infra with pinned deps, but also have premade envs for sharing within e3sm. |
|
Opening one last thread for now about something specific that may help me make my points clearest: Clarification: in https://github.com/andrewdnolan/E3SM/tree/dev-env/shared-python-env/dev-env, what was the reasoning for creating the dir dev-env and not simply putting stuff in repo root? Are there downsides I am not aware of? As I see it right now, there are two distinct things in this proposal, and separating them helps me give you constructive feedback:
I realize the branch above has a deploy.py mechanism that resolves local envs; what I am asking for is for this to be formalized as a first-class mechanism to bootstrap e3sm that we test. On supported machines, this bootstrapping can just resolve the dev-env you're offering (when/if people can agree on its content) As a meta point:
I'd strongly advocate we keep these two items separates. If you agree with this distinction, I suspect we can work find a workable way to get to common consensus that will likely satisfy all mainstream needs. Supporting the first item alone is a step in the right direction for portability, maintainability, and usability, and further supporting the second item is a step towards convenience on supported e3sm machines. Doing it in this progression will make it more durable without sacrificing the advantages of a shared env. |
|
A question that occured to me: what are the OMEGA and EAMxx buildnml's using from pyyaml? The goal of the buildnml is to write out the yaml files (or namelist files) based on the configuration specified in create_newcase. Does it need pyyaml to write what is really a plain text fille? |
|
The design document says "CIME does not always launch a component's buildnml as a subprocess. When the file defines a |
|
Is this process right? A developer opens a PR that requires an addition to the python environment for all users. In that PR, they would also have to update dev-env/pixi.toml . When that branch is merged to next, the integrator would have to make sure the local install on each machine is updated for when next testing runs that night with the new dependency. Then when it passes and goes to master, users will get the new dependency automatically because they always do "source /lcrc/soft/climate/e3sm-dev-env/load_latest.sh". |
|
@jasonb5 can you look at this and comment on how it compares to your vision for how people will interact with CIME in the future? |
|
One thing I don't think we've covered yet: matching the environment to the E3SM version.
What we need on top of that:
This answers @rljacob's earlier question about the workflow. A PR that changes dependencies bumps the version in @andrewdnolan, could we work this into the design and prototype? |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
TL;DR: @xylar and I are proposing one E3SM-owned, locked Python environment that E3SM users select with a one-line loader before running any CIME commands. It frees component and test scripts from using just the Python standard library, gives us the same Python and package versions on every machine (nightly testing included), and formalizes a choice we already make by hand. A prototype exists, and I'd like to open the floor for objections/concerns to a shared environment approach.
Design document: E3SM development and test environment | Prototype branch:
andrewdnolan:dev-env/shared-python-envMotivation
Version 4 of E3SM will see the introduction of EAMxx and Omega as officially supported components. Both models use YAML files for model configuration, instead of traditional namelist files. This means that when each component's respective
buildnmlis called,pyyaml, which is not part of the Python standard library, must be importable.CIME imports
buildnmlinto the interpreter that is already running./case.setup, so PyYAML has to be installed in whicheverpython3the developer happened to load, and nothing in E3SM guarantees that:pip install --userat runtime. It is a pragmatic fix, but it needs network access, writes to your home directory and installs whatever release is newest the first time it runs. The package versions behind a given case then depend on who ran it, where and when, which makes exact reproducibility difficult.Currently, component scripts can only safely assume the standard library, and the Python you get depends on the machine.
The proposal
One E3SM-owned environment, defined in a top level
dev-env/1 directory and locked with Pixi (conda-forge packages, hash-pinned).One installer, two modes: a read-only shared installation per facility, or a "developer" anywhere (no admin rights). Shared deployment is a single command and took under a minute in initial prototyping.
Selected from your shell, never from machine configuration:
Python only, and fail fast. Compilers, MPI and NetCDF stay with machine configuration, and CIME stops with a clear message if the environment is missing.
What this opens up
pip install.This formalizes what we already do
On Chrysalis the system
python3is 3.6 and CIME requires 3.9 or newer, so developers already have to load some Python before running CIME. This proposal just formalizes which one gets loaded. Nightly testing already does this on Chrysalis and Perlmutter, where E3SM_test_scripts sources a sharedcime-envloader before invoking CIME.Where things stand
Status: the installer and loaders work on Chrysalis; the rest of the rollout and validation is in design doc §8, and nothing is published yet. I'd like to open up discussion about the shared environment and how we package that shared environment. Once we have a consensus on the packaging and deployment approach, I can complete the prototype and do a test deployment on a few choice machines. At that point we'd need to alert the broader E3SM user community, as this plan will affect everyone that runs E3SM. But, I'd like to hold off on that until we have full buy-in for the shared environment and an agreed upon packaging approach.
Footnotes
dev-envande3sm-dev-envare just placeholder names and are open to discussion. ↩All reactions