Repository Structure

This repository structure corresponds loosely to the top-level structure of the UFS, but with many submodules replaced by subtrees for convenience. See this description of the submodule hierarchy of the UFS. This table shows the canonical upstream repositories associated with different subtrees of this repository:

.                          # ~ https://github.com/ufs-community/ufs-weather-model
├── FMS                    # https://github.com/NOAA-GFDL/FMS
├── FV3                    # https://github.com/NOAA-EMC/fv3atm
│   ├── atmos_cubed_sphere # https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere
│   └── ccpp
│       ├── framework      # https://github.com/NCAR/ccpp-framework
│       └── physics        # https://github.com/NCAR/ccpp-physics
├── serialbox              # https://github.com/GridTools/serialbox
└── stochastic_physics     # https://github.com/noaa-psd/stochastic_physics

In some cases these are actual submodules, and in other cases they are subtrees.

Deprecation of serialize build support

As September, 2023 we have deprecated the infrastructure for building and testing the model in a Docker container. All development and testing now occurs in the nix environment, described below. This means that the latest version of the repository no longer includes code to facilitate building the model to output serialized data through serialbox, which previously was done using a Docker image; if you would still like to use this functionality, be sure to check out commit b3793303df9800de341a3f8e8b234c6f4dc2b7a1 or earlier.

Developing the model and running tests

Nix

The primary mode of development in this repository now runs through the nix package manager. This package manager is available on Mac and Linux, and provides a light weight means to distribute isolated software environments.

To begin, install nix following these instructions.

(optional) We host binaries using a tool called cachix, and this will greatly speed up any builds. To use our binaries, install cachix and then run

cachix use vulcanclimatemodeling

Without using the cachix cache, FV3 and all its dependencies will need to build from source (~20 minutes). This only happens once per machine, but it is slow.

Developing

To develop the model, you can use the environment specified in shell.nix by running

nix-shell

Then configure the build to use nix

cd FV3 && ./configure nix

And build the model (from the root directory)

make build_repro

And the wrapper

make build_wrapper

At this point you can run the tests.

Testing the fortran model

Tests of the pure fortran model are included in the tests subdirectory. You can see which ones run in continuous integration by inspecting .circleci/config.yml.

Regression tests which check the bit-reproducibility of results are performed for a set of reference configurations included in tests/pytest/config. Please read the README in tests/pytest for more information about these regression tests and how to update the reference checksums.

Testing the model in different compile modes

The fortran model can be built with different sets of compiler flags, defined in the FV3/conf/*.configure.fv3 files. These sets of compiler flags facilitate running the model in different modes. For production workflows, we run the model with the repro set of compiler flags; for debugging it can be useful to compile and run the model with the debug set of compiler flags, which can facilitate catching array out of bounds errors and the like.

To build the model in repro mode use:

make build_repro

To build the model in debug mode use:

make build_debug

To build executables in both repro and debug mode use:

make build

The test infrastructure is written to detect and run tests only for which the appropriate executable exists, since regression tests produce different results when running in repro versus debug mode.

To run the full suite of fortran model tests for the compiled executables, use:

make test_fortran

Since the tests are time-consuming, during development it can often be helpful to select specific tests that pertain to the code you are developing, e.g. with the -k option of pytest:

pytest -v -k test_use_prescribed_sea_surface_properties tests/pytest/test_regression.py

There are also make rules which select tests to run based on pytest marks, e.g.:

make test_fortran_basic

Testing the Python wrapper

The FV3/wrapper subdirectory contains a python wrappper that can be used to call the fortran model for interactive ML. It's usage docs can be seen here.

To build the wrapper some additional python requirements must be available in the environment:

• fv3config
• pace-util
• numpy
• pyyaml
• xarray
• cython
• mpi4py

The [nix environment][#developing] is setup automatically with these dependencies, and is the recommend development environment for iterative development (edit/build/test) of the wrapper. In other environments, e.g. HPC/DOCKER these builds dependencies will need to be manually installed.

Once the dependencies are installed, including the repro-mode version of the fortran executable, the wrapper can be built like this

make build_wrapper

To install the wrapper in some python environment, you can use the wheel built above

pip install FV3/wrapper/dist/fv3gfs_wrapper*.whl

To test the wrapper

# if you want to test FV3/wrapper in-place uncomment the line below
# otherwise it will use the version installed in the python environment
# export PYTHONPATH=$(pwd)/FV3/wrapper:$PYTHONPATH

make test_wrapper

Authentication

The input data files required by our tests are stored in a GCS bucket, stored in the us-central1 region. This data is free to use, but we have enabled requestor-pays to avoid paying for network transfer costs incurred by external users. You will need to authenticate with your own google cloud project credentials to access this data. Detailed instructions are out of scope, but usually involves the setting the following environmental variables

export FSSPEC_GS_REQUESTER_PAYS="on"
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json

For more information see this documentation.