Skip to content

Development

Jump to bottom
Carles Tena edited this page Oct 6, 2026 · 1 revision

NES Development

This page can be used by developers to better understand certain parts of the code:

Table of Contents

Vertical interpolation

# We force the interpolation with float64 to avoid negative values
# We don't know why the negatives appear with float32
if kind == 'linear':
    dst_data[t, :, j, i] = np.array(
        np.interp(new_levels,
                  np.array(self.lev['data'], dtype=np.float64),
                  np.array(var_info['data'][t, :, j, i], dtype=np.float64)),
        dtype=var_info['data'].dtype)
else:
    dst_data[t, :, j, i] = np.array(
        interp1d(np.array(self.lev['data'], dtype=np.float64),
                 np.array(var_info['data'][t, :, j, i], dtype=np.float64),
                 kind=kind,
                 bounds_error=False,
                 fill_value=fill_value)(new_levels),
        dtype=var_info['data'].dtype)

Write Grib2 files

NES uses ecCodes to write grib2 files.

import numpy as np
from eccodes import codes_grib_new_from_file
from eccodes import codes_keys_iterator_new
from eccodes import codes_keys_iterator_next
from eccodes import codes_keys_iterator_get_name
from eccodes import codes_get_string
from eccodes import codes_keys_iterator_delete
from eccodes import codes_clone
from eccodes import codes_set
from eccodes import codes_set_values
from eccodes import codes_write
from eccodes import codes_release

new_grib_file_path = "/Path/to/the/new/grib2/file.grb"
grib_template_path = "/Path/to/the/template/grib2/file.grb"
grib_keys = {'old_key': 'new_value'}
data = np.random.rand(24, 15, 100, 100)
lat_flip = False  # Sometimes the data is writing with inverse latitudes

# Open New grib file
fout = open(new_grib_file_path, 'wb')

# Open template
fin = open(grib_template_path, 'rb')

gid = codes_grib_new_from_file(fin)

# Iterate throw template keys
iterid = codes_keys_iterator_new(gid, 'ls')
while codes_keys_iterator_next(iterid):
    keyname = codes_keys_iterator_get_name(iterid)
    keyval = codes_get_string(gid, keyname)
    print("%s = %s" % (keyname, keyval))

codes_keys_iterator_delete(iterid)

# Clone the template for each variable, timestep and level
# Here is shown a simple 1 var 2D case
clone_id = codes_clone(gid)

# Adding grib2 new key values to file
for key, value in grib_keys.items():
    if value not in ['', 'None', None]:
        try:
            # The value must be in the correct datatype otherwise it will raise an error
            codes_set(clone_id, key, value)
        except Exception as e:
            print("Something went wrong while writing the Grib key '{0}': {1}".format(key, value))
            raise e

# Select the data
newval = data[0, 0, :, :]  # 2D data

# Sometimes the data is writing with inverse latitudes
if lat_flip:
    newval = np.flipud(newval)

# Grib2 cannot write NaN
# TODO Check default NaN Value
newval[np.isnan(newval)] = 0.

# Write the data in float32 sometimes it makes disappear some regions
codes_set_values(clone_id, np.array(newval.ravel(), dtype='float64'))

# Write all variable data
codes_write(clone_id, fout)

# Close everything
codes_release(gid)
fout.close()
fin.close()

Upload to PyPI

Every time a version is released, we should update NES in PyPI. To do so, we will first create a source distribution:

python setup.py sdist

We can check if it can be installed doing:

pip install dist/NES-1.X.X.tar.gz

If everything is correct, we can proceed to upload our distribution to the website. For this we need to install twine:

pip install twine

And get an API key in PyPI. I recommend creating a folder in your home directory called .pypirc, with the following content:

[pypi]
  username = __token__
  password = pypi-[key]

That way you won't be asked for the your credentials anymore. Make sure the version name has been updated in init.py, and no more changes are needed since it is not allowed to update the repository for that version once it is uploaded even if deleted.

To upload it, you can do:

twine upload dist/NES-1.X.X.tar.gz

Clone this wiki locally