Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added monthly mean vars (ta, va, zg) to era5 cmorizer via recipe #1644

Merged
merged 1 commit into from May 19, 2020

Conversation

egalytska
Copy link
Contributor

@egalytska egalytska commented May 5, 2020

Tasks

  • Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
  • Give this pull request a descriptive title that can be used as a one line summary in a changelog
  • Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
  • Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
  • Please use yamllint to check that your YAML files do not contain mistakes

If you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.


Closes #issue_number

I would like to comment on a few issues:

  1. In order to cmorize geopotential height (zg) I had to edit ESMValCore/esmvalcore/cmor/_fixes/native6/era5.py to convert "geopotential" to "geopotential height". Should I create a new request in ESMValCore to add those couple lines?
  2. After additional ERA5 data was moved to native6, seems that 2 variables "temperature" (ta) and "temperature_of_snow_layer" (tsn) are now found simultaneously by recipe_era5.yml when given era5_name: temperature. Maybe, it worth renaming "temperature" to "t" or so?

@egalytska egalytska added the era5 label May 5, 2020
@egalytska egalytska requested a review from mattiarighi May 5, 2020
@bouweandela bouweandela self-requested a review May 12, 2020
@bouweandela
Copy link
Member

bouweandela commented May 12, 2020

I have created an issue with era5cli to see if the filenames it generates can be changed to look more like the filenames used with other climate data: eWaterCycle/era5cli#53

@bouweandela
Copy link
Member

bouweandela commented May 12, 2020

Should I create a new request in ESMValCore to add those couple lines?

Yes

@bouweandela
Copy link
Member

bouweandela commented May 18, 2020

@valeriupredoi Would you be able to test this?

@mattiarighi
Copy link
Contributor

mattiarighi commented May 18, 2020

The recipes fails with several variables, not only with the new ones.
The error looks always the same:

2020-05-18 18:12:56,169 UTC [48027] INFO    Starting task monthly/zg in process [48027]
2020-05-18 18:12:56,213 UTC [48028] ERROR   Failed to run fix_metadata([<iris 'Cube' of air_temperature / (K) (time: 12; pressure_level: 37; latitude: 721; longitude: 1440)>, <iris 'Cube' of temperature_in_surface_snow / (K) (time: 12; latitude: 721; longitude: 1440)>, <iris 'Cube' of air_temperature / (K) (time: 12; pressure_level: 37; latitude: 721; longitude: 1440)>, <iris 'Cube' of temperature_in_surface_snow / (K) (time: 12; latitude: 721; longitude: 1440)>], {'project': 'native6', 'dataset': 'ERA5', 'short_name': 'ta', 'mip': 'Amon', 'frequency': 'mon', 'check_level': <CheckLevels.DEFAULT: 3>})
2020-05-18 18:12:56,268 UTC [47969] INFO    Progress: 8 tasks running, 1 tasks waiting for ancestors, 8/17 done
2020-05-18 18:12:56,315 UTC [47969] INFO    Maximum memory used (estimate): 4.5 GB
2020-05-18 18:12:56,316 UTC [47969] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2020-05-18 18:12:56,318 UTC [47969] ERROR   Program terminated abnormally, see stack trace below for more information
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/pf/b/b309057/SOFTWARE/miniconda3/envs/esmvaltool/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_task.py", line 693, in _run_task
    output_files = task.run()
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/_task.py", line 240, in run
    self.output_files = self._run(input_files)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/preprocessor/__init__.py", line 426, in _run
    product.apply(step, self.debug)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/preprocessor/__init__.py", line 294, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/preprocessor/__init__.py", line 236, in preprocess
    result.append(_run_preproc_function(function, items, settings))
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/preprocessor/__init__.py", line 222, in _run_preproc_function
    return function(items, **kwargs)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/cmor/fix.py", line 107, in fix_metadata
    cube_list = fix.fix_metadata(cube_list)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/cmor/_fixes/native6/era5.py", line 325, in fix_metadata
    cube = self._fix_coordinates(cube)
  File "/mnt/lustre01/pf/b/b309057/ESMValTool/core/esmvalcore/cmor/_fixes/native6/era5.py", line 277, in _fix_coordinates
    coord = cube.coord(axis=axis)
  File "/pf/b/b309057/SOFTWARE/miniconda3/envs/esmvaltool/lib/python3.8/site-packages/iris/cube.py", line 1497, in coord
    raise iris.exceptions.CoordinateNotFoundError(msg)
iris.exceptions.CoordinateNotFoundError: 'Expected to find exactly 1  coordinate, but found none.'

@bouweandela
Copy link
Member

bouweandela commented May 19, 2020

The recipes fails with several variables, not only with the new ones.

This is exactly the error reported under point 2 at the top of this pull request. Maybe you can try with the new default DRS proposed here? ESMValGroup/ESMValCore#645

@mattiarighi
Copy link
Contributor

mattiarighi commented May 19, 2020

That does not help. I think the problem is the use of _ as a separator in the variable name, it should be changed to - in era5cli.

However, I managed to test thir PR by running each variable separately and moving out the non required files from the input directory.

Since the problem is not related to this PR, we can merge it. I also tested ESMValGroup/ESMValCore#640 in this branch.

@egalytska can you please open an issue about the Point 2. above (if not already done)?

@mattiarighi mattiarighi merged commit cdf95f7 into master May 19, 2020
6 checks passed
@mattiarighi mattiarighi deleted the ERA5_cmor_via_recipe_update branch May 19, 2020
@bouweandela
Copy link
Member

bouweandela commented May 19, 2020

That does not help. I think the problem is the use of _ as a separator in the variable name, it should be changed to - in era5cli.

That is exactly the kind of problem ESMValGroup/ESMValCore#645 tries to solve. Why did it not work? Maybe we can discuss further in that pull request?

@egalytska
Copy link
Contributor Author

egalytska commented May 19, 2020

Dear all. Just an update on this issue:
(i) By the time I downloaded ERA5 data I did not know I could do it via era5cli. So I used CDS API.
(ii) Raw ERA5 files I named as era5_temperature_monthly.nc . After data was moved to ~work/bd0854/DATA/ESMValTool2/RAWOBS/ my recipe started giving an error. That's when I realized that for e.g. ERA-Interim raw data is named as ERA-Interim_t_monthly.nc. So if I would name ERA5 raw data as era5_t_*_monthly.nc, I assume there would be no problem at all. (I consider it my mistake).
(iii) Sorry, it takes me longer than it should. I am still somewhat learning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants