CMIP6 reference

[oloapinivad]

This is a branch aiming at being able to compute a new reference for performance indices based on cmip6 data.
It includes both a py-climatology-create.py script which takes care of climatology generation starting from original observational data. This creates also the yml file necessary for the climatology definion an further comparison. The climatology is now made by a yearly averaged file and a by seasonly averaged file, so that season-dependent PI can be estimated.

Once this is established, cmip6-clim-evaluate.py run the performance indices over a set of 10 CMIP6 models and then compute the mean of each PI, updating the climatology file. This is later used by a new heatmap figure which provide a very detailed summary of the model performance. An example here below:

PI4_EC23_historical_EC-Earth3_r1i1p1f1_1990_1990.pdf

[oloapinivad]

Further improvements including

• EC22 climatology has now included the average of about 10 cmip6 models
• A heatmap/score card is now included for a fast comparison against CMIP6
• Regional differences are now recorderd, provided multiple performance indices.
• Slowdown when processing CMIP6 data #48 solved with better chunking and correct time selection

[oloapinivad]

Massive changes has been introduced:

• A new script to create the climatology has been put in place py-climatology-create.pyand discussed in Procedure to create a new climatology #49
• EC23 climatology is now introduced (both naming and existing might be temporary)
• The season and regional averages are now in place: output is saved in yaml format aside of the original tables. A figure is also produced in the form a score card (especially the quality of the latter has to be improved).

[oloapinivad]

Multiple improvements:

• robust climatology creation

• histogram for inspection of climatology: an example here:
  tas_CRU_r360x180_1990_2019_ALL.pdf

• season average available

• high-level detail heatmap

• yml file replacing txt file for raw output

• pytest updated

[oloapinivad]

Documentation has been further updated. @jhardenberg I think you might want to have a look, I feel we are very close to merge

[oloapinivad]

A benchmark.py script has been introduced to produce a barplot and assess the performance of the code on different cores, addressing #53 . This showed some strange behaviour which might be further investigated, probably associated - but not only - to #50. The new barplot is automatically placed within the documentation so that every new branch has a reference for this.

[oloapinivad]

replaced the chunking function with a new one which uses an estimate of the required computational time for each variable, which improves the scaling with many cores.

[oloapinivad]

further improvement of docs

[oloapinivad]

Replaced the workflow testing from conda to mamba. It is about 2x faster.

[oloapinivad]

rolling back since mamba works only with python 3.10 :-)

[oloapinivad]

@jhardenberg please let me know if I can merge or if you want to have some more time to double check!