Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Example 3 does not run with Python 3 #65

Closed
larsbuntemeyer opened this issue Dec 11, 2019 · 7 comments
Closed

Example 3 does not run with Python 3 #65

larsbuntemeyer opened this issue Dec 11, 2019 · 7 comments

Comments

@larsbuntemeyer
Copy link

larsbuntemeyer commented Dec 11, 2019 •
edited
Loading

Example 3: indice TG90p with callback
runs it with py2.7 and icclim V4.2.11
crashes with py3.6 and icclim V4.2.13
i get:

2019-12-11 10:23:09,491    ********************************************************************************************
2019-12-11 10:23:09,491    *                                                                                          *
2019-12-11 10:23:09,493    *          icclim                                                            V4.2.13       *
2019-12-11 10:23:09,493    *                                                                                          *
2019-12-11 10:23:09,493    *                                                                                          *
2019-12-11 10:23:09,493    *          Wed Dec 11 10:23:09 2019 GMT                                                    *
2019-12-11 10:23:09,493    *                                                                                          *
2019-12-11 10:23:09,493    *          BEGIN EXECUTION                                                                 *
2019-12-11 10:23:09,493    *                                                                                          *
2019-12-11 10:23:09,493    ********************************************************************************************
2019-12-11 10:24:45,325 Loading data...
2019-12-11 10:24:57,910 Daily Percentiles calculation for out-of-base years. Please be patient...
Traceback (most recent call last):
  File "test_TG90p.py", line 32, in <module>
    icclim.indice(indice_name='TG90p', in_files=filenames, var_name='tas', slice_mode='AMJJAS', time_range=[dt1, dt2], base_period_time_range=[base_dt1, base_dt2], out_file=out_f, out_unit='%', callback=callback.defaultCallback2)
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/icclim.py", line 523, in indice
    save_percentile=save_percentile) ## tuple: (dt_centroid_arr, dt_bounds_arr, indice_arr) 
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/icclim.py", line 962, in get_indice_from_dict_temporal_slices
    bootstrapping=bootstrapping)
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/calc_percentiles.py", line 382, in get_percentile_dict
    percentile_array = return_perc_array_2_compute_bootstrapping(dt_arr, arr_filled, t_units, t_calendar, ytd, window_width, percentile, interpolation, ignore_Feb29th, only_leap_years, bootstrapping)
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/calc_percentiles.py", line 298, in return_perc_array_2_compute_bootstrapping
    percentile_array[i,:,:] = np.percentile(arr_filled[ind_2_calc,:,:], percentile, axis=0, interpolation=interpolation)
IndexError: index 3652 is out of bounds for axis 0 with size 3652
@larsbuntemeyer larsbuntemeyer changed the title Example 3 does not run with Python 3.6.7 Example 3 does not run with Python 3 Dec 11, 2019
@pagecp
Copy link
Collaborator

pagecp commented Dec 16, 2019

Thanks for this issue report and sorry for the delayed response.
I will try to reproduce the bug. Could you provide me the input file?

@larsbuntemeyer
Copy link
Author

Hi, thanks a lot! I actually use original cmip5 data, like the file in the original example. I will try to provide that data.

@larsbuntemeyer
Copy link
Author

larsbuntemeyer commented Dec 16, 2019 •
edited
Loading

Here is the data download: https://swift.dkrz.de/v1/dkrz_ffd3ca9004324ad28243244b834f92b1/tas_mpi/eddy/pool/sims/cmip5/MPI-ESM-LR/day/tas/tas_day_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc?temp_url_sig=5b5aacfa7594d9a44756ebffafe1f2b086106388&temp_url_expires=2019-12-23T16:35:17Z

Here is my script

import os
import datetime

from icclim import icclim

import logging

form= '%(asctime)s | %(levelname)8s | %(module)10s | %(funcName)20s | %(message)s'
logging.basicConfig(level=logging.DEBUG, format=form)

# base period
base_dt1 = datetime.datetime(1961,1,1)
base_dt2 = datetime.datetime(1970,12,31)

# studied period
dt1 = datetime.datetime(1980,1,1)
dt2 = datetime.datetime(2000,12,31)

out_f     = 'TG90p_year_MPI-ESM-LR_historical_r1i1p1_1980-2000.nc' # OUTPUT FILE: summer half-year values of TG90p
filenames = 'tas_day_MPI-ESM-LR_historical_r1i1p1_18500101-20051231.nc'

icclim.indice(indice_name='TG90p', in_files=filenames, var_name='tas', slice_mode='year', time_range=[dt1, dt2], base_period_time_range=[base_dt1, base_dt2], out_file=out_f, out_unit='%')

and here is the error:

2019-12-16 16:39:57,718 |     INFO | logging_info |        start_message |    ********************************************************************************************
2019-12-16 16:39:57,719 |     INFO | logging_info |        start_message |    *                                                                                          *
2019-12-16 16:39:57,720 |     INFO | logging_info |        start_message |    *          icclim                                                            V4.2.13       *
2019-12-16 16:39:57,720 |     INFO | logging_info |        start_message |    *                                                                                          *
2019-12-16 16:39:57,720 |     INFO | logging_info |        start_message |    *                                                                                          *
2019-12-16 16:39:57,720 |     INFO | logging_info |        start_message |    *          Mon Dec 16 16:39:57 2019 GMT                                                    *
2019-12-16 16:39:57,720 |     INFO | logging_info |        start_message |    *                                                                                          *
2019-12-16 16:39:57,720 |     INFO | logging_info |        start_message |    *          BEGIN EXECUTION                                                                 *
2019-12-16 16:39:57,720 |     INFO | logging_info |        start_message |    *                                                                                          *
2019-12-16 16:39:57,720 |     INFO | logging_info |        start_message |    ********************************************************************************************
2019-12-16 16:40:00,372 |     INFO |     icclim |               indice | Loading data...
2019-12-16 16:40:53,796 |     INFO |     icclim | get_indice_from_dict_temporal_slices | Daily Percentiles calculation for out-of-base years. Please be patient...
Traceback (most recent call last):
  File "tg10p.py", line 24, in <module>
    icclim.indice(indice_name='TG90p', in_files=filenames, var_name='tas', slice_mode='year', time_range=[dt1, dt2], base_period_time_range=[base_dt1, base_dt2], out_file=out_f, out_unit='%')
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/icclim.py", line 523, in indice
    save_percentile=save_percentile) ## tuple: (dt_centroid_arr, dt_bounds_arr, indice_arr) 
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/icclim.py", line 962, in get_indice_from_dict_temporal_slices
    bootstrapping=bootstrapping)
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/calc_percentiles.py", line 382, in get_percentile_dict
    percentile_array = return_perc_array_2_compute_bootstrapping(dt_arr, arr_filled, t_units, t_calendar, ytd, window_width, percentile, interpolation, ignore_Feb29th, only_leap_years, bootstrapping)
  File "/work/ch0636/g300046/conda_envs/py3/lib/python3.6/site-packages/icclim-4.2.13-py3.6.egg/icclim/calc_percentiles.py", line 298, in return_perc_array_2_compute_bootstrapping
    percentile_array[i,:,:] = np.percentile(arr_filled[ind_2_calc,:,:], percentile, axis=0, interpolation=interpolation)
IndexError: index 3652 is out of bounds for axis 0 with size 3652

It actually works with py2.7 and icclim 4.2.11

@pagecp
Copy link
Collaborator

pagecp commented Jan 3, 2020

Sorry for taking so long to reply.
I found the bug, but I am not sure yet how to fix it properly.
The bug happens if the first year of the period to be calculated is a non-leap year.
I will work on resolving this bug asap.

@larsbuntemeyer
Copy link
Author

thanks a lot!

@pagecp
Copy link
Collaborator

pagecp commented Mar 29, 2021

Please try the latest icclim commit. It was a very complicated bug to fix... It was due to the implementation of non-standard calendar year types for the percentile calculations. The implementation was partial and it is why it was not always working.
I leave the issue open until you confirm that it works on your side.

@pagecp pagecp mentioned this issue Mar 29, 2021
Closed
@bzah
Copy link
Member

bzah commented Jan 10, 2022

With version 5.0.0 of icclim, python 2 is no longer supported and the development was done with python 3.7 or newer.
I think we can safely close this issue, especially because the input file is no longer accessible.

However, it make me think that we should really run the examples in the CI, this would make sure each new PR run them and no example is broken. As often, we can take inspiration of what xclim does.

@bzah bzah closed this as completed Jan 10, 2022
@bzah bzah mentioned this issue Jan 10, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@larsbuntemeyer @pagecp @bzah