Not able to download 2022 data

[thisisashukla]

Hi Saswata, thanks for creating this library. This work adds a lot of value to my work. However I am stuck with an issue and your help would really be appreciated.
I am trying to download the data for 2022 and getting the following error

Downloading: rain for year 2022
Download Successful !!!
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Input In [1], in <module>
      4 end_yr = 2022
      5 variable = 'rain' # other options are ('tmin'/ 'tmax')
----> 6 data = imd.get_data(variable, start_yr, end_yr, fn_format='monthwise')

File /opt/conda/lib/python3.9/site-packages/imdlib/core.py:497, in get_data(var_type, start_yr, end_yr, fn_format, file_dir, sub_dir, proxies)
    493             f.write(response.content)
    495     print("Download Successful !!!")
--> 497     data = open_data(var_type, start_yr, end_yr, fn_format, file_dir)
    498     return data
    500 except requests.exceptions.HTTPError as e:

File /opt/conda/lib/python3.9/site-packages/imdlib/core.py:327, in open_data(var_type, start_yr, end_yr, fn_format, file_dir)
    325 # Check consistency of data points
    326 if len(data) != nlen:
--> 327     raise Exception("Error in file reading,"
    328                     "mismatch in size of data-length")
    330 # Reshape data into a shape of
    331 # (days_in_year, lon_size_class, lat_size_class)
    332 data = np.transpose(np.reshape(data, (days_in_year, lat_size_class,
    333                                       lon_size_class), order='C'), (0, 2, 1))

Exception: Error in file reading,mismatch in size of data-length```

[thisisashukla]

The following code can be used to reproduce the error:

import imdlib as imd

start_yr = 2021
end_yr = 2022
variable = 'rain' # other options are ('tmin'/ 'tmax')
data = imd.get_data(variable, start_yr, end_yr, fn_format='yearwise')

[thisisashukla]

i checked the code and found out that the value of len(data) (line number 326) is coming out to be 0 for 2022 data.

[iamsaswata]

Hi @thisisashukla.
Thanks for your kind words and reaching out to me.

I am happy you reported the full code-block. This error is not related to imdlib, but due to the unavailability of 2022 IMD data at this point in time.

If you look into the downloaded data after running your code (in the created folder 'rain'), you should see two files, namely 2021.grd and 2022.grd. If you look into the properties of these two files, you will notice 2021.grd has a size of ~25mb but 2022.grd has a size of ~0 mb. Why 2022.grd has no data? Because it is not released yet from IMD (yearly archive version).

Unless the data is added in the imd server, you should use the "max year" as 2021 in your program.

[thisisashukla]

Thanks for your response @iamsaswata, after raising the issue I took a dive in the code and reached the same conclusion mentioned by you. I am able to see that https://imdpune.gov.in/lrfindex.php this website has an option to download daily rainfall data.
May be you can look at how can you enable pulling data day wise and then aggregating it for currently running year. Any practical use case would most probably need the latest data. Please consider this enhancement if you have the necessary bandwidth at your end.

[iamsaswata]

Appreciate your suggestion to the fullest. It looks like you are talking about the real-time data from the mentioned site. In this regard, I am happy to tell you it is already in the pipeline , and already discussed in the imdlib issues few weeks back, more regarding it can be found in #17 and #18 (if you are interested).

And just an endnote to this, and perhaps not related to imdlib, which is I dont believe 'any practical use cases' must have or always need real-time/latest data. But surely they do come handy in different use cases.