ADD: Auto Read Function

[syedhamidali]

• Closes #xxxx
• Tests added
• Changes are documented in history.md

Description: Enhanced Radar File Reading Functionality with Timeout Support

Summary:

This PR introduces the auto_read.py module, which provides an enhanced read function designed to improve the radar file reading process in the xradar library. The read function iterates over all available file-opening functions within xradar.io to attempt to read radar files, with added support for georeferencing and verbose output. A key feature of this function is its ability to handle timeouts, ensuring that long-running file reads do not block execution indefinitely.

Key Features:

• Automatic Format Detection: The read function attempts to open the provided radar file using all available open_ functions in the xradar.io module, automatically selecting the correct file format.
• Timeout Support: The function now includes an optional timeout parameter. If specified, the file read operation is wrapped in a timeout handler, raising a TimeoutException if the read operation exceeds the allowed time.
• Georeferencing: The function can automatically georeference the radar data by adding x, y, z coordinates if the georeference parameter is set to True.
• Verbose Mode: A verbose option is available to print detailed processing information, aiding in debugging and providing insights into which file-opening method was successful.
• Custom Exception Handling: The function raises a ValueError if the file cannot be opened by any supported format, providing clear feedback in cases of unsupported file types or corrupted files.

Usage:

• Basic Usage:

  dtree = xd.io.read(file_path)

• With Timeout and Verbose Output:

  dtree =  xd.io.read(file_path, verbose=True, timeout=10)

Other tests

import os
from open_radar_data import DATASETS

# Dictionary to track which prefixes have been processed
prefix_processed = {}

# Iterate through the files in the registry
for file in DATASETS.registry_files:
#     print(file)

    # Skip files with .gz extension as they are not supported by xradar
    if file.endswith('.gz'):
        continue

    # Extract the prefix of the file (up to the first underscore, dot, or space)
    prefix = file.split('_')[0].split('.')[0].split()[0]

    # Check if a file from this prefix group has already been processed
    if prefix in prefix_processed:
        continue  # Skip the file if a file from this group has already been processed

    try:
        # Attempt to fetch the file path
        filepath = DATASETS.fetch(file)
        print(f"Reading: {os.path.basename(file)}")
        # Attempt to read the radar data
        radar = xd.io.read(filepath, timeout=10)
        
        # If no exception is raised, print success
        print('+++++++++++')
        print('| SUCCESS |')
        print('+++++++++++\n')

        # Mark this prefix as processed
        prefix_processed[prefix] = True

        # Delete the radar object after processing
        del radar

    except Exception as e:
        # Print the error and continue with the next file
        print(f"Error processing file {os.path.basename(file)}: \n{e}")
        print('+++++++++++')
        print('| FAILED |')
        print('+++++++++++\n')
        continue

Reading: 110635.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: 2013051000000600dBZ.vol
+++++++++++
| SUCCESS |
+++++++++++

Reading: 20220628072500_savevol_COSMO_LOOKUP_TEMP.nc
Error processing file 20220628072500_savevol_COSMO_LOOKUP_TEMP.nc: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: 71_20181220_060628.pvol.h5
+++++++++++
| SUCCESS |
+++++++++++

Reading: DES_VOL_RAW_20240522_1600.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: DWD-Vol-2_99999_20180601054047_00.h5
+++++++++++
| SUCCESS |
+++++++++++

Reading: KATX20130717_195021_V06
+++++++++++
| SUCCESS |
+++++++++++

Reading: KLBB20160601_150025_V06
+++++++++++
| SUCCESS |
+++++++++++

Reading: MLA2119412050U.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: MLL2217907250U.003.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: RAW_NA_000_125_20080411181219
+++++++++++
| SUCCESS |
+++++++++++

Reading: SUR210819000227.RAWKPJV
+++++++++++
| SUCCESS |
+++++++++++

Reading: T_PAGZ35_C_ENMI_20170421090837.hdf
+++++++++++
| SUCCESS |
+++++++++++

Reading: User1_184_20240601_013257.hpl
+++++++++++
| SUCCESS |
+++++++++++

Reading: Z__C_RJTD_20230801200000_RDR_JMAGPV_RS47937_Gar0p250km0p35deg_PRkdp_N11_ANAL_cfrad.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: cfrad.20080604_002217_000_SPOL_v36_SUR.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: convsf.19990811.221202.cdf
Error processing file convsf.19990811.221202.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: cor-main131125105503.RAW2049
+++++++++++
| SUCCESS |
+++++++++++

Reading: corcsapr2cmacppiM1.c1.20181111.030003.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: csapr.nc
Error processing file csapr.nc: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: ddop.910815.212303.cdf
Error processing file ddop.910815.212303.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: ddop.910815.213931.cdf
Error processing file ddop.910815.213931.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: ddop.910815.214454.cdf
Error processing file ddop.910815.214454.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: ddop.910815.215545.cdf
Error processing file ddop.910815.215545.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: example_plot_ppi_single_sweep.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: gucxprecipradarcmacppiS2.c1.20220314.021559.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: houkasacrcfrM1.a1.20210922.150006.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: sample_rainbow_5_59.vol
+++++++++++
| SUCCESS |
+++++++++++

Reading: sgpkasacrcrrasterC1.a1.20130419.012153.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: sgpkazrgeC1.a1.20190529.000002.cdf
Error processing file sgpkazrgeC1.a1.20190529.000002.cdf: 
File could not be opened by any supported format in xradar.io.
+++++++++++
| FAILED |
+++++++++++

Reading: sgpxsaprcfrvptI4.a1.20200205.100827.nc
+++++++++++
| SUCCESS |
+++++++++++

Reading: swx_20120520_0641.nc
+++++++++++
| SUCCESS |
+++++++++++

[codecov]

Codecov Report

Attention: Patch coverage is 93.33333% with 5 lines in your changes missing coverage. Please review.

Project coverage is 92.97%. Comparing base (ccc1c9d) to head (749fc5e).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
xradar/io/auto_read.py	93.24%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #205      +/-   ##
==========================================
+ Coverage   92.84%   92.97%   +0.12%     
==========================================
  Files          26       27       +1     
  Lines        5004     5079      +75     
==========================================
+ Hits         4646     4722      +76     
+ Misses        358      357       -1     

Flag	Coverage Δ
notebooktests	79.72% <49.33%> (+0.70%)	⬆️
unittests	91.94% <92.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[kmuehlbauer]

@syedhamidali Checking if a file can be opened with one of our backends and return the dataset/datatree is indeed a nice idea and a useful feature. Thanks for tackling this to make xradar more user-friendly.

For the implementation we might think about using the guess_can_open-method of each BackendEntrypoint.

We would have to implement such function for our backends first, but this would be a great opportunity to do this finally.

In the long run we should think about implementing xradar.open_dataset() and xradar.open_datatree() which do essentially the same as the xarray counterparts, but solely iterates over xradar backends (when engine=None). This would also remove the need of a new read-function.

@openradar/xradar More thoughts here?

[syedhamidali]

@kmuehlbauer That’s a great idea, especially in terms of aligning with the xarray model. I do have a few questions, though. What would you typically open as a dataset—would it be a single sweep? In our case, we often work with datatree as the file structure. I was thinking of implementing something like open_datatree and open_mfdatatree or read_volume, where the node names would follow a /volume_{}/sweep_{} pattern.

I’ve done something similar with the imd_data model in radarx. You can check out an example at radarx-rtd. Moving forward, the read_imd function will be integrated into the xradar package, and the radarx package itself will be contributed to the openradar community. I’ve been focusing on radarx primarily for visualization (both traditional and interactive) and potentially some microphysics retrievals, and I’d be happy to contribute these efforts.

[kmuehlbauer]

@syedhamidali Thanks for sharing your efforts here. Great to hear that the IMD backend will find it's way into xradar.

I like the idea behind radarx. Did you conceive the package more as companion to xradar being capable of things mentioned in #174? Or should it play in the same league as Py-ART, wradlib etc implementing algorithms for some specific purposes? Maybe split this out to it's own issue for better visibility and discussion?

[syedhamidali]

@kmuehlbauer Sorry for the late response, as I am currently at the AMS SLS conference. radarx will indeed act as a companion to xradar, addressing most of the features mentioned in that comment. However, I’m also considering adding additional functionality such as retrieving DSDs and performing thermodynamic calculations like the evaporative cooling rate etc. For this, we would need temperature profiles, which could be sourced from ERA5 data or nearby soundings.