pipdb: a simple library for interacting with Precipitation Imaging Probe datasets, maintained by Fraser King
pipdb is a simple query API for parsing, visualizing and performing particle size distribution calculations for Precipitation Imaging Package (PIP) data stored on DeepBlue. For additional details regadring the processing of the DeepBlue data from raw PIP files into NetCDF-4 format, please see the associated GitHub repository: PIP_Processing.
This project is currently being used for a journal article submitted to AGU's Earth and Space Science.
To test the capabilities of this library yourself before installing locally, check out our interactive notebook on Google Colab below.
To install this package on your system:
-
Clone of this GitHub repository:
git clone https://github.com/frasertheking/pipdb/ -
Create a conda environment, and install the required package dependencies using conda:
conda env create -f pipdb.yml -
Activate the pipdb environment:
conda activate pipdb -
Install the package so you can use it anywhere on your system:
python setup.py install
With the package installed, you can now import it into any of your scripts using:
import pipdb
While basic, this API handles the reading and visualizing of many common particle size distribution (PSD) parameters of interest. It loads NetCDF data into a standard xarray.Dataset object that can then be interacted with however you see fit.
More specifically, this API allows users to eaily:
- Load PIP data from NetCDF into xarray (single day, full year, multi-year)
- Plot site locations
- Extract individual variables of interest
- Print general statistics for each of the included dataset variables
- Curve fit PSD parameters
- Plot PSD variables of interest (1D and 2D)
- Plot mean PSD variables over time
- Separate dataset into rain and snow
- Compare between original and adjusted L4-derived products
We include an example interactive notebook in the examples folder which shows how to perform each the of aforementioned capabilities for some example data. For example:
Generating daily quicklooks:
Calculating mean statistics:
A Comprehensive Northern Hemisphere Particle Microphysics Dataset from the Precipitation Imaging Package
The data for this project is hosted online on UM's DeepBlue repository.
We have collected PIP microphysical data from a variety of measurement locations across the northern hemisphere. Data originally in a proprietary ASCII format has been converted to the more universally recognized NetCDF-4 format for ease of sharing and compatibility within the academic community. The conversion process, undertaken using a combination of bash and Python, ensures broader compatibility with various data analysis tools and platforms. A quality assurance (QA) procedure has been undertaken to ensure the integrity of the data. Post QA, the data is transformed into daily NetCDF-4 files following the Climate and Forecast (CF) conventions (version 1.10) and compressed with a level 2 deflation for optimized file size. Additional details into the data curation process can be found in our journal article publication.
For a brief overview of the data study sites and coverage periods, please see the figure below.
For additional API documentation, please see our docs page: https://pipdb.readthedocs.io/
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Note that, as a living project, code is not as clean as it could (should) be, and unit tests need to be produced in future iterations to maintain stability.
- Fraser King, University of Michigan, kingfr@umich.edu
- Claire Pettersen, University of Michigan
This project was primarily funded by NASA New (Early Career) Investigator Program (NIP) grant at the University of Michigan.