Document method to obtain NWM data from GCP

[jarq6c]

This ticket serves as a discussion area for downloading National Water Model data from Google Cloud Platform using hydrotools.

Installation

Assuming a UNIX-like environment, execute the following commands. The primary outcome should be a Python virtual environment with the hydrotools.nwm_client[gcp] package installed from PyPI.

$ python3 -m venv env
$ source env/bin/activate
(env) $ python3 -m pip install -U pip wheel
(env) $ python3 -m pip install hydrotools.nwm_client[gcp]
(env) $ touch main.py

Basic Example

Open main.py in your preferred editor and add the following code:

# Import the NWM Client for Google Cloud Platform
from hydrotools.nwm_client import gcp as nwm
import pandas as pd

# Instantiate model data service
model_data_service = nwm.NWMDataService()

# Create a location mapping with your desired comid and 
#  fake USGS site codes
#  Note: This is a work around until we fix the USGS requirement
my_mapping = pd.DataFrame({
    "nwm_feature_id": [101, 179],
    "usgs_site_code": ["my_site_a", "my_site_b"]
}).set_index("nwm_feature_id")
model_data_service.crosswalk = my_mapping

# Retrieve forecast data
forecast_data = model_data_service.get(
    configuration = "short_range",
    reference_time = "20220126T01Z"
    )

# Look at the data
print(forecast_data.info(memory_usage='deep'))
print(forecast_data[[
    'value_time',
    'value',
    'nwm_feature_id',
    'measurement_unit',
    'usgs_site_code',
    'variable_name',
    'configuration',
    'reference_time'
]].head(100))
count_row = len(forecast_data.index)  # Gives number of rows
count_col = len(forecast_data.columns)  # Gives number of columns
print("Total Rows: " + str(count_row))
print("Total Columns: " + str(count_col))

Run the script using:

(env) $ python3 main.py

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 36 entries, 0 to 35
Data columns (total 8 columns):
 #   Column            Non-Null Count  Dtype         
---  ------            --------------  -----         
 0   reference_time    36 non-null     datetime64[ns]
 1   value_time        36 non-null     datetime64[ns]
 2   nwm_feature_id    36 non-null     int64         
 3   value             36 non-null     float32       
 4   usgs_site_code    36 non-null     category      
 5   configuration     36 non-null     category      
 6   measurement_unit  36 non-null     category      
 7   variable_name     36 non-null     category      
dtypes: category(4), datetime64[ns](2), float32(1), int64(1)
memory usage: 2.0 KB
None
            value_time  value  nwm_feature_id measurement_unit usgs_site_code variable_name configuration      reference_time
0  2022-01-26 02:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
1  2022-01-26 03:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
2  2022-01-26 04:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
3  2022-01-26 05:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
4  2022-01-26 06:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
5  2022-01-26 07:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
6  2022-01-26 08:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
7  2022-01-26 09:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
8  2022-01-26 10:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
9  2022-01-26 11:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
10 2022-01-26 12:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
11 2022-01-26 13:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
12 2022-01-26 14:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
13 2022-01-26 15:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
14 2022-01-26 16:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
15 2022-01-26 17:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
16 2022-01-26 18:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
17 2022-01-26 19:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
18 2022-01-26 02:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
19 2022-01-26 03:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
20 2022-01-26 04:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
21 2022-01-26 05:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
22 2022-01-26 06:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
23 2022-01-26 07:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
24 2022-01-26 08:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
25 2022-01-26 09:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
26 2022-01-26 10:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
27 2022-01-26 11:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
28 2022-01-26 12:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
29 2022-01-26 13:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
30 2022-01-26 14:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
31 2022-01-26 15:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
32 2022-01-26 16:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
33 2022-01-26 17:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
34 2022-01-26 18:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
35 2022-01-26 19:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
Total Rows: 36
Total Columns: 8

[jarq6c]

Added with #179