-
Notifications
You must be signed in to change notification settings - Fork 7
/
01-intro-deep-dive.Rmd
135 lines (87 loc) · 6.78 KB
/
01-intro-deep-dive.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
---
title: "Introduction"
description: |
"NOAA's Enterprise Hydrofabric System"
author:
- name: "Mike Johnson"
url: https://github.com/mikejohnson51
affiliation: Lynker, NOAA-Affiliate
affiliation_url: https://lynker.com
output:
distill::distill_article:
toc: true
toc_depth: 2
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, out.width = "100%", message = TRUE)
library(sf); library(mapview)
```
# What is a hydrofabric?
The first question generally raised is, "_what is a hydrofabric_?" To date, the term has been been used to describe artifacts as narrow as a set of cartographic lines, all the way to the entire spatial data architecture needed to map and model the flow of water and flood extents.
For our purposes here, the hydrofabric is the base data that allows NextGen to run. It provides:
- the landscape and flow network _discritizations_
- the _connectivity_ (topology) of the network features
- the locations _where information will be reported_ (nexus's)
- the divide and flowpath attributes needed to support hydrologic modeling and routing
The hydrofabric also establishes a system of _linked data_ and Web infrastructure that can relate to, and extract from, linked sources like:
- the USGS Next Generation Monitoring Location Pages (e.g. [here](https://waterdata.usgs.gov/monitoring-location/06752260/))
- The Internet of Water Geoconnex PID registry (e.g. [here](https://reference.geoconnex.us/collections))
- climate-catalogs (e.g. [here](https://github.com/mikejohnson51/climateR-catalogs))
- landscape characteristic catalogs
```{r, echo = FALSE}
knitr::include_graphics('../man/figures/what-is-a-hydrofabric.png')
```
## Who cares about a hydrofabric?
Discritizing the land surface into computational elements is fundamental to all modeling tasks. Without it, distributed and lumped models have no way to apply the needed model formulations or computer science applications to achieve meaningful results. Therefore anyone who cares about the science and application of water resource modeling should care about the underlying data as it drives the locations where forecasts are made, the attributes that inform a model, and the spatial elements in which formulation are valid.
However, describing the earths surface - particularly at continental scales - is a tricky task. Automated techniques can get us a long way in representation, however the modeling task at hand and local knowledge should be used in developing an authoritative product. Through time, local knowledge has been collected in a number of places, but never centralized. Further, one off products (like the NHDPlus) have been used to guide all modeling task even in cases when its resolution, or representation is not well suited.
The aim of NOAAs work in this space is to develop a federal reference fabric to support all flavors of modeling, and a national instance of that reference fabric to support heterogeneous model application.
Equally important is the software tools to support flexibility and community uptake; the data models to support interoperability, community engagement, and long term stability; and a reference data set with the quality assurances that when one uses the product they are getting a well vetted resource that will be able to play nicely with the growing Ngen framework.
# Current Version:
The most up to date NextGen hydrofabric and resources can be accessed from the public facing [Lynker Spatial account](https://www.lynker-spatial.com/index.html).
In practice we strive to develop these products to take advantage of the following:
### Leading data science
#### Formats:
- [GPKG](https://www.geopackage.org)
- Arrow/[Parquet](https://parquet.apache.org/docs/overview/) Tables
#### Access Patterns:
- [GDAL VSI](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwil-fWKx_D-AhWCIzQIHXIHDD4QwqsBegQIAxAE&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DauK_gPR-e7M&usg=AOvVaw2ITVtXkwdDj5PCzIfSQwbW)
- [SQLITE](https://sqlite.org/index.html)
- [Arrow](https://arrow.apache.org)
#### Distribution System
- [s3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) (through AWS)
### Hydroscience Conceptual Models & Web Infastrucutre
- The hydrofabric features are grounded in the OGC [HY Feature conceptual model](https://docs.opengeospatial.org/is/14-111r6/14-111r6.html).
```{r, echo = FALSE}
knitr::include_graphics('../man/figures/hyfeature.png')
```
- The [Mainstems logical model](https://www.sciencedirect.com/science/article/pii/S1364815220309841)
- The OGC Engineering Report ["Hydrologic Modeling and River Corridor Applications of HY_Features Concepts"](https://docs.ogc.org/per/22-040.html)
- The Network Linked Data Index (NLDI) [here](https://waterdata.usgs.gov/blog/nldi-intro/), and [here](https://rconnect.usgs.gov/dataRetrieval/articles/nldi.html) and [here](https://hyriver.readthedocs.io/en/latest/autoapi/pynhd/pynhd/index.html)
- The [Geoconnex Registry](https://geoconnex.internetofwater.dev)
- The lynker-spatial [hydrolocation inventory]
### Formal Realization Representations
- The HY Features conceptual model is conflated with the [Simple Feature Access](https://www.ogc.org/standard/sfa/) Spatial Data model to provide a logical model for how the feature realizations are represented in the hydrofabric data model.
```{r, echo = FALSE}
knitr::include_graphics('../man/figures/ogc_sfs.png')
```
# Software
Extracting subsets from the primary hydrofabric data product does not require code, however, if you are eager to build, modify and expand on the existing products the
```{r, eval = FALSE}
remotes::install_github("NOAA-OWP/hydrofabric")
```
Hydrofabric provides a easy install for a variety of hydroscience, data science, and spatial libraries that are needed.
Attaching this library, similar to the `tidyverse`, installs and loads a set of software designed to manipulate, modify, describe, process, and quantify hydrologic networks and land surface attributes:
```{r}
library(hydrofabric)
```
`library(hydrofabric)` will load the core packages (alphabetical):
* [climateR](https://github.com/mikejohnson51/climateR) for accessing federated data stores for parameter and attributes estimation
* [hfsubsetR](https://github.com/lynker-spatial/) for cloud-based hydrofabric subsetting
* [hydrofab](https://github.com/mikejohnson51/hydrofab) a tool set for "fabricating" multiscale hydrofabrics
* [ngen.hydrofab](https://github.com/mikejohnson51/ngen.hydrofab) NextGen extensions for hydrofab
* [nhdplusTools](https://github.com/doi-usgs/nhdplusTools/) for network manipulation
* [zonal](https://github.com/mikejohnson51/zonal) for catchment parameter estimation
Additionally it will load key geospatial data science libraries:
* `dplyr` (data.frames)
* `sf` (vector)
* `terra` (raster)