The US Environmental Protection Agency's NHDPlus dataset provides enhanced hydrologic attributes to the medium resolution National Hydrography Dataset. A core component of the dataset is the NHDPlus flow network. This network allows users to trace upstream or downstream for purposes of discovery, analysis or estimation. Several applications for NHDPlus navigation exist currently - each of which has subtle differences from one another that may be confusing or complicating for users seeking to do their own comparative analysis or write their own navigation logic.
This repository project seeks to straight-forwardly categorize the various products and document their differences making, at least, an attempt to put forward solid example logic using the PostgreSQL database. While a portion of this analysis is based on my own solid experiences over the years with navigation, some is by necessity conjecture. Please enter a GitHub issue if you find anything patently incorrect. And as always if you need clarification or additional background please consult EPA directly via nhdplus-support@epa.gov.
At present there are a number of applications and services which allow users to traverse the NHDPlus stream network. For this project I am proposing a rough division into three main branches: "Classic", "NLDI" and my own hybrid, "WATERS v3.0". The term "Classic" is entirely my own and meant without irony and with respect. I chose this tag over the competing "legacy" term used for the older logic by the NLDI project. Your feedback is most welcome if you have better terminology.
Classic Navigation has existed as a desktop GIS tool now for more than dozen years. I do not actually know when the logic was first released, sometime in the early 2000s seems a solid guess. The code was written in .Net by EPA contractors as an ArcGIS Desktop add-in using SQL Server Express LocalDB and released by EPA as the NHDPlus VAA Navigator Toolbar. The current version requires ArcGIS Desktop 10.2.x.
Classic Navigation logic is also a core component of the NHDPlus BasinDelineator Tool. The current version of this tool requires ArcGIS Desktop 10.5.1 and the Esri Spatial Analyst extension.
Around 2009 EPA ported the Classic logic into Oracle to drive services under the Watershed Assessment, Tracking & Environmental Results System (WATERS). The result was the Navigation Service and derivative services including the Navigation Delineation Service and Upstream Downstream Search Service. These public services are now almost a decade old providing navigation and delineation functionality to a wide variety of users and EPA applications.
Then around 2014 EPA began investigating the potential of utilizing PostgreSQL/PostGIS in the cloud. As part of this initiative the navigation logic was ported as-is into PL/pgSQL to drive ArcGIS Server hosted geoprocessing services. While these services were never released officially they existed for many years for evaluation purposes.
While I have never seen the source code for the desktop tools, I contend through experience that the Classic logic remains very similar or identical between the three aforementioned code bases over the past decade. If you believe otherwise please drop me a line.
Beginning around 2016 USGS spearheaded the creation of the Hydro Network Linked Data Index (NLDI). As a core component of the system is network navigation, the NLDI team under Dave Blodgett at USGS reimplemented NHDPlus network navigation. While Classic logic is still available within NLDI when using the legacy=true parameter, the NLDI team sought to leverage PostgreSQL recursion techniques and more powerful cloud servers via a full logic rewrite.
The resulting NLDI services use a PostgreSQL backend with a Java middleware component. The source code is hosted on GitHUB. Special attention should be paid to the navigation query SQL.
The NLDI team made several design decisions that speed up the network calculations over the Classic implementation. For example NLDI removes all measure evaluation meaning navigation occurs from and to whole flowlines only. Overall the NLDI rewrite is blazingly fast compared head-to-head with Classic navigation - particularly for small to medium sized navigations. However that same comparison will expose some differences that perhaps need elaboration. In addition I have found some scaling problems with the existing NLDI logic that perhaps could be surmounted with larger and larger hardware - but that I and many NHDPlus users do not have.
In the summer of 2018 I finally dove headlong into a prototype that I hoped would take the blazingly fast parts of NLDI but reinject some of the details of Classic navigation and allow better scaling on small hardware. Thus this third logic is provided part of the project repository for review and discussion. Over the past six months I have done a lot of comparisons between the three approaches - sometimes raising thorny questions in the process. As a result I have been very pleased with the new logic's performance and believe the logic is solid and defendable. So much so that v3.0 is currently driving the geoprocessing services behind EPA's WATERS GeoViewer. But the intention is not just to confuse the situation with yet another set of logic but rather to document each difference carefully perhaps with an eye towards eventually bringing all three logic families closer together.
While outside the scope of this project there are some other navigators out there. One such alternative is the long-forgotten version 2.0 from around 2013. The idea was to entirely rebase NHDPlus Navigation within the Oracle Network Data Model. While this resulted in a working prototype, the system simply does not scale as results are required to be marshaled in database memory. So modest navigations work well, but the Mississippi will just run the database out of memory.
- NHDPlus Source Datasets
- Classic Navigation logic
- NLDI Navigation logic
- WATERS Navigation v3.0 logic
- Test Cases
- Difference Summary
- I have avoided listing any contact information of any of the principle stakeholders in the navigators. If however you would like me to directly reference you here just drop me a line and I will add you in.