CONUS Site Data Provider by abhatthal · Pull Request #228 · opensha/opensha

abhatthal · 2026-05-08T20:33:17Z

This PR adds support for USGS basin and sediment depth data (#205) via a new Site Data Provider retrieving NSHM18 and NSHM23 site data for the Conterminous U.S.
The data is retrieved from the USGS GitLab and doesn't use servlets (e.g., like with the AbstractBinarySiteDataLoader)

Overview

VersionResolver: Interface to map git tags to version of site data models
CONUS_Versions: Enumeration (implements VersionResolver) that maps NSHM models for CONUS
AbstractGitLabDownloader: Retrieves data from GitLab at a specified VersionResolver using a URL provided by concrete implementations
CONUS_Downloader: Extends AbstractGitLabDownloader to download CONUS site data if not already downloaded
AbstractGriddedSiteDataLoader: Downloads site data as needed using an AbstractGitLabDownloader and then loads the Regions and site data mapping into memory
GriddedSiteDataUtil: Utility class for working with gridded data (i.e. logic from nshmp-haz for snapping locations to a grid)
CONUS_SiteDataProvider: Extends AbstractGriddedSiteDataLoader to load data with a CONUS_Downloader, defines the applicable region for the provider, and provides adjustable parameters for selecting the hazard model.

Integration

Created a new SiteData type for Sediment thickness (Zsed) and created an entry in the SiteTranslator to map to the existing Sediment Thickness parameter
Created instances of the CONUS_SiteDataProvider in the OrderedSiteDataProviderList for retrieval of Z1.0, Z2.5, and Zsed.
Tested behavior of CONUS_Downloader manually with a CLT main function.
Validated site data retrieval across IM Event Set Calculator and Hazard Curve Application (matches site data in CSV files) for locations snapped within gridded spacing and inside the corresponding Regions

…S GitLab

… generic failure to read.

…te data loader

…load data if found in a gridded region

…list javadoc

…location on the grid.

…rieval

kevinmilner · 2026-05-08T23:17:49Z

Two things so far:

Why take in minLat, maxLat, minLon, maxLon, then build a region from it for getApplicableRegion()? Why not just take in the region in the constructor and return that? Is there a reason to convert the original region into a rectangular one defined by the extrema?

I'm not yet seeing the benefit of loading the GeoJSON to GriddedRegion's over Region's, aren't you just using them for Region.contains(Location) which isn't specific to a GriddedRegion?

Done. Now uses applicableRegion directly.

kevinmilner · 2026-05-08T23:20:54Z

You'll also need to add the mapping in setParameterValue(Parameter, Collection<SiteDataValue<?>>) to do the actual parameter translation.

Done. Confirmed Zsed available via SiteDataCombinedApp.

kevinmilner · 2026-05-12T17:59:33Z

Please change the default store dir from ~/.site_data to ~/.opensha/site_data, since we already use ~/.opensha to store other cached data.

kevinmilner · 2026-05-12T18:05:23Z

I cannot get this to successfully download, have you run into any similar issues? I'm running the main method of CONUS_Downloader with the NSHM23 command line argument. These are the errors I have gotten:

This many times:

Downloading data for NSHM (2023) from https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
May 12, 2026 10:54:47 AM org.opensha.commons.data.siteData.AbstractGitLabDownloader downloadSiteData
SEVERE: e:
java.io.IOException: Server returned HTTP response code: 406 for URL: https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2000)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at java.base/java.net.URL.openStream(URL.java:1161)
	at org.opensha.commons.data.siteData.AbstractGitLabDownloader.downloadSiteData(AbstractGitLabDownloader.java:229)
	at org.opensha.commons.data.siteData.CONUS_Downloader.main(CONUS_Downloader.java:93)


May 12, 2026 10:54:47 AM org.opensha.commons.data.siteData.AbstractGitLabDownloader downloadSiteData
SEVERE: Failed to download data for NSHM (2023) at https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
The GitLab server must be down for maintenance. Please try again later.

...and this once after a long pause:

Downloading data for NSHM (2023) from https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
May 12, 2026 10:57:25 AM org.opensha.commons.data.siteData.AbstractGitLabDownloader downloadSiteData
SEVERE: e:
java.net.ConnectException: Connection timed out
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:579)
	at java.base/sun.nio.ch.Net.connect(Net.java:568)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
	at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:174)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:183)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:533)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:638)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
	at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at java.base/java.net.URL.openStream(URL.java:1161)
	at org.opensha.commons.data.siteData.AbstractGitLabDownloader.downloadSiteData(AbstractGitLabDownloader.java:229)
	at org.opensha.commons.data.siteData.CONUS_Downloader.main(CONUS_Downloader.java:93)


May 12, 2026 10:57:25 AM org.opensha.commons.data.siteData.AbstractGitLabDownloader downloadSiteData
SEVERE: Failed to download data for NSHM (2023) at https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
The GitLab server must be down for maintenance. Please try again later.

Meanwhile, I can download the file fine in browser or via wget without error, which works robustly while the Java downloader is failing every time. It's not just bad timing, I keep trying both in java and outside of java and the behavior is consistent.

kevin@steel:/tmp$ wget "https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data"
--2026-05-12 11:02:26--  https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
Resolving code.usgs.gov (code.usgs.gov)... 137.227.229.60, 2001:49c8:0:126c::241
Connecting to code.usgs.gov (code.usgs.gov)|137.227.229.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1085624 (1.0M) [application/zip]
Saving to: ‘nshm-conus-6.1.3.zip?ref_type=tags&path=site-data’

…of coordiante extrema.

… the Accept header explicitly.

abhatthal · 2026-05-12T21:15:20Z

I cannot get this to successfully download, have you run into any similar issues? I'm running the main method of CONUS_Downloader with the NSHM23 command line argument. These are the errors I have gotten:

This many times:

Downloading data for NSHM (2023) from https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
May 12, 2026 10:54:47 AM org.opensha.commons.data.siteData.AbstractGitLabDownloader downloadSiteData
SEVERE: e:
java.io.IOException: Server returned HTTP response code: 406 for URL: https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2000)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at java.base/java.net.URL.openStream(URL.java:1161)
	at org.opensha.commons.data.siteData.AbstractGitLabDownloader.downloadSiteData(AbstractGitLabDownloader.java:229)
	at org.opensha.commons.data.siteData.CONUS_Downloader.main(CONUS_Downloader.java:93)


May 12, 2026 10:54:47 AM org.opensha.commons.data.siteData.AbstractGitLabDownloader downloadSiteData
SEVERE: Failed to download data for NSHM (2023) at https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
The GitLab server must be down for maintenance. Please try again later.

...and this once after a long pause:

Downloading data for NSHM (2023) from https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
May 12, 2026 10:57:25 AM org.opensha.commons.data.siteData.AbstractGitLabDownloader downloadSiteData
SEVERE: e:
java.net.ConnectException: Connection timed out
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:579)
	at java.base/sun.nio.ch.Net.connect(Net.java:568)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
	at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:174)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:183)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:533)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:638)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
	at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at java.base/java.net.URL.openStream(URL.java:1161)
	at org.opensha.commons.data.siteData.AbstractGitLabDownloader.downloadSiteData(AbstractGitLabDownloader.java:229)
	at org.opensha.commons.data.siteData.CONUS_Downloader.main(CONUS_Downloader.java:93)


May 12, 2026 10:57:25 AM org.opensha.commons.data.siteData.AbstractGitLabDownloader downloadSiteData
SEVERE: Failed to download data for NSHM (2023) at https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
The GitLab server must be down for maintenance. Please try again later.

Meanwhile, I can download the file fine in browser or via wget without error, which works robustly while the Java downloader is failing every time. It's not just bad timing, I keep trying both in java and outside of java and the behavior is consistent.

kevin@steel:/tmp$ wget "https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data"
--2026-05-12 11:02:26--  https://code.usgs.gov/ghsc/nshmp/nshms/nshm-conus/-/archive/6.1.3/nshm-conus-6.1.3.zip?ref_type=tags&path=site-data
Resolving code.usgs.gov (code.usgs.gov)... 137.227.229.60, 2001:49c8:0:126c::241
Connecting to code.usgs.gov (code.usgs.gov)|137.227.229.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1085624 (1.0M) [application/zip]
Saving to: ‘nshm-conus-6.1.3.zip?ref_type=tags&path=site-data’

Fixed. I couldn't replicate the issue on macOS, but I tried on Expanse and found the 406 bug. I explicitly add client headers to Accept / and follow redirects with copied headers and it works now using OpenJDK 17.

…sediment-depth-data-retrieval

kevinmilner

Looks good, and no issues downloading now!

abhatthal added 14 commits April 8, 2026 17:58

Create CONUS downloader to retrieve NSHM site data files from the USG…

3a90e2c

…S GitLab

Add more robust extraction failure logging for zip file corruption or…

e9f9cb9

… generic failure to read.

Create interface for resolving git version tags in CONUS_Versions

b255b71

Abstract the CONUS_Downloader logic into an AbstractGitLabDownloader

2309cd1

Create Sediment Thickness (zSed) site data type and mapping

b1d9dcf

Update GitLab downloaders to use NIO Path instead of File.

81004a8

Create outline for abstract site data loader

1e3ddee

Implement getClosestDataLocation and getValue for abstract gridded si…

edb2c64

…te data loader

Fix AbstractGriddedSiteDataLoader to load site data at construction

d1753f8

Create concrete base class

d4c5c36

Fix concrete class to snap to grid for corresponding region and only …

306e3e0

…load data if found in a gridded region

Remove duplicate CCA providers and update default site data provider …

34c677b

…list javadoc

getValue checks if the region contains the location, not the closest …

0417ea1

…location on the grid.

Merge branch 'master' into feature/usgs-basin-sediment-depth-data-ret…

f37db91

…rieval

abhatthal requested a review from kevinmilner May 8, 2026 20:34

kevinmilner reviewed May 8, 2026

View reviewed changes

abhatthal added 2 commits May 11, 2026 17:42

Set SedimentThicknessParam with site data values in SiteTranslator

0e4adde

Fix typos

a0b59a3

kevinmilner reviewed May 12, 2026

View reviewed changes

abhatthal added 2 commits May 12, 2026 13:11

Save site data under .opensha. Use applicableRegion directly instead …

3e2ad51

…of coordiante extrema.

Fix GitLab downloader 406 error. Now follows redirects and propagates…

eca2f6c

… the Accept header explicitly.

Merge remote-tracking branch 'origin/master' into feature/usgs-basin-…

49229a5

…sediment-depth-data-retrieval

kevinmilner approved these changes May 13, 2026

View reviewed changes

Updated list numbers for OrderedSiteDataProviderList javadoc.

4bd68ae

abhatthal merged commit bcc1e0b into opensha:master May 13, 2026
2 checks passed

abhatthal deleted the feature/usgs-basin-sediment-depth-data-retrieval branch May 13, 2026 16:31

Navigation Menu

Uh oh!

CONUS Site Data Provider - #228

CONUS Site Data Provider#228
abhatthal merged 20 commits into
opensha:masterfrom
abhatthal:feature/usgs-basin-sediment-depth-data-retrieval

abhatthal commented May 8, 2026

Uh oh!

kevinmilner May 8, 2026

Uh oh!

abhatthal May 12, 2026

Uh oh!

kevinmilner May 8, 2026

Uh oh!

abhatthal May 12, 2026

Uh oh!

kevinmilner May 12, 2026

Uh oh!

abhatthal May 12, 2026

Uh oh!

kevinmilner commented May 12, 2026

Uh oh!

abhatthal commented May 12, 2026

Uh oh!

kevinmilner left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

abhatthal commented May 8, 2026

Overview

Integration

Uh oh!

kevinmilner May 8, 2026

Choose a reason for hiding this comment

Uh oh!

abhatthal May 12, 2026

Choose a reason for hiding this comment

Uh oh!

kevinmilner May 8, 2026

Choose a reason for hiding this comment

Uh oh!

abhatthal May 12, 2026

Choose a reason for hiding this comment

Uh oh!

kevinmilner May 12, 2026

Choose a reason for hiding this comment

Uh oh!

abhatthal May 12, 2026

Choose a reason for hiding this comment

Uh oh!

kevinmilner commented May 12, 2026

Uh oh!

abhatthal commented May 12, 2026

Uh oh!

kevinmilner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants