Update for more robust scraping from NDBC websites #242

shinbunya · 2021-07-14T19:03:15Z

This makes the following changes for NDBC website scraping:

Adapt to non numerical station IDs.
Ignore unnecessary columns in the station list.
Ignore "Station not found" return.

krober10nd · 2021-07-14T22:16:44Z

Thanks! A quick test worked out nicely for the elevation data.

 m = Make_f15( m, '01-Jan-2018 03:00', '01-Feb-2018 22:00', 1.0, 'sta_database',{'CO-OPS',[1]}

producing something like this after writing a fort.15 file.

144  	 	 ! NSTAE 
-64.695000 32.370000 Bermuda Biological Station ID:2695535 
-64.703300 32.373300 Bermuda, St. Georges Island ID:2695540 
-66.982900 44.904600 Eastport ID:8410140 
-67.204700 44.657000 Cutler Farris Wharf ID:8411060 
-68.204300 44.392200 Bar Harbor ID:8413320 
-70.244200 43.658100 Portland ID:8418150 
-70.563300 43.320000 Wells ID:8419317 
-70.741100 43.079700 Seavey Island ID:8419870 
-71.050300 42.353900 Boston ID:8443970 
-71.164100 41.704300 Fall River ID:8447386 
-69.950800 41.688100 Chatham, Lydia Cove ID:8447435 
-70.671100 41.523600 Woods Hole ID:8447930 
-70.096400 41.285300 Nantucket Island ID:8449130 
-71.326100 41.504300 Newport ID:8452660 
.
.
.

However, if I want to record both elevation and velocity at various stations, it would only populate elevation stations.

 m = Make_f15( merged, '01-Jan-2018 03:00', '01-Feb-2018 22:00', 1.0, 'sta_database',{'CO-OPS',[1,2]}

and if I did just want velocity recording stations:

 m = Make_f15( merged, '01-Jan-2018 03:00', '01-Feb-2018 22:00', 1.0, 'sta_database',{'CO-OPS',[2]}

I got an error like this:

22  for sta_database = sta_databases
Error using xmlread (line 112)
Java exception occurred:
java.net.UnknownHostException: opendap.co-ops.nos.noaa.gov
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
	at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
	at
        sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        	at
        sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at
        sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        	at
        sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
	at
        sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
        	at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
	at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)

Error in tidal_stations_parser (line 46)
        DOMnode = xmlread(xml);

Error in Make_f15 (line 337)
    obj = tidal_stations_parser(obj,string(sta_database(1:2:end-1)),...

My other thought is perhaps we could get WriteFort15 to just write an external elev_stat.151 directly if asked?

Anyway thank you again.

shinbunya · 2021-07-15T00:10:06Z

I should've mentioned the following thing at the same time. I'm not sure if this is relevant to the error that @krober10nd mentioned, but it seems that the correct way to call the function is different from what the manual suggests.
In an example in the manual, we have
m = Make_f15( m, TS, TE, DT, 'tidal_database', TPXO9, 'const', ... {CONST},'sta_database',{'CO−OPS','NDBC',[1]} ) ;
However, the way I succeeded was
m = Make_f15( m, TS, TE, DT, 'tidal_database', TPXO9, 'const', ... {CONST},'sta_database',{'CO−OPS',1,'NDBC',2} ) ;
If you want to get met stations
m = Make_f15( m, TS, TE, DT, 'tidal_database', TPXO9, 'const', ... {CONST},'sta_database',{'CO−OPS',1,'NDBC',2,'NDBC',3} ) ;
This can be related to what you observed when you invoked Make_f15.

shinbunya · 2021-07-15T00:11:48Z

Also, as far as I could read from the source code, it didn't look like we can have both CO-OPS and NDBC for elevation stations (nor velocity/met stations). I may be wrong about this.

krober10nd · 2021-07-15T01:38:04Z

ah yes that makes more sense. We'll have to fix the user guide. By the way, I've already incorporated some of the changes on the Overleaf that you sent through but the appendix still remains out to date.

Coming back to this, when I try to add on NDBC data I get this (my mesh covers the entire WNAT so it should have some stations show up)

>> m = Make_f15( m, '01-Jan-2018 03:00', '01-Feb-2018 22:00', 1.0, 'sta_database',{'CO-OPS',1,'NDBC',3} );
Error using textscan
First input can not be empty. Expected a non-empty character vector or a valid file-id.

Error in tidal_stations_parser (line 110)
            C = textscan(NDBC,'%s','Delimiter','\t');

Error in Make_f15 (line 337)
    obj = tidal_stations_parser(obj,string(sta_database(1:2:end-1)),...

because on 107-108

 NDBC = urlread('http://www.ndbc.noaa.gov/stndesc.shtml');
  ii1 = strfind(NDBC,'<pre>'); ii2 = strfind(NDBC,'</pre>');

these indices can't seem to locate any strings with <pre> on that link for NDBC.

shinbunya · 2021-07-15T02:22:19Z

What do you get if you type in the following line in the matlab console?
NDBC = urlread('http://www.ndbc.noaa.gov/stndesc.shtml');
I get a html containg...

<pre>
NDBC owned meteorological buoys
WATCH ADRIFT PAYLOAD
BUOY BUOY LATITUDE LONGITUDE DEPTH RADIUS TYPE
STATION HULL DEG MIN SEC N/S DEG MIN SEC E/W (M) (YDS)
41001 3D90 34 46 30 N 72 10 2 W 4431 4653 SCOOP
41002 3DV33 31 45 42 N 74 57 24 W 3740 3811 SCOOP
...
51101 3DV49 24 21 40 N 162 4 30 W 4849 4962 SCOOP
</pre>

krober10nd · 2021-07-15T12:22:20Z

@shinbunya interestingly I get . Maybe it's because I'm in Brasil?

>> NDBC = urlread('http://www.ndbc.noaa.gov/stndesc.shtml')

NDBC =

    '<html>
     <head><title>301 Moved Permanently</title></head>
     <body bgcolor="white">
     <center><h1>301 Moved Permanently</h1></center>
     <hr><center>CloudFront</center>
     </body>
     </html>
     '

>>

shinbunya · 2021-07-15T12:35:36Z

Yeah, it sounds like an only reasoning of this unreasonable behavior that I can think of. The url gives me info shown in the attached image on my web browser. Should we contact NDBC? If you share an image of what you get on your web browser, I will contact NDBC with it, just to let them know the situation.

krober10nd · 2021-07-15T12:47:39Z

Ah so my browser automatically redirects to https://www.ndbc.noaa.gov/stndesc.shtml this appears to be the most recent site. Check if it still works on your end please.

krober10nd · 2021-07-15T12:49:08Z

~~Oh this is weird that's the same link! It seems to open work occasionally for me then. Oh well.~~
It has https instead of http

WPringle · 2021-07-15T15:59:14Z

Fantastic, thanks for updating this!

shinbunya · 2021-07-15T16:41:55Z

Thanks guys!

Shintaro Bunya and others added 2 commits July 14, 2021 14:44

Update for more robust scraping from NDBC websites

5fc9934

Merge branch 'CHLNDDEV:Projection' into Projection

2323fd2

update link

feb67be

update REAME and docstring in Make_f15

3206198

krober10nd approved these changes Jul 15, 2021

View reviewed changes

krober10nd merged commit a0661ca into CHLNDDEV:Projection Jul 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update for more robust scraping from NDBC websites #242

Update for more robust scraping from NDBC websites #242

shinbunya commented Jul 14, 2021

krober10nd commented Jul 14, 2021 •

edited

shinbunya commented Jul 15, 2021

shinbunya commented Jul 15, 2021

krober10nd commented Jul 15, 2021 •

edited

shinbunya commented Jul 15, 2021 •

edited

krober10nd commented Jul 15, 2021

shinbunya commented Jul 15, 2021

krober10nd commented Jul 15, 2021 •

edited

krober10nd commented Jul 15, 2021 •

edited

WPringle commented Jul 15, 2021

shinbunya commented Jul 15, 2021

Update for more robust scraping from NDBC websites #242

Update for more robust scraping from NDBC websites #242

Conversation

shinbunya commented Jul 14, 2021

krober10nd commented Jul 14, 2021 • edited

shinbunya commented Jul 15, 2021

shinbunya commented Jul 15, 2021

krober10nd commented Jul 15, 2021 • edited

shinbunya commented Jul 15, 2021 • edited

krober10nd commented Jul 15, 2021

shinbunya commented Jul 15, 2021

krober10nd commented Jul 15, 2021 • edited

krober10nd commented Jul 15, 2021 • edited

WPringle commented Jul 15, 2021

shinbunya commented Jul 15, 2021

krober10nd commented Jul 14, 2021 •

edited

krober10nd commented Jul 15, 2021 •

edited

shinbunya commented Jul 15, 2021 •

edited

krober10nd commented Jul 15, 2021 •

edited

krober10nd commented Jul 15, 2021 •

edited