Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nc toolbox reading https links #96

Open
cbowyer opened this issue Feb 14, 2017 · 10 comments
Open

nc toolbox reading https links #96

cbowyer opened this issue Feb 14, 2017 · 10 comments

Comments

@cbowyer
Copy link

cbowyer commented Feb 14, 2017

Hi,

I have had a code which talked to an OPeNDAP http database (OceanColor SST), and had it working very successfully prior to the US government changing to a https (secure) connection.

I have tried to read the netcdf file from the OPeNDAP using the command
"this_URL_ds = ncdataset(this_URL)" but it no longer seems to work.

given the URL, ncdataset is now trying to access is on a https server, is this expected behavior (i.e. it not working)?

Regards,

Carlin
@hohonuuli
Copy link
Member

hohonuuli commented Feb 14, 2017 via email

@cbowyer
Copy link
Author

cbowyer commented Feb 14, 2017

Sure. See below:

I checked this just now and im still getting an error. Thanks.

this_URL = 'https://oceandata.sci.gsfc.nasa.gov:443/opendap/MODISA/L3SMI/2005/005/A2005005.L3m_DAY_SST_sst_4km.nc';

setup_nctoolbox
%loadURLS
this_URL_ds = ncdataset(this_URL);

@hohonuuli
Copy link
Member

When I try to connect to that URL it returns a 403 (Forbidden) status code. Status code 403 responses are the result of the web server being configured to deny access to the requested resource by the client. I don't think this was intentional as you can download the data from https://oceandata.sci.gsfc.nasa.gov/opendap/MODISA/L3SMI/2005/005/A2005005.L3m_DAY_SST4_sst4_4km.nc.html. I suspect that the opendap server is not configured correctly, you should contact someone at NASA and let them know ... maybe https://oceancolor.gsfc.nasa.gov/staff/gene/

@rsignell-usgs
Copy link
Contributor

rsignell-usgs commented Feb 15, 2017 •
edited
Loading

Isn't this the answer? Just drop the :443 and use
https://oceandata.sci.gsfc.nasa.gov/opendap/MODISA/L3SMI/2005/005/A2005005.L3m_DAY_SST4_sst4_4km.nc.html

@coopy1989, I'm surprised you could access the port 443 from a government site before. Usually they only allow access on port 80.

@cbowyer
Copy link
Author

cbowyer commented Feb 15, 2017

FYI - https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6420, this provides some background as to why things changed (my codes worked perfectly prior to this, with the port 80 link).

Thanks for the responses. Because of this (https://www.mathworks.com/support/bugreports/1072120), i assumed that maybe nctoolbox may have only been configured to also read only http links. rsignell, i tried that link, and for me it unfortunately doesn't work (I dont think it matters but Im reading this from UWA in Australia).

The odd thing is that if i use an inbuilt matlab net cdf read function i.e. ncread/ncinfo, the link (through port 443) can be read fine, but the array which comes in isnt at all correct in terms of numbers (i suspect matlab isnt correctly reading the integer format correctly?), and this is the reason i used the nctoolbox in the first place as the toolbox provided a much more simple way of reading in the OPeNDAP nc file.

When I jumped on the OceanColor forum (to grap that link at the top of this post), since last time i looked (in early Jan) it seems as a number of people are having teething issues reading the :443 links. So maybe its time to go back to Ocean color, if you are certain that the nctoolbox code is ok for https/:443 links?

Cheers,

Carlin

@cbowyer
Copy link
Author

cbowyer commented Mar 21, 2017

So I have picked this up again,

I looked at the nc files which come from OPeNDAP. On further inspection these are now CF compliant (Climate Forecasting), which means that the 'data' within them is in a Geo2D format. Thats fine, nctoolbox can deal with that.

What i did notice in the errors were http errors (see attached), something to do with the java which obviously is used to read the files. If the java is unable to read the 'https' URL links, is this something which will be updated with a java update or is it something in the codes of ncdataset which needs to be tweaked to allow that?

Thanks, Carlin

@rsignell-usgs
Copy link
Contributor

@coopy1989 , attachments don't work on github. Can you include a few examples inline?

@cbowyer
Copy link
Author

cbowyer commented Mar 23, 2017

I know its messy, but not too long, so ive included the entirety of the error.

See below - Given that there is a lot of mention of HTTP in the Java bit, this leads me to suspect that there is some http limitation with Java.

egdata = ncdataset('https://oceandata.sci.gsfc.nasa.gov:443/opendap/MODISA/L3SMI/2015/093/A2015093.L3m_DAY_SST_sst_4km.nc');
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at sun.security.ssl.InputRecord.readFully(Unknown Source)
at sun.security.ssl.InputRecord.read(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
at sun.security.ssl.AppOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at ucar.nc2.util.net.HTTPMethod.execute(HTTPMethod.java:284)
at ucar.nc2.dataset.NetcdfDataset.checkIfDods(NetcdfDataset.java:762)
at ucar.nc2.dataset.NetcdfDataset.disambiguateHttp(NetcdfDataset.java:713)
at ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:674)
at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:424)
at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:407)
at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:392)
at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:379)
Error using ncdataset (line 97)
Failed to open https://oceandata.sci.gsfc.nasa.gov:443/opendap/MODISA/L3SMI/2015/093/A2015093.L3m_DAY_SST_sst_4km.nc

Caused by:
Error using ncdataset (line 81)
Java exception occurred:
ucar.nc2.util.net.HTTPException: java.net.SocketException: Connection reset

	at ucar.nc2.util.net.HTTPMethod.execute(HTTPMethod.java:289)

	at ucar.nc2.dataset.NetcdfDataset.checkIfDods(NetcdfDataset.java:762)

	at ucar.nc2.dataset.NetcdfDataset.disambiguateHttp(NetcdfDataset.java:713)

	at ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:674)

	at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:424)

	at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:407)

	at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:392)

	at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:379)

Caused by: java.net.SocketException: Connection reset

	at java.net.SocketInputStream.read(Unknown Source)

	at java.net.SocketInputStream.read(Unknown Source)

	at sun.security.ssl.InputRecord.readFully(Unknown Source)

	at sun.security.ssl.InputRecord.read(Unknown Source)

	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

	at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)

	at sun.security.ssl.AppOutputStream.write(Unknown Source)

	at java.io.BufferedOutputStream.flushBuffer(Unknown Source)

	at java.io.BufferedOutputStream.flush(Unknown Source)

	at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)

	at
    org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
    
	at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)

	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)

	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)

	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)

	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)

	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)

	at ucar.nc2.util.net.HTTPMethod.execute(HTTPMethod.java:284)

	... 7 more

@hohonuuli
Copy link
Member

hohonuuli commented Mar 23, 2017 •
edited by rsignell-usgs
Loading

I'll try to find some time to look into this tomorrow.

@rsignell-usgs
Copy link
Contributor

Thanks @hohonuuli !

@vostok61 vostok61 mentioned this issue Jun 4, 2018
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hohonuuli @rsignell-usgs @cbowyer