Problem with remote access

[BobSimons]

With a very old version of netcdf-java (4.3.21? dated Aug 26, 2014),
I was able to open and read a remote file
from a server that supported byte ranges (Accept-Ranges) with e.g.,
NetcdfFile.open("http://data.nodc.noaa.gov/thredds/fileServer/aquarius/nodc_binned_V4.0/monthly/sss_binned_L3_MON_SCI_V4.0_2011.nc")
I haven't used more recent versions of netcdf-java.

Today, with netcdf-java 4.6.4, I get:
Exception in thread "main" java.io.IOException: java.io.EOFException: Reading http://data.nodc.noaa.gov/thredds/fileServer/aquarius/nodc_binned_V4.0/monthly/sss_binned_L3_MON_SCI_V4.0_2011.nc at 20 file length = 20
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:427)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:394)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:381)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:369)
at [my code]

Of course, I could access this particular file via THREDDS's OPeNDAP service,
but this is my test code to ensure that netcdf-java can read a remote file via byte ranges.

I also tried NetcdfDataset.openDataset(fullName) and get a similar error message:
Exception in thread "main" java.io.IOException: java.io.EOFException: Reading http://data.nodc.noaa.gov/thredds/fileServer/aquarius/nodc_binned_V4.0/monthly/sss_binned_L3_MON_SCI_V4.0_2011.nc at 20 file length = 20
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:427)
at ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:748)
at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:427)
at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:410)
at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:395)
at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:382)
at [my code]

The file is accessible via byte ranges / Accept-Ranges:
curl --head http://data.nodc.noaa.gov/thredds/fileServer/aquarius/nodc_binned_V4.0/monthly/sss_binned_L3_MON_SCI_V4.0_2011.nc
HTTP/1.1 200 OK
Date: Mon, 29 Feb 2016 23:26:50 GMT
Last-Modified: Sat, 12 Dec 2015 08:29:21 GMT
Accept-Ranges: bytes
Content-Type: application/x-netcdf
Content-Length: 2597612
Vary: Accept-Encoding,User-Agent

Thank you for your help.

[DennisHeimbigner]

How did you tell the open'd file which bytes to get?

[BobSimons]

With 4.6.4, I can't even open the files. The errors occur when I try to open the files. See the open commands and the stack traces above.

[ethanrd]

@DennisHeimbigner ucar.unidata.io.http.HTTPRandomAccessFile is an instance of ucar.unidata.io.RandomAccessFile that uses HTTP "Accept-Ranges" headers to treat HTTP byte-range accessible files as RAFs.

[DennisHeimbigner]

So, I did the following experiment.

1. created a testfile TestByteRange to open
   http://data.nodc.noaa.gov/thredds/fileServer/aquarius/nodc_binned_V4.0/monthly/sss_binned_L3_MON_SCI_V4.0_2011.nc
2. downloaded sss_binned_L3_MON_SCI_V4.0_2011.nc
3. Set up a localhost thredds server (version 4.6.X == github master) to serve up the downloaded file
   3,. ran TestByteRange against the file from the localhost server.

#1 failed, as expected. #3 suceeded. So, my tentative conclusion is that data.nodc.noaa.gov/thredds
is the problem. Any way you can set up a parallel server using 4.6.4 and see if that fails?

[dopplershift]

As a point of reference, http://data.nodc.noaa.gov/thredds/ is running 4.2.10, which is positively ancient.

However, we shouldn't lose sight of the fact that this is reported to work on ~4.3.21. So unless we're convinced 4.6.4 is in much better compliance with the standard for "Accept-Ranges", there still could be a problem on our end.

[BobSimons]

I'm not at NODC. I was just using their server as a test.
Based on your test #3 above and another test of mine, I think reading with
Byte Ranges works in 4.6.4 (at least under certain circumstances). That's
mostly what I cared about.
Yes, the problem I reported may be a problem related to reading from
pre-4.6.4 versions of THREDDs -- I don't want to pursue that.
You can close this Bug Report.
Thank you for your efforts.

On Tue, Mar 8, 2016 at 3:33 PM, DennisHeimbigner notifications@github.com
wrote:

So, I did the following experiment.

1. created a testfile TestByteRange to open

http://data.nodc.noaa.gov/thredds/fileServer/aquarius/nodc_binned_V4.0/monthly/sss_binned_L3_MON_SCI_V4.0_2011.nc
2. downloaded sss_binned_L3_MON_SCI_V4.0_2011.nc
2. Set up a localhost thredds server (version 4.6.X == github master) to
serve up the downloaded file
3,. ran TestByteRange against the file from the localhost server.

#1 #1 failed, as expected. #3
#3 suceeded. So, my tentative
conclusion is that data.nodc.noaa.gov/thredds
is the problem. Any way you can set up a parallel server using 4.6.4 and
see if that fails?

—
Reply to this email directly or view it on GitHub
#455 (comment).

Sincerely,

Bob Simons
IT Specialist
Environmental Research Division
NOAA Southwest Fisheries Science Center
99 Pacific St., Suite 255A (New!)
Monterey, CA 93940 (New!)
Phone: (831)333-9878 (New!)
Fax: (831)648-8440
Email: bob.simons@noaa.gov

The contents of this message are mine personally and
do not necessarily reflect any position of the
Government or the National Oceanic and Atmospheric Administration.
<>< <>< <>< <>< <>< <>< <>< <>< <><