Access to cams qv data on thredds server hangs

[adriantompkins]

We are finding in class that access to the thredds server for the CAMS output seems to hang, not completing even in 15 or 20 minutes:

cesm_data_path = "http://thredds.atmos.albany.edu:8080/thredds/dodsC/CESMA/"
atm_control = xr.open_dataset(cesm_data_path + "cpl_1850_f19/concatenated/cpl_1850_f19.cam.h0.nc")

affecting both the radeq and rce notebooks... I've swapped out this humidity dataset for data from ERA5 reanalysis in my fork, and calculate averages using the cdo package, but wanted to flag it as an issue.

[brian-rose]

Interesting, I can't seem to reproduce the problem on my end. I ran that part of the code in the Radiative Equilibrium notebook both on our (private access) Jupyterhub and on the public Binder, and was able to access the data from our thredds server normally.

I wonder if there was a temporary outage, or are you seeing this problem repeatedly?

[ktyle]

@adriantompkins I can check our THREDDS server logs to see if there is an identifiable problem. Could you please let me know the hostnames and/or IP #s that your class requests might be coming in from? I do know that THREDDS does not scale well to multiple simultaneous requests, particularly if the datasets the requests are accessing are large.

[adriantompkins]

@ktyle hi Kevin, not sure if something has changed there, last week was not working at all in a reasonable timeframe in the lab at ICTP and last night at home on my laptop it wasn't running for me either as I tried again for today's lecture, but instead tonight that thredds request is completing in about 40 seconds no issues at all! Maybe there was a temporary network issue? Sorry to have bothered you with it. @brian, I ran the rce notebook using ERA5 humidity instead. That is really a nice notebook, and I love the feedback one that follows too. This resource is amazing.

[adriantompkins]

@ktyle - so last night was working fine, this morning in class instead I get a

Error:curl error: Timeout was reached
curl error details: 
Warning:oc_open: Could not read url
...
File src/netCDF4/_netCDF4.pyx:2449, in netCDF4._netCDF4.Dataset.__init__()

File src/netCDF4/_netCDF4.pyx:2012, in netCDF4._netCDF4._ensure_nc_success()

OSError: [Errno -68] NetCDF: I/O failure: 'http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/pressure/air.mon.1981-2010.ltm.nc'

I wondered if it was an IP issue, so I switched to my mobile phone hotspot, but still didn't work. Internet seems to be working ok, can open US-based websites in a browser, including www.esrl.noaa.gov .... very strange.

Would suggest checking for the following code lines:

if (Adrian_is_lecturing):
    thredds_transfer_speed=0
    print("hahahaha")

;-)

[brian-rose]

Closing this as it's not really a climlab code issue, but I opened brian-rose/ClimateLaboratoryBook#103 to track migrating the book to use ARCO data wherever possible to get around the limitations of THREDDS.