Thredds is slow to respond (a minute or so)

[davidcaron]

I added a feature to finch, as discussed with David Huard before: the user can provide an opendap catalog url to subset all the datasets in this catalog.

The url I used for my tests was this one: https://boreas.ouranos.ca/thredds/catalog/birdhouse/pcic/BCCAQv2/catalog.html where there are around 100 datasets. So the process lists all the datasets, filters which ones correspond to the parameters, and launch the subset for each dataset, one at a time.

When I launch the process, I can see in the logs that the process pauses once in a while for no obvious reason. After a minute or 2 and a couple NetCDF errors, the request succeeds and the subsetting continues for the next dataset... It's very hard to explain why that could happen.

It's hard to reproduce, but it happened 4-5 times already, so I thought I would mention it, maybe someone has an idea.

Has anyone experienced unexplainable slowdowns using Thredds?

[Zeitsperre]

Not sure if I can help but do you have any error readouts copied down?

[davidcaron]

I don't, unfortunately. When these happened, I wasn't sure if the error was in my xarray commands or not. But if it happens again, I'll make sure to keep more information.

[Zeitsperre]

I'm wondering if the errors are from the files themselves. We've had a few issues with the BCCAQv2 data set (metadata problems, shifted grid values). It might not be coming from THREDDS.

[tomLandry]

@Zeitsperre @huard I see above a comment "We've had a few issues with the BCCAQv2 data set (metadata problems, shifted grid values)." Are these problem solved? If not, are they breaking computations? We're presently copying the files...

[Zeitsperre]

@tomLandry The grid shift problem was dealt with on the provider end and the corrected data is what @davidcaron is accessing so that shouldn't be the issue. There may be some errors still, but those might only affect expected results of analyses, nothing that would cause entire processes to fail.

[huard]

@davidcaron Still relevant ?

[Zeitsperre]

Oof, yeah this is a bit stale. @davidcaron is this still an issue?