test failure because RREDC website has changed

[kandersolar]

The test iotools/test_tmy.py::test_read_tmy3_remote started failing because of a 404 on the remote TMY file it tries to read:

https://dev.azure.com/solararbiter/pvlib%20python/_build/results?buildId=3664&view=logs&j=fc432b8b-e2e3-594e-d8b1-15597b6c1d62&t=309866e1-2cf4-5f00-3d0a-999fc3a0f279&l=112

The obvious fix would be to update the URL to wherever that file lives now, but I didn't find it with a quick look around. It's probably in the zipfiles at https://nsrdb.nrel.gov/data-sets/archives.html, but that's not really an option unless we add unzipping functionality, which seems pretty out of scope.

We could have it read from the pvlib github repo instead via: https://raw.githubusercontent.com/pvlib/pvlib-python/master/pvlib/data/723170TYA.CSV. Or just remove the test entirely if TMY files aren't available from NREL via direct links anymore?

[mikofski]

I also just noticed on Friday that the RREDC sites were all down, but I could not find where they are now hosted. I think need to contact NREL. They may not know that the site is down, or they may have finally decided not to share these files at all anymore. Once we know what the deal is, then we should consider sourcing them from somewhere else or archiving them in pvlib if their license allows this.

[kandersolar]

Per some emails with the NSRDB team, the old individual-file URLs are no more, but the original data TMY3 files are available through that large (~4GB) zipfile at https://nsrdb.nrel.gov/data-sets/archives.html. As far as this issue is concerned, I think I would vote for fetching the file from the pvlib github repo. Another option that just came to mind would be to programmatically set up a temporary local http server and fetch the file from localhost, but that's a level of complexity I'm not sure we want to get into.

[mikofski]

Similar to Kevin's http idea, but more boilerplate would be to use an existing archival service like AWS-S3, Dropbox, Box, Google Drive, or Azure/OneDrive. Most services have a free tier that will work fine for pvlib. Does NREL object to pvlib archiving the files?

Optionally GitHub offers Git LFS, and storage, but it's premium. I'm not sure what value, or convenience Git LFS is, but I believe it could be used with any archive service like the ones I mentioned above.

So that's my vote: set up some archive on like Dropbox either with our without Git-LFS.

Another question, should we ship wheels & conda packages with or without the data? Probably best to just access it as needed lazily. Also, should it cache user side, that could get complicated.

[cwhanse]

I'm -3 on shipping 4GB of old TMY2/3 weather files with pvlib, if that's being considered.

@kanderso-nrel is NREL doing away with all TMY files, or, only retiring the server with the up-to-2010 TMY3 files?

If the individual TMY3 files are no longer to be available on a server, I'm unsure what value iotools.read_tmy3_remote provides.

[kandersolar]

@cwhanse My understanding is that TMY3 files are being retired in favor of the current NSRDB PSM3 TMY data rather than "done away with" entirely. The files still exist if someone really needs them, but they have been moved to that 4GB zipfile archive. Note that this is only for the TMY3 dataset; TMY data from the current NSRDB (PSM3) are still very much a thing.

I think I am +1 to not caring about remote access for TMY3 anymore and nixing the test. I'm hesitant to get into the business of hosting bulk weather data through the pvlib project (cloud archive or otherwise), especially if the files are still available (albeit somewhat inconveniently) from the original source.

[cwhanse]

@kanderso-nrel just to confirm, it is the case that the NSRDB will continue to contain site- and year-specific data for years prior to 2011?

[kandersolar]

@kanderso-nrel just to confirm, it is the case that the NSRDB will continue to contain site- and year-specific data for years prior to 2011?

Depends on what exactly you mean by NSRDB. As far as I am aware, the satellite-based product currently available at https://nsrdb.nrel.gov/ is unchanged and will continue to provide individual site- and year-specific data going back to 1998, as well as TMY data calculated from the 1998-present dataset. It is only the older TMY3 dataset calculated from ground station data that is being archived. Hopefully that's unambiguous -- if I'm still not getting to the point, maybe it would be best to email Manajit and Aron, the sources of all NSRDB truth :)

[cwhanse]

That's clear to me now, thanks.

[kandersolar]

It would be nice to stop the CI from throwing a fit about this. Seems like there are two things to decide:

1. Do we want to re-host the original TMY3 datasets somewhere? I'm not sure if NREL would object, and even if they didn't, I don't see much value in re-hosting it because the satellite PSM3 dataset is preferred anyway.
2. Should we remove the remote server capability of iotools.read_tmy3 and the associated test_read_tmy3_remote test?

Personally I vote no for 1 and yes for 2.

[wholmgren]

I agree.

[cwhanse]

I agree.

Seconded