Is this suggestion related to a problem? Please describe.
There are times when an upstream data source becomes unavailable or is deemed too much of a security risk by IT teams. It can be impossible to tell in advance whether either of these is going to be the case, so it helps to be able to define alternative sources for the same data, where feasible (i.e. not an API call)
Describe the solution you'd like
Convention for exposing all upstream URLs so that if a user wants to swap out an existing public data source with a functional equivalent (e.g. the same data, but archived on zenodo, or stored on internal servers), they can.
This would hopefully mitigate against issues caused by the data sources becoming unavailable as modelblock devs could archive versions where they know of volatility and document the possibility for users to point to an archive in their config.
Describe alternatives you've considered
The only alternative I can reasonably think of is to expect users to archive the downloaded data. This might include some harmonisation of the data done on the modelblock side so that it isn't identical to the data found at the upstream source. Also, since the downloaded data is config-dependent, I don't think it could be archived by the modelblock devs except for some "obvious" variations of the config.
Additional context
Reproducibility is only as good as the availability of upstream data sources. Over the years, we've seen sources become brittle or entirely unavailable. In PyPSA-Eur, this has led to archiving lots of data sources in an S3 bucket, available via data.pypsa.org.
We've also worked with system operators who are completely blocked from accessing the internet. They could work with pre-download data but then they can't vary the config if it would require more / different data to be downloaded. Pointers to the source data that they can host internally would help here.
Is this suggestion related to a problem? Please describe.
There are times when an upstream data source becomes unavailable or is deemed too much of a security risk by IT teams. It can be impossible to tell in advance whether either of these is going to be the case, so it helps to be able to define alternative sources for the same data, where feasible (i.e. not an API call)
Describe the solution you'd like
Convention for exposing all upstream URLs so that if a user wants to swap out an existing public data source with a functional equivalent (e.g. the same data, but archived on zenodo, or stored on internal servers), they can.
This would hopefully mitigate against issues caused by the data sources becoming unavailable as modelblock devs could archive versions where they know of volatility and document the possibility for users to point to an archive in their config.
Describe alternatives you've considered
The only alternative I can reasonably think of is to expect users to archive the downloaded data. This might include some harmonisation of the data done on the modelblock side so that it isn't identical to the data found at the upstream source. Also, since the downloaded data is config-dependent, I don't think it could be archived by the modelblock devs except for some "obvious" variations of the config.
Additional context
Reproducibility is only as good as the availability of upstream data sources. Over the years, we've seen sources become brittle or entirely unavailable. In PyPSA-Eur, this has led to archiving lots of data sources in an S3 bucket, available via
data.pypsa.org.We've also worked with system operators who are completely blocked from accessing the internet. They could work with pre-download data but then they can't vary the config if it would require more / different data to be downloaded. Pointers to the source data that they can host internally would help here.