Allow interaction with locally-cached scenario data files via locally-cached ScenarioList

[danielolsen]

Purpose

It would be convenient to be able to analyze Scenarios with downloaded data even when we cannot access the server.

Design idea

I think we can follow the existing design pattern, where we look for data either in ~/ScenarioData or on the server, except in this case we prioritize the server first instead of the local folder (since the ScenarioList file changes, while PF/PG/profiles are mostly constant once they're created).

• Every time we successfully instantiate a Scenario, we cache a copy of ScenarioList.csv in ~/ScenarioData.
• Every time we attempt to load an existing Scenario (e.g. anything but Scenario('')), we add a try/except block around the getting of the ScenarioList from the server. If we cannot read from the server, either:
  • we explicitly ask the user via an input() call whether they want to continue in a read-only fashion using the cached data in ~/ScenarioData, or
  • we have an optional parameter to the Scenario init (default False) which will use a locally cached ScenarioList if one is available.

[rouille]

We need to handle the case where the scenario is listed in the ScenarioList file on the local machine but the data are not there. This can happen when for instance the user has loaded the scenario and retrieved the PG and the solar profile to calculate the solar curtailment. Later, offline, he wants to map the LMP and does not have the data to do so.

[jenhagg]

Not sure if I'm being too optimistic, but I think we can handle this by abstracting the ssh connection (or data access) as we have talked about earlier. What this might look like:

• put everything from transfer_data.py inside an SshConnection class
• implement the same methods in a LocalConnection class, but have it do things how you would locally, rather than over ssh
• create a wrapper class that has an instance of each of those, and tries local first then over ssh.

The wrapper could be configured to update the local cache, and either fail when ssh doesn't work or use a "best effort" approach and log what it ends up doing, or whatever desired behavior is - essentially your last couple bullet points.

Might make more sense for the names to suggest a local/remote data store (instead of connection), but same idea either way.

As I started writing, I thought this might address @rouille's point, but realized that's a separate case. Though it seems like a well written error message might suffice? We could indicate to the user that the local data is insufficient, and once they connect to the server and try again it will download the missing data.

[danielolsen]

@rouille True, that problem would not be solved. An FileNotFoundError will be thrown when the file cannot be found, or we can catch that and throw something more informative.

@jon-hagg, would that require us to have a folder structure in the local machine that mirrors what's on the server? Right now, the files in ScenarioData are totally flat, organized only by filename. We could easily design it to look in folders on the server or in the one main folder locally though. Feel free to edit the main post with alternative design ideas, I wanted to record our discussions so far but they are by no means the only way to do it.

[rouille]

I don't think you would need to change the folder structure on the local machine. If it uses the local ScenarioList.csv file, the Analyze class will do as usual and try to load the files in the ScenarioData folder in $HOME.

[jenhagg]

@danielolsen yeah I suppose it would require the same structure, but ideally would be handled by the code. I also realized this could be considered part of the story of creating a database, which in my mind involves abstracting the way we access data. So any calling code won't know/care how the data is stored, just access it through an api (python or REST). In this case, we could have multiple data storage implementations - one for local, one for database, one for accessing files on the server, one for a redis cache, etc. And if they all implement the same api we can configure any combination of them under the hood (without changing client code), or any "mode" of operation (greedy sync, best effort, minimum bandwidth, etc). Sorry in advance if this idea is already obvious to people, not sure what's been discussed before or how familiar everyone is with this kind of pattern.