Improve flexibility of simulation engine input preparation process

[danielolsen]

What

Allow simulation engine inputs to be created without uploading them to the server or touching the ScenarioList/ExecuteList files.

Why

• For use with the simulation engine in a 'manual' mode (not controlled by PowerSimData)
• To improve testability of the input generation process

How

• Refactor prepare_mpc_file method of powersimdata.scenario.execute.SimulationInput to remove the matfile copy/upload step.
  • This can be moved into the prepare_simulation_input method of powersimdata.scenario.execute.Execute.
• Similarly, refactor the prepare_profile method of powersimdata.scenario.execute.SimulationInput to separate the logic for determining whether and how to prepare profiles from the process of copying/uploading them and/or creating symlinks.

[rouille]

An entry is added to the ScenarioList/ExecuteList files in the create_scenario method of the Create class right before switching to the execute state.

That said I am wondering if we could not develop methods in the helpers module that would dump the profiles as csv and the grid (and eventually storage) MATfile. This methods could be used in the Create class since we can access the transformed grid and profile and we could refactor the SimulationInput class instantiated in the prepare_simulation_input method of the Execute class to use these helpers module. That way we don't have to touch the ScenarioList/ExecuteList files.

[danielolsen]

If we can refactor the SimulationInput class to not include any communication, then we can have that be the helper, and invoke it from either Create or Execute. So we would call Execute -> SimulationInput and Create -> SimulationInput, rather than Execute -> SimulationInput -> NewHelper.

[rouille]

It should not be too hard. We will have to move some of the method of SimulationInput such as create_folder into the Execute class and create a new powersimdata.scenario.simulation_input module enclosing the refactored class that will be imported in both the powersimdata.scenario.create and powersimdata.scenario.execute modules.

[BainanXia]

Reading through this conversation, I think the ideal case could be the PowerSimData has the capability to prepare scenario input in two modes:

• Integration mode: this will incorporate with interactions with ScenarioList.csv/ExecutionList.csv and the server with REISE.jl installed.
  • remote ssh mode: same as our current setup
  • local mode: align with the containerization effort. This will allow the user to run scenario using our software framework without having the same system structure (a remote server) as we do, in which case, PowerSimData and REISE.jl + Gurobi could be run locally on the same machine in different containers.
• Detachable mode: this is the mode we are talking about here. We only use PowerSimdData to prepare all essential inputs (potentially in a local folder that the user specifies), profiles + grid, for a user customized scenario.

We could make the default mode to be Integration mode -> remote ssh mode for the convenience of our usage. But a general user should be able to choose which mode the PowerSimData is configured/a particular scenario will be handled in.