Handle the error if the script failed to change 'ScenarioList.csv' or 'ExecuteList.csv'

[BainanXia]

I've encountered this issue twice so far: the status of a scenario is switched to 'extracted' after extract_simulation_output() however, the state of a scenario is unchanged, which is still 'execute' instead of 'analyze'.

The latest one happened to be scenario 1083, which is one of the 15 scenarios that we reran on BECOMPUTE01 due to the data loss during the transfer. Below is the log of the error:

SCENARIO: Julia | Texas_2030_Ambitious_OB2_Mesh100x7+6

--> State
execute
--> Status
extracted
Reading bus.csv
Reading plant.csv
Reading gencost.csv
Reading branch.csv
Reading dcline.csv
Reading sub.csv
Reading bus2sub.csv
Reading zone.csv
--> Loading ct
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-4d64b5dd0e66> in <module>
      3 # Suppress the printing here
      4 pm.block_print()
----> 5 scenario_info = ScenarioInfo(scenario)
      6 pm.enable_print()

~/.pyenv/versions/3.8.3/lib/python3.8/site-packages/powersimdata/design/scenario_info.py in __init__(self, scenario)
     93 
     94     def __init__(self, scenario):
---> 95         _check_state(scenario)
     96         self.info = scenario.info
     97         self.pg = scenario.state.get_pg()

~/.pyenv/versions/3.8.3/lib/python3.8/site-packages/powersimdata/design/scenario_info.py in _check_state(scenario)
     16     """
     17     if scenario.state.name != "analyze":
---> 18         raise TypeError("Scenario state must be 'analyze.'")
     19 
     20 

TypeError: Scenario state must be 'analyze.'

I changed the state into 'analyze' manually on the server afterwards.
One of the potential reasons that the issue shows up is I was trying to extract a bunch of scenarios in a loop and threads are competing the file access.

[BainanXia]

Moving the discussions with @danielolsen on Slack here for records:

Maybe too many processes trying to edit the ScenarioList at once, and one gets blocked or overwritten?

Then should we handle such case specifically or at least throw an error if the script failed to change the file?

Different problems would require different solutions. If the file is busy, we would need to detect a bad return from awk. If it gets overwritten... I guess we would want to wait 5 seconds and confirm the change was made (and try again if not?)

We do have a lock for the two files, right? So it can’t be overwritten?

This may be something that we can address via Jon's new object, if that is indeed the problem. Let's try to recreate the bug first though

[danielolsen]

One of the potential reasons that the issue shows up is I was trying to extract a bunch of scenarios in a loop and threads are competing the file access.

This has bit us in the ass a few times when we did this without adding sleep(x) in the loop (e.g. when I was trying to move a bunch of scenarios to the archive disk). As useful as the Scenario framework has been, I don't think it's fully ready for these sorts of pressure tests and we should avoid them as much as we can.

[rouille]

I think we should use a lock file every time we manipulate the ScenarioList and ExecuteList files. Right now, we have one lock file, scenario.lockfile, which is used when we generate the id of the scenario. We should rename this file scenario_list.lockfile and we should have a execute_list.lockfile to lock the ExecuteList file.