_down_select_var: remove_recalculated_sectors() deletes all gases that are not specified by sector

[jkikstra]

When replicating the Getting Started Tutorial on my pc, everything works as expected.
See this folder for the files used, if you'd like to reproduce.

Now, with the same script, but different minimal input data, aneris seems to break at _downselect_var().
To reproduce, use the files in this folder.

When history data has suffix |History with model data suffix being |Unharmonized, I get the error:

Traceback (most recent call last):
  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 40, in <module>
    call_harmonization(hist_file)
  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 32, in call_harmonization
    harmonize(model, history=history, regions=regions, rc=rc, output_path=output_path, output_prefix=output_prefix)
  File "c:\users\kikstra\documents\github\aneris\aneris\cli.py", line 64, in harmonize
    driver.harmonize(scenario)
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 476, in harmonize
    scenario).results()
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 353, in process
    self._downselect_var()  # only prefix|*|suffix
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 297, in _downselect_var
    assert(len(self.hist) > 0)
AssertionError

--> see #14 for further specification of the prefix/suffix issue addressed above. Below, in this thread (#14) we focus further on down_select_var() behaviour wrt aggregates.

When history data has suffix |Unharmonized, to more closely mimic the Getting Started Tutorial (with model data suffix being |Unharmonized), I get:

Traceback (most recent call last):
  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 41, in <module>
    call_harmonization(hist_file)
  File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 31, in call_harmonization
    harmonize(model, history=history, regions=regions, rc=rc, output_path=output_path, output_prefix=output_prefix)
  File "c:\users\kikstra\documents\github\aneris\aneris\cli.py", line 64, in harmonize
    driver.harmonize(scenario)
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 490, in harmonize
    self.config['harmonize_year'], self.add_5regions
  File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 605, in _harmonize_regions
    'Model is empty after downselecting regional values')
RuntimeError: Model is empty after downselecting regional values

[gidden]

Hey @jkikstra - in its current implementation, both history and model values must have the exact same variable name. So if you have, for example:

foo|Emissions|BC|History
foo|Emissions|BC|Unharmonized

this will fail.

For sure, the error message here should be better. Can you make a separate issue to that effect? We could, for example, check the data before running to see if there are no values in either dataset matching the expected prefix/suffix, for example.

But in the near term, can you try this replacing |History with |Unharmonized?

[gidden]

Note that in the real fix, IMHO, is to get rid of the prefix/suffix stuff. This came from the original variable namings for CMIP6.

[jkikstra]

@gidden: sure, I'll make a new isssue about dealing with prefix/suffix, specifying the two options you gave ( a. getting rid of it [or at least making aneris more flexible in dealing with it], b. adding a checking routine and giving a clearer error message).
**--> this discussion on prefix and suffix has moved to #14 **

downselecting - empty model
As for your other point, I had already tried to do that; see the second error message in my message. With matching prefix and suffix, I get a RuntimeError.
It indicates that 'Model is empty after downselecting regional values'.
The model was fully cleared by the line model = utils.remove_recalculated_sectors(model, prefix, suffix).
I now realise that this was caused by aneris deleting all summed/aggregated values for each gas that can contain of multiple sectors, while I only had aggregated values in my dummy data set.
For the current effort, this behaviour should probably be changed, do you agree?

[gidden]

Ok - I'll try to look at this tonight.

…

On Mon, Nov 4, 2019 at 10:44 AM jkikstra ***@***.***> wrote: @gidden <https://github.com/gidden>: sure, I'll make a new isssue about dealing with prefix/suffix, specifying the two options you gave ( a. getting rid of it [or at least making aneris more flexible in dealing with it], b. adding a checking routine and giving a clearer error message). *downselecting - empty model* As for your other point, I had already tried to do that; see the second error message in my message. *With matching prefix and suffix*, I get a RuntimeError. It indicates that 'Model is empty after downselecting regional values'. The model was fully cleared by the line model = utils.remove_recalculated_sectors(model, prefix, suffix). I now realise that this was caused by aneris deleting all summed/aggregated values for each gas that can contain of multiple sectors, while I only had aggregated values in my dummy data set. For the current effort, this behaviour should probably be changed, do you agree? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12?email_source=notifications&email_token=AAKUAEJAWRGUCPUNSATOB3LQR7VHTA5CNFSM4JHNLNN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC6VRZY#issuecomment-549279975>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKUAEKIAGZT33HV7Z23WJTQR7VHTANCNFSM4JHNLNNQ> .

[gidden]

@jkikstra would you be able to write a small unit test that fails that highlights the aggregate gas total issue? If so, then I can directly write a patch that satisfies the test.

[jkikstra]

@gidden Yes, I can do this tomorrow morning.

---edit---
was worked on in #15