Refactor postprocessing

[henhuy]

Steps:

• Move helper functions into own module
• Refactor postprocessing.py to use predefined calculations
• Switch to string-based scalars/sequences-keys
• Make function map_var_names optional (activated by default)
• Refactor attributes in add_component_info to be adaptable and provide default attributes

[henhuy]

Hey @jnnr, I started refactoring. Please look at predefined calculations and tell me if you like it that way!
We should also add docstrings to each calculation to inform users about the underlying calculation.

[jnnr]

Thanks for the new approach! I really like the modularity.
For now, I have only one comment.
I can review again after the next steps.
I could check if some parts have more implicit assumptions than desirable. The goal in this PR is to reproduce the existing postprocessed results, right? So this could happen separately.

[jnnr]

Multiplying ep_costs and differentiating capacity and storage_capacity is orthogonal (the latter is just a filter). So this is a bit of duplication. If it was possible to wrap calculations, this could be more clean. But not necessary for the first refactoring.

[henhuy]

Hey @jnnr,

refactored postprocessing to use string-based indexes and get component attributes from scalar_params.
This allowed me to remove much of the reindexing functions and to simplify many functions :)
Please have a look.
Also I moved some more specific calculations from helper.py into postprocessing.py (like calculation of losses and summed_variable_costs).

[jnnr]

Thanks for your work, @henhuy! I tried to reproduce the results for an oemof-B3-scenario and found these differences:

• In the sequences, the third level of the multiindex is altered from e.g. B-ch4, B-ch4-boiler, flow to B-ch4, B-ch4-boiler, "('B-ch4', 'B-ch4-boiler_large', 'flow')"
• Sequences "by_variable" have disappeared

Scalars did not change, which is good!

Update: This seems to happen because the first call here alters the results for the second call. Can be fixed by reverting the order, but maybe it makes sense to understand better what is happening.

oemoflex/oemoflex/model/datapackage.py

Lines 327 to 329 in 9f05634

	data_scal, rel_paths_scal = self._get_scalars(self, es)
	data_seq, rel_paths_seq = self._get_sequences(self, es)

[henhuy]

I tried to reproduce the results for an oemof-B3-scenario

Thanks for checking. Which scenario did you test? I will add scenario results to tests and try to reproduce the error.

[jnnr]

I tried to reproduce the results for an oemof-B3-scenario

Thanks for checking. Which scenario did you test? I will add scenario results to tests and try to reproduce the error.

I checked with 2050-80-gas_moreCH4. You should be able to see it with the dump you already uploaded. What we do in oemof-B3 is call ResultsDataPackage.from_energysytem, which also processes the sequences.

[henhuy]

Refactoring is done.
Tests are still failing - I have to check why (datapackage test, which I did not touch).

@jnnr: could you check once more if your oemof-B3-scenario returns the same results?

[jnnr]

Refactoring is done. Tests are still failing - I have to check why (datapackage test, which I did not touch).

@jnnr: could you check once more if your oemof-B3-scenario returns the same results?

Have checked it and found only marginal differences (decimal round-off errors). I had to swap the order of postprocessing as suggested earlier to preserve the postprocessed timeseries.

[henhuy]

The 3.7 build fails as used frictionless version tries to import cached_property from functools (ImportError: cannot import name 'cached_property' from 'functools' (/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/functools.py)) which is not supported in python < 3.8.
Should we pin python version to >=3.8?
My opinion: As python 3.10 is out - supporting 3.8, 3.9 and 3.10 should be sufficient!
What do you think @jnnr ?

[jnnr]

My opinion: As python 3.10 is out - supporting 3.8, 3.9 and 3.10 should be sufficient! What do you think @jnnr ?

I am ok with that. oemof.solph supports 3.8 -3.10, too.