allow usage of own powerplants for selected country

[martacki]

in build_powerplants.py add the feature to add own carriers (add_my_carriers) and restrict to a specific build year (restrict_buildyear), which both can be configured in config.yaml (my_carriers_for_countries, my_carriers_switch, restrict_buildyear).

[FabianHofmann]

Hey! I think the setup it nice and should work well. Just as a counterproposal:

There is one optional, additional dataset, called custom_powerplants.csv in the resource folder. This is added to the standard powerplant list if in config.py the default

custom_powerplants: false

is either set to true or to a pandas.query string. The latter enables you to filter the additional data to the settings you want, e.g.:

custom_powerplants: Country in ['DE', 'FR'] & YearCommisioned < 2050

The same filter option could be enabled for the standard powerplant list, e.g.

powerplants_filter: Country not in ['DE', 'FR'] & YearCommisioned < 2050

(The powerplants_filter should come first in the config.yaml, though. )

This could strongly reduce the code such as :

def add_custom_carriers(ppl):
    custom_ppl_query = snakemake.config['electricity']['custom_powerplants'']
    if not custom_ppl_query:
        return ppl
    # import custom_ppl here
    if isinstance(custom_ppl_query, str):
        custom_ppl.query(custom_ppl_query, inplace=True)
    return ppl.append(cusom_ppl)

And of course the filter for the standard ppl, not as a function but in the 'main' code.

[coroa]

Couple of comments:

• custom is a better name, than my
• input data which is not generated, has to live in data: so put it powerplants in data/custom_powerplants.csv
• @FabianHofmann's proposal to take advantage of the pandas query syntax sounds great to me.
• The use of such a query string means of course that you have to preprocess the csv file and clean the YearCommissioned column to a single integer.

[coroa]

A couple of general comments about your code. Don't implement them. Instead, go ahead with @FabianHofmann's proposal.

[coroa]

in such case use a

if switch in ('replace-all', 'replace-selection', 'add'):

form

[coroa]

add_ppls.groupby('Fueltype').mean().index

is a very strange construct.

Do you mean?

add_ppls.Fueltype.unique()

[coroa]

x == True or x == False is a bit silly, when you think about what it means. Use x for the former and not x or ~ x (in this case for pandas), instead.

More importantly,

for pattern in search_pattern:
        ppl = ppl[~ppl['YearCommissioned'].str.contains(pattern)]

has very bad performance characteristics, as you create copies of the power plant list in a loop. Take advantage of the fact that ~ ppl['YearCommissioned'].str.contains(pattern) creates a slim boolean selection mask, ie. instead start with take all, and then narrow the mask down:

take_b = pd.Series(True, index=ppl.index)
for pattern in search_pattern:
    take_b &= ~ppl['YearCommissioned'].str.contains(pattern)
ppl = ppl.loc[take_b]

[coroa]

Took me a while to reason about, what you are searching for, instead use:

search_pattern = [str(x) for x in range(int(year),2050)]

or even

search_pattern = map(str, range(int(year), 2050))

[martacki]

Hi both, thanks for your input. I made few changes from both your proposals, yet I would like to keep the separation - filtering the year is a different issue than replacing or adding custom powerplants.

[martacki]

* The use of such a query string means of course that you have to preprocess the csv file and clean the YearCommissioned column to a single integer.

I did a histogram to see how many of the data is wrong processed by just casting to int. It's less than 1%, so I won't do any preprocessing as there are already so many NaN values, this will not have such a huge impact
EDIT: Assuming the data/custom_powerplants.csv will have correct or roughly correct entries

[FabianHofmann]

close this in favour of #84