Transport demand workflow missing country data infilling

[brynpickering]

What happened?

Running the full workflow, it has become apparent that the rule create_controlled_road_transport_annual_demand_and_installed_capacities misses data from some countries. This creates an error in the workflow in this set of method calls:

euro-calliope/scripts/transport/road_transport_controlled_charging.py

Lines 95 to 104 in 5d5b4eb

	df_ev_chargeable_distance = (
	df_ev_numbers.align(battery_size, level="vehicle_type")[1]
	.squeeze()
	.mul(df_ev_numbers)
	.groupby(level=["country_code", "year"])
	.sum()
	.loc[country_codes]
	.unstack("year")
	.mul(transport_scaling_factor)
	)

Version

1.2.0.dev

Relevant log output

Traceback (most recent call last):
  File "/Users/bryn.pickering/Repos/calliope-project/euro-calliope/.snakemake/scripts/tmpwia8kyhz.road_transport_controlled_charging.py", line 159, in <module>
    df_charging_potentials = extract_national_ev_charging_potentials(
  File "/Users/bryn.pickering/Repos/calliope-project/euro-calliope/.snakemake/scripts/tmpwia8kyhz.road_transport_controlled_charging.py", line 100, in extract_national_ev_charging_potentials
    df_ev_numbers.align(battery_size, level="vehicle_type")[1]
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 889, in __getitem__
    return self._getitem_tuple(key)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 1060, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 791, in _getitem_lowerdim
    return self._getitem_nested_tuple(tup)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 865, in _getitem_nested_tuple
    obj = getattr(obj, self.name)._getitem_axis(key, axis=axis)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 1113, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 1053, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis, raise_missing=False)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 1254, in _get_listlike_indexer
    indexer, keyarr = ax._convert_listlike_indexer(key)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexes/multi.py", line 2568, in _convert_listlike_indexer
    raise KeyError(f"{keyarr[mask]} not in index")
KeyError: "['ALB' 'BIH' 'MKD' 'MNE' 'NOR' 'SRB' 'CHE'] not in index"

[adrienmellot]

Good spot, I guess that's the issue of usually only running the minimal workflow. Perhaps we should adjust the minimal workflow to include one of the countries for which data is usually missing?
It seems that it's the JRC data missing countries. Should we directly address that in the JRC data processing upstream or should we do it in the transports script directly?

[adrienmellot]

Addressed in #394 directly as the code is anyway modified.