Tutorial snakefile run: IndexError in simplify_network

[GridGrapher]

Checklist

• I am using the current main branch or the latest release. [c408092]
• I am running on an up-to-date pypsa-earth environment. Update via mamba env update -f envs/environment.yaml. (Used mamba instead of conda)

Describe the Bug

Trying to run snakemake on the tutorial config errors out on jobid: 5, specifically:

IndexError: index 0 is out of bounds for axis 0 with size 0

for this line:

pypsa-earth/scripts/simplify_network.py

Line 651 in c408092

i_aggreg_bus = n.buses.loc[i_suffic_load][buses_in_country].index[0]

Steps to reproduce

Install following the instructions on the pypsa-earth github page
Copy config of 'config.tutorial.yaml' into 'config.yaml'
Run snakemake -j 1 solve_all_networks

Error Message

INFO:pypsa.io:Imported network elec.nc has buses, carriers, generators, lines, loads, storage_units, transformers
INFO:__main__:Mapping all network lines onto a single 380kV layer
INFO:__main__:Simplifying connected link components
INFO:__main__:Removing stubs
Traceback (most recent call last):
  File "/home/ubuntu-code/Code/pypsa-earth/.snakemake/scripts/tmpq13wahiq.simplify_network.py", line 807, in <module>
    n, merged_nodes_map = merge_isolated_nodes(
  File "/home/ubuntu-code/Code/pypsa-earth/.snakemake/scripts/tmpq13wahiq.simplify_network.py", line 655, in merge_isolated_nodes
    i_aggreg_bus = n.buses.loc[i_suffic_load][buses_in_country].index[0]
  File "/home/ubuntu-code/miniconda3/envs/pypsa-earth/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5320, in __getitem__
    return getitem(key)
IndexError: index 0 is out of bounds for axis 0 with size 0


Error in rule simplify_network:
    jobid: 5
    input: networks/elec.nc, resources/costs.csv, resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
    output: networks/elec_s.nc, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/busmap_elec_s.csv, resources/bus_regions/connection_costs_s.csv
    log: logs/simplify_network/elec_s.log (check log file(s) for error details)

[pz-max]

Thanks @GridGrapher for reporting. This seems related to issue #554 and PR #632 ...

[ekatef]

@GridGrapher, thanks a lot for the detailed report!

@pz-max is absolutely right, it is definitely connected with #632. Looking into it

[ekatef]

@GridGrapher, as a quick fix I can suggest to change value of p_threshold_merge_isolated to 300 from 10

Currently #635 is in progress which improves implementation of merge_isolated_nodes and will resolve the issue you reported

[davide-f]

@ekatef @pz-max I think the open draft PR #635 will soon solve some issues.
I was verifying whether other issues occurred and merged.
There were some minor elements that were causing problems along the way

[GridGrapher]

The latest commits solve the bug I was having, however now its errors on another step.
I appended n_clusters and N for an overview.

INFO:pypsa.io:Imported network elec_s.nc has buses, carriers, generators, lines, loads, storage_units
WARNING:__main__:Keyword argument feature is only valid for algorithm `hac`. Given feature `solar+onwind-time` will be ignored.
Traceback (most recent call last):
  File "/home/ubuntu-code/Code/pypsa-earth/.snakemake/scripts/tmpqwy8bz7j.cluster_network.py", line 736, in <module>
    clustering = clustering_for_n_clusters(
  File "/home/ubuntu-code/Code/pypsa-earth/.snakemake/scripts/tmpqwy8bz7j.cluster_network.py", line 582, in clustering_for_n_clusters
    busmap = busmap_for_n_clusters(
  File "/home/ubuntu-code/Code/pypsa-earth/.snakemake/scripts/tmpqwy8bz7j.cluster_network.py", line 477, in busmap_for_n_clusters
    n_clusters = distribute_clusters(
  File "/home/ubuntu-code/Code/pypsa-earth/.snakemake/scripts/tmpqwy8bz7j.cluster_network.py", line 310, in distribute_clusters
    n_clusters >= len(N) and n_clusters <= N.sum()
AssertionError: Number of clusters must be 17 <= n_clusters <= 99 for this selection of countries. 
 n_clusters = 6 
 country  sub_network
BJ       0               1
         11              1
         12              1
NG       1              83
         10              1
         13              1
         14              1
         15              1
         16              1
         2               1
         3               1
         4               1
         5               1
         6               1
         7               1
         8               1
         9               1
dtype: int64

pypsa-earth/scripts/cluster_network.py

Line 306 in 508f80c

n_clusters >= len(N) and n_clusters <= N.sum()

Working my way through the documentation so I can understand the full scope.

[pz-max]

@GridGrapher it seems that your config.yaml has too little clusters. Ideally the number of clusters that you specified (6) should be sufficient. So, I am wondering if the error is that you run on an old config.yaml. Can you check it when you comapre it with newest version of the config.tutorial.yaml?

If there is no difference, what if you set both of these value to 20 ?

pypsa-earth/config.default.yaml

Lines 65 to 66 in 508f80c

	p_threshold_drop_isolated: 20 # [MW] isolated buses are being discarded if bus mean power is below the specified threshold
	p_threshold_merge_isolated: 300 # [MW] isolated buses are being merged into a single isolated bus if bus mean power is above the specified threshold

[GridGrapher]

@pz-max You were right that the config was still the old one.
I ran cp config.tutorial.yaml config.yaml which didn't replace the updated lines.

Now that config.yaml is up to date the tutorial runs as normal, thanks!