Fix islanded buses and add clustering by networks - clean branch

[ekatef]

A cleaned version of #614

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

Very nice PR :)
temporary comments in the meantime :)
Let us know when ready

[ekatef]

Thanks a lot! :)

Wouldn't it be more convenient to move back to #614? I have been experimenting a bit with git history and #614 has been cleaned-up as well. Although, checking results are better here :) Let me know please your preferences

Currently there are some problems with reproducibility of the workflow. Not sure what exactly is the reason

[ekatef]

Mmmm...
It looks like we have a new task with better data: more connections between buses are captured now. Which leads to some troubles when clustering:

simplified network

clustered network

(with minimum 28 buses)

Note please that the each of the seemingly isolated buses in elec_s.nc now has some company. The first my idea is to drop all the isolated buses remained after clustering. Or, probably, to remove all the isolated pair/groups of buses earlier, in simplify_network?

[ekatef]

Update: the above are effects of some specific config settings. The workflow and topology model seems to be quite sensitive to it. Probably, some kind of visual guide would be helpful to understand the model behaviour better.

By proper config setting, Testing results for Central Asia look quite appropriate:

kmeans (by load)

hac (by load)

hac (by gdp)

gadm

What is really surprising, hac algorithm doesn't require to decrease a number of clusters to a very few in this config setup. Not sure it's good news as I was not able to understand which exactly config changes had such an effect

[ekatef]

@davide-f, I think this PR is ready.

Testing on CD as well as CD + ZM was successful for all clustering types. I think we may assume that the introduced changes work properly.

However, there are some pitfalls along the workflow which can make difficult to reproduce clustering results. In particular, I'd keep in sight using inf values in aggregate_to_substations():

pypsa-earth/scripts/simplify_network.py

Lines 442 to 451 in 6f01921

	dist[
	buses_i
	] = np.inf # bus in buses_i should not be assigned to different bus in buses_i
	for c in n.buses.country.unique():
	incountry_b = n.buses.country == c
	dist.loc[incountry_b, ~incountry_b] = np.inf
	busmap = n.buses.index.to_series()
	busmap.loc[buses_i] = dist.idxmin(1)

It can lead to the errors which are difficult to catch as idxmin() being applied to the row of inf gives a pretty arbitrary result. Being overlayed with variations in output of clustering algorithms and the overall complexity of our workflow, it can give really fascinating consequences :D

[ekatef]

Sorry for messing-up the branch history :( Haven't meant it, honestly!

[davide-f]

A little comment, please modify but this PR looks good to me :)

@pz-max do you want to take a look or shall we go?
This is a quite heavy PR.

[davide-f]

In this case, let's have False as adefault: that means this merge is not executed.

[ekatef]

Done

[pz-max]

A little comment, please modify but this PR looks good to me :)

@pz-max do you want to take a look or shall we go? This is a quite heavy PR.

@davide-f no need to review. Trusting you both to do the best 🥇

[davide-f]

Merged, many thanks @ekatef !

[ekatef]

Super :) Thank you so much, @davide-f!