Clustering problem seemingly caused by addition of new TYNDP projects

[koen-vg]

Checklist

• I am using the current master branch or the latest release. Please indicate.
• I am running on an up-to-date pypsa-eur environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

Commit c5026aa seems to have caused a strange kind of regression. It causes an additional sub_network with a single node to be added to Germany, but one without any load, meaning that that node isn't clustered correctly (since no nodes are allocated to it in the clustering).

The problem goes away after removing the following line from data/links_tynpd.csv:

Germany-UK Hybrid Interconnector,Fetteresso (UK),Emden (DE),800,,2000,under consideration,,https://eepublicdownloads.blob.core.windows.net/public-cdn-container/tyndp-documents/TYNDP2024/240220_TYNDP2024_project_portfolio.xlsx,-2.383,56.991,7.207,53.376

The node in question has a generator column with the value onwind, so I guess that the mechanism of the problem is that some onshore wind farm happens to be very close to one of the above HVDC terminals, and gets assigned to the corresponding bus. But the carrier is DC for this node, and so with this carrier and a generator attached, it doesn't get simplified/merged into the German AC grid.

The above is kind of conjecture; I didn't really have time to dig deep into the details. I'm not super familiar with the building and simplification of the base network, so I'm not 100% where to fix the problem either.

For reference, the problem only actually surfaced a while later in the workflow for the sector-coupled network when clustering the gas network.

[tgi-climact]

Hi @koen-vg ,

I would like to share with you my problem which seems to be related to the one you have. Here is the bug description.

I am using s181_39m as configuration (with one node per region, except for one region where I have 3).

I have a working workflow on a macos machine and the same code (and virtual environment) on a linux computation server that does not work.

On the computation server, the error is raised in prepare_sector_network because a bus is not defined (DE1 0) (see error log attached). Looking into the details, I see that the busmap_s and the busmap have been built to map the buses 5770 and 5773 to DE7 0 (simplify), then DE1 0 (cluster).

de1_0_error.log

But DE1 0 is the only DC bus, which leads to its deletion in remove_elec_base_techs in prepare_sector_network. This explains why the bus is missing in prepare_sector_network.

I've checked the clustermaps (built in prepare_sector_network) and added the carrier of each bus in the initial network and the clustered network. The buses 5770 and 5773 are the only DC buses (initial) that are clustered into DC buses (cluster) (see clustermaps attached). This explains why we only encounter the problem only on DE1 0 whereas there are several DC buses in the initial network.

clustermaps_extended.csv

Comparing with the macos execution, I see that the base_network.py does not behave in the same way on both machines. More specifically, both machines are not selecting the same set of buses using the kd-tree nearest neighbours algorithm with :

tree = spatial.KDTree(buses[["x", "y"]])

On macos, the selected buses to attach TYNDP HVDC are mapped to AC buses in the clustered network. This avoid any unwanted deletion in remove_elec_base_techs in prepare_sector_network. What I am able to explain is that both set of buses (on macos or linux) are colocated. So there is some kind of operating system randomness leading to this strange behavior.

So what to do ?

1. I suggest to find a way to avoid operating system dependent output for base_network.py. The first idea that came to my mind was to reduce the set of buses that are inputs to _add_links_from_tyndp.
2. Even if both operating systems do not select the same set of buses, the rest of the workflow should work in a comparable way. This could be related to the missing load are are mentionning in the clustering stage.

@fneum Do you have any specific insights on this one ?

[koen-vg]

The issue with randomness reminds me a little of #313, but looking at the documentation it seems like scipy.spatial.KDTree isn't supposed to be random... Not sure what's going on there.

[tgi-climact]

The issue with randomness reminds me a little of #313, but looking at the documentation it seems like scipy.spatial.KDTree isn't supposed to be random... Not sure what's going on there.

Anyway, your fix works on my side too.

[koen-vg]

For the maintainers: would it be an option to comment out the link in question until a proper fix has been found?

[FabianHofmann]

For the maintainers: would it be an option to comment out the link in question until a proper fix has been found?

I saw that issue before, and my hotfix was to force all bus carriers to AC (triggered by a new config option) in simplify_network. Should I propose that fix in a PR?

[koen-vg]

I'm sure you have a better understanding of the details so if you think this is a good solution then go for it.

Just for my curiosity: are there any downsides to forcing the AC carrier, or put in another way, was there a particular point to allowing for other carriers in the first place?

[FabianHofmann]

I honestly don't know... Could you check in your case what the distribution of AC and DC bus carriers is after the simplify_network with your hotfix?

[koen-vg]

Sure I can try to have a quick look.

[tgi-climact]

I saw that issue before, and my hotfix was to force all bus carriers to AC (triggered by a new config option) in simplify_network. Should I propose that fix in a PR?

I had already thought of something like this. My idea was to filter out carriers other than AC (i.e. DC buses). But, they are not always co-located, so filtering out some DC buses will degrade the clustering. On the other hand, forcing AC as carrier on DC buses might be an option. Again, not sure of the impact.

[koen-vg]

I honestly don't know... Could you check in your case what the distribution of AC and DC bus carriers is after the simplify_network with your hotfix?

Well I can confirm that with the default configuration, applying the hotfix of removing the offending link has the consequence of leaving no buses with carrier DC in the simplified network at all. When not removing the link, there is a single bus with carrier DC, which is the one causing trouble.

[p-glaum]

I am currently investigating why the specific TYNDP line causes this error. Because I think to solve this problem sustainably, we should find another method besides just removing the TYNDP line. It is somehow connected to the fact that all the other DC buses are mapped to their AC connection point (in the method remove_stubs), and hence we do not have any DC buses after simplify_network. Only this one DC bus, 5570, is not mapped to an AC bus but stays as a DC bus.

[tgi-climact]

I am currently investigating why the specific TYNDP line causes this error. Because I think to solve this problem sustainably, we should find another method besides just removing the TYNDP line. It is somehow connected to the fact that all the other DC buses are mapped to their AC connection point (in the method remove_stubs), and hence we do not have any DC buses after simplify_network. Only this one DC bus, 5570, is not mapped to an AC bus but stays as a DC bus.

Great, thanks ! From what I've seen, it's probably due to the geographical location of the line. As I said in one of my previous comments, there is no colocated AC bus (as far as I can see).

[fneum]

This is fixed in #1067. I just wanted to highlight that we will likely switch the base network data from the ENTSO-E website to OSM data with a more uniform process of adding TYNDP and other planned transmission projects soon.