MissingRuleException issued when changing the region

[ekatef]

Checklist

• I am using the current main branch or the latest release. Please indicate.
• I am running on an up-to-date pypsa-earth environment. Update via conda env update -f pypsa-earth/envs/environment.yaml.

Describe the Bug

When changing the country (e.g. from NG to KZ or CN), an attempt to run solve_network rule leads to MissingRuleException which randomly complains about missing data files or a cutout file.

All the data files are placed in "pypsa-distribution/pypsa-earth/data" and "pypsa-distribution/pypsa-earth/cutouts", and are fetched if "NG" is specified.

Error Message

(pypsa-earth-upd16) pypsa-distribution % snakemake -j 1 solve_network  
Restricted license - for non-production use only - expires 2024-10-28
No parameters matching '_test' found
Building DAG of jobs...
Executing subworkflow pypsaearth.
~/opt/miniconda3/envs/pypsa-earth-upd16/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
Config file configs/bundle_config.yaml is extended by additional config specified via the command line.
WARNING:snakemake.logging:Config file configs/bundle_config.yaml is extended by additional config specified via the command line.
Building DAG of jobs...
WARNING:snakemake.logging:Building DAG of jobs...
MissingRuleException:
No rule to produce data/gebco/GEBCO_2021_TID.nc (if you use input functions make sure that they don't raise unexpected exceptions).
ERROR:snakemake.logging:MissingRuleException:
No rule to produce data/gebco/GEBCO_2021_TID.nc (if you use input functions make sure that they don't raise unexpected exceptions).

Depending on a run, there may be also data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif or cutouts/africa-2013-era5-tutorial.nc reported as a missed file.

[ekatef]

The issue is caused by data workflow settings for pypsa-earth sub-workflow. The following step allow to run the workflow.

1. Add a scenario config to pypsa-distribution/pypsa-earth/configs/scenarios. Currently there is a config for Nigeria indexed by git with this commit.

There are also a number of git-tracked configuration files, in particular config.distribution.yaml and config.pypsa-earth.yaml, as well as pypsa-earth/config.yaml, but I reckon they are not relevant for setup a particular workflow run.

2. Change definitions for some input files from sub-workflow to direct path definition. Here is a suggestion for the changes which can be used to fix that. The idea is to keep sub-workflow only for those input files which should be produced by the sub-workflow.

[ekatef]

@davide-f thanks a lot for discussion and fixes.

After some additional testing, I confirm that content of pypsa-distribution/pypsa-earth/configs/scenarios does not have any effect on this issue.

The issue with path definitions is not perfectly reproducible: it persisted when the workflow has been re-run using the previously loaded data (those placed in pypsa-distribution/pypsa-earth/data), but has disappeared for them after these data were removed manually and loaded agaian using pypsa-earth submodule. The issue for cutout still exists.

Change of the paths to local ones may be used as quick-and-dirty fix, while some in-depth investigation may be helpful to understand what is going on there. I suspect, an upcoming release of Snakemake may change a lot there. So, probably it's just worth to wait a bit.

[ekatef]

As the last resort, it's possible to reset pypsa-earth submodule using some git magic:

git submodule deinit -f --all
git submodule update --init

[ekatef]

Closed as fixed with #36