Error in rule simplify_network:Pypsa-eur

[Javier-Andres-Gonzalez]

Hi, I tried to run pypsa-eur (snakemake -call results/test-elec/networks/elec_s_6_ec_lcopt_Co2L-24H.nc --configfile config/test/config.electricity.yaml)
many times, and it keep failed in the step of simplify_network, showing "NameError: name 'config' is not defined" and "errors in rule simply_network".
Does anyone has the experience or suggestion on how to solve it?
The information about the errors in the terminal is as the follows:

Error in rule simplify_network:
jobid: 4
input: resources/networks/elec.nc, data/costs_2030.csv, resources/regions_onshore.geojson, resources/regions_offshore.geojson
output: resources/networks/elec_s.nc, resources/regions_onshore_elec_s.geojson, resources/regions_offshore_elec_s.geojson, resources/busmap_elec_s.csv, resources/connection_costs_s.csv
log: logs/test-elec/simplify_network/elec_s.log (check log file(s) for error details)
conda-env: C:\Users\romin\pypsa\pypsa-eur.snakemake\conda\8c2b696ba135d3886bd6a114623c8c83_

RuleException:
CalledProcessError in file C:\Users\romin\pypsa\pypsa-eur\rules/build_electricity.smk, line 404:
Command 'C:/Users/romin/miniconda3/envs/pypsa-earth-env/python.exe "C:\Users\romin\pypsa\pypsa-eur.snakemake\scripts\tmpzegn1oac.simplify_network.py"' returned non-zero exit status 1.
File "C:\Users\romin\pypsa\pypsa-eur\rules/build_electricity.smk", line 404, in __rule_simplify_network
File "C:\Users\romin\miniconda3\envs\pypsa-earth-env\lib\concurrent\futures\thread.py", line 58, in run
Removing output files of failed job simplify_network since they might be corrupted:
resources/connection_costs_s.csv
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-10-26T002300.707239.snakemake.log

[davide-f]

Hello @Javier-Andres-Gonzalez !
Could you please share more log? This does not give much information for comments unfortunately.
In general, please, make sure you are running the latest version; doing "git pull" can often solve many issues

[Javier-Andres-Gonzalez]

Hello @davide-f!

thank you for your prompt response.

I attach the log of the failure

2023-10-26T232821.927934.snakemake.log

Thanks

[davide-f]

Unfortunately the log is not saying much, could you post the output from the command (bash/prompt)?

[Javier-Andres-Gonzalez]

Yes @davide-f!
I am running this command:

snakemake -call results/test-elec/networks/elec_s_6_ec_lcopt_Co2L-24H.nc --configfile config/test/config.electricity.yaml

I follow what the tutorial recommends (https://pypsa-eur.readthedocs.io/en/latest/tutorial.html)

[davide-f]

Not sure exactly, but this seems more related to pypsa-eur than pypsa-earth.
Maybe you could post a question on Discord to see if somebody has experience on that.
What do you think?

[ekatef]

We had some exchange on this issue in #908.

As discussed with @Javier-Andres-Gonzalez, the problem has been connected with different design of PyPSA-Earth and PyPSA-Europe. Since Javier is interested in modeling Argentina, pypsa-earth is needed, which tutorial is available here https://pypsa-earth.readthedocs.io/en/latest/short_tutorial.html

@Javier-Andres-Gonzalez happy to be kept updated :D

As @davide-f has mentioned, feel free also to use pypsa-earth Discord, which still has a considerable knowledge base on usability issues.

[davide-f]

Apologies, I kept the two issues disconnected! Cool that the problem was identified :) I got confused by the "pypsa-eur" path in the log

[ekatef]

No, no! You are absolutely right and thank you for keeping track of the issue 🙂
I had to post the update rather here than in #908

[Javier-Andres-Gonzalez]

Hello @davide-f!
thank you for you response.
I was finally able to run the example model with "NG" AND "BJ" successfully.
Now I am trying to run the model for Argentina ("AR") but I cannot run it.
What I do is the following: from the config.yaml file of the model I ran in the example I modify the country and run that same file. Is that ok?

Error summary:

Benchmark: unable to collect cpu and memory benchmark statistics
CRITICAL:snakemake.logging:Benchmark: unable to collect cpu and memory benchmark statistics
Waiting at most 5 seconds for missing files.
WARNING:snakemake.logging:Waiting at most 5 seconds for missing files.
MissingOutputException in rule retrieve_databundle_light in file C:\Users\romin\meet\3-tercer-intento\pypsa-earth\Snakefile, line 149:
Job 9 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider
to increase the wait time with --latency-wait:
data/landcover
ERROR:snakemake.logging:MissingOutputException in rule retrieve_databundle_light in
file C:\Users\romin\meet\3-tercer-intento\pypsa-earth\Snakefile, line 149:
Job 9 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider
to increase the wait time with --latency-wait:
data/landcover
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-12-16T234513.642545.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake\log\2023-12-16T234513.642545.snakemake.log

Thanks!

[ekatef]

Hey @Javier-Andres-Gonzalez! Thanks a lot for posting the problem here. The error means that the workflow can't load the files needed to run the full-scale model. Not sure about the reasons. Probably, @davide-f may have some insights on it.

You can fix this issue by manual loading of the required files, using urls specified in bundle_config.yaml:

• common data should be loaded via one of these urls and placed into pypsa-earth/data folder;
• natura.tiff raster on protected areas can be loaded from one of these urls and put into same pypsa-earth/data folder;
• weather data for South America are available here, and should go into pypsa-earth/cutouts folder.

[Javier-Andres-Gonzalez]

Hi Ekaterina!
I tried to download the Argentina model by manually downloading the data but I got this error.

There seems to be a problem running Snakemake, specifically in the retrieve_databundle_light rule. The error indicates that some output files are missing after the job has completed successfully.

I attach images of the config.yaml file after running the tutorial.
config.docx

Thanks!!

[davide-f]

Hello :D
Great to have you here :)

May you clarify the error you are receiving?

As a comment I'd recommend to pull the latest changes, if you do so, you may be able to run the workflow smoothly from start to end. I recently did that for south America with no external inputs.
We added a cutout for southamerica, so you shouldn't need to create one on your own.
Be aware that it is downloaded from Google drive and it has limited downloads.
After the first successful retrieve databundle execution, set it to false to avoid the files to be averwritten.
To update all files, the simplest approach is to clone again the repository.
Bit do not delete the folder you already have! It can be useful.

Please keep build Natura raster to false as you can use the default one that covers the entire world.

[Javier-Andres-Gonzalez]

Davide-f!

Thank you for your quick response.

I have just finished running the AR model successfully.

I will continue now to explore with Jupyter notebook the downloaded models.

Thank you very much for all the advice.

[davide-f]

A pleasure :D