Error in downloading bundle bundle_cutouts_mainland_africa - host gdrive (when run pypsa-earth for Kenya)

[Haiping007]

Hi, I have run smoothly run pypsa-earth for Kenya before. But today, there are showing errors about 'cutouts' when I run pypsa-earth for Kenya again with modified CO2 emission constrain. (I tested the tutorial with 'NG' and 'BJ' which is with no problem.)
Could you help me? Thank you!

The detailed information of the errors are as follows:

INFO:snakemake.logging:
This is the repository path: C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth
Had to go 0 folder(s) up.
INFO:main:Retrieving data for 1 countries.
WARNING:main:DISCLAIMER LICENSES: the use of PyPSA-Earth is conditioned
to the acceptance of its multiple licenses.
The use of the code automatically implies that you accept all the licenses.
See our documentation for more information.
Link: https://pypsa-earth.readthedocs.io/en/latest/introduction.html#licence
INFO:main:Bundles to be downloaded:
bundle_natura_earth
bundle_cutouts_mainland_africa
bundle_data_earth
bundle_landcover_africa
INFO:main:Downloading bundle bundle_natura_earth - Host zenodo
INFO:main:Downloading resource 'natura' from cloud 'https://sandbox.zenodo.org/record/1183583/files/natura_global.zip?download=1'
0%| | 0/100 [00:00<?, ?it/s]WARNING:main:Failed download resource 'natura' from cloud 'https://sandbox.zenodo.org/record/1183583/files/natura_global.zip?download=1'.
0%| | 0/100 [00:00<?, ?it/s]
INFO:main:Downloading bundle bundle_natura_earth - Host gdrive
Downloading 1WmAAwY0TAcHg8YZyAGof0bPewFacm7my into C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\tempfile.zip...
78.2 MiB Done.
INFO:main:Download resource 'natura' from cloud 'https://drive.google.com/file/d/1WmAAwY0TAcHg8YZyAGof0bPewFacm7my/view'.
INFO:main:Downloading bundle bundle_cutouts_mainland_africa - Host zenodo
INFO:main:Downloading resource 'cutouts' from cloud 'https://sandbox.zenodo.org/record/1223907/files/cutouts_africa.zip?download=1'
0%| | 0/100 [00:00<?, ?it/s]WARNING:main:Failed download resource 'cutouts' from cloud 'https://sandbox.zenodo.org/record/1223907/files/cutouts_africa.zip?download=1'.
0%| | 0/100 [00:00<?, ?it/s]
INFO:main:Downloading bundle bundle_cutouts_mainland_africa - Host gdrive
Downloading 1WHv5Dm1GtrDZj-AxJZd3T-NMIBXty3eV into C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\tempfile.zip...
Done.
WARNING:main:Error in downloading bundle bundle_cutouts_mainland_africa - host gdrive
ERROR:main:Bundle bundle_cutouts_mainland_africa cannot be downloaded
INFO:main:Downloading bundle bundle_data_earth - Host zenodo
INFO:main:Downloading resource 'common' from cloud 'https://sandbox.zenodo.org/record/1183583/files/data_v0.1.zip?download=1'
0%| | 0/100 [00:00<?, ?it/s]WARNING:main:Failed download resource 'common' from cloud 'https://sandbox.zenodo.org/record/1183583/files/data_v0.1.zip?download=1'.
0%| | 0/100 [00:00<?, ?it/s]
INFO:main:Downloading bundle bundle_data_earth - Host gdrive
Downloading 1jN5iV_iaRa5lc1XOaEtletKPUgPNOVuf into C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\tempfile.zip...
2.2 GiB Done.
INFO:main:Download resource 'common' from cloud 'https://drive.google.com/file/d/1jN5iV_iaRa5lc1XOaEtletKPUgPNOVuf/view'.
INFO:main:Downloading bundle bundle_landcover_africa - Host zenodo
INFO:main:Downloading resource 'landcover' from cloud 'https://sandbox.zenodo.org/record/1183583/files/landcover_africa.zip?download=1'
0%| | 0/100 [00:00<?, ?it/s]WARNING:main:Failed download resource 'landcover' from cloud 'https://sandbox.zenodo.org/record/1183583/files/landcover_africa.zip?download=1'.
0%| | 0/100 [00:00<?, ?it/s]
INFO:main:Downloading bundle bundle_landcover_africa - Host gdrive
Downloading 1qnJ0HAWZ4z_9q3PtDEUrFmZWBZYg-WNW into C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\tempfile.zip...
46.8 MiB Done.
INFO:main:Download resource 'landcover' from cloud 'https://drive.google.com/file/d/1qnJ0HAWZ4z_9q3PtDEUrFmZWBZYg-WNW/view'.
INFO:main:Bundle successfully loaded and unzipped:
bundle_natura_earth
bundle_cutouts_mainland_africa
bundle_data_earth
bundle_landcover_africa
Benchmark: unable to collect cpu and memory benchmark statistics
CRITICAL:snakemake.logging:Benchmark: unable to collect cpu and memory benchmark statistics
Waiting at most 5 seconds for missing files.
WARNING:snakemake.logging:Waiting at most 5 seconds for missing files.
MissingOutputException in rule retrieve_databundle_light in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 149:
Job 9 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
cutouts/cutout-2013-era5.nc
ERROR:snakemake.logging:MissingOutputException in rule retrieve_databundle_light in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 149:
Job 9 completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
cutouts/cutout-2013-era5.nc
Removing output files of failed job retrieve_databundle_light since they might be corrupted:
data/ssp2-2.6/2030/era5_2013/SouthAmerica.nc, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/ssp2-2.6/2030/era5_2013/Asia.nc, data/custom_powerplants.csv, data/eia_hydro_annual_generation.csv, data/eez/eez_v11.gpkg, data/ssp2-2.6/2030/era5_2013/Africa.nc, data/ssp2-2.6/2030/era5_2013/NorthAmerica.nc, data/ssp2-2.6/2030/era5_2013/Oceania.nc, data/natura.tiff, data/hydrobasins/hybas_world_lev05_v1c.shp, data/hydrobasins/hybas_world_lev04_v1c.shp, data/hydrobasins/hybas_world_lev06_v1c.shp, data/ssp2-2.6/2030/era5_2013/Europe.nc, data/hydro_capacities.csv, data/gebco/GEBCO_2021_TID.nc, data/costs.csv, data/landcover
WARNING:snakemake.logging:Removing output files of failed job retrieve_databundle_light since they might be corrupted:
data/ssp2-2.6/2030/era5_2013/SouthAmerica.nc, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/ssp2-2.6/2030/era5_2013/Asia.nc, data/custom_powerplants.csv, data/eia_hydro_annual_generation.csv, data/eez/eez_v11.gpkg, data/ssp2-2.6/2030/era5_2013/Africa.nc, data/ssp2-2.6/2030/era5_2013/NorthAmerica.nc, data/ssp2-2.6/2030/era5_2013/Oceania.nc, data/natura.tiff, data/hydrobasins/hybas_world_lev05_v1c.shp, data/hydrobasins/hybas_world_lev04_v1c.shp, data/hydrobasins/hybas_world_lev06_v1c.shp, data/ssp2-2.6/2030/era5_2013/Europe.nc, data/hydro_capacities.csv, data/gebco/GEBCO_2021_TID.nc, data/costs.csv, data/landcover
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-10-23T223101.674443.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake\log\2023-10-23T223101.674443.snakemake.log

[davide-f]

Hello @Haiping007 !
Thanks for writing!
Unfortunately, those issues are related to the fact that sandbox zenodo, our major data storage platform is having problems and the backup option (gdrive) has limitations in the number of access a day.
As an alternative approach, you could manually download those databundles as zip file and unzip them manually.
After having done that, in the config, you can safely specify enable->retrieve_databundle: false so to avoid snakemake to execute the retrieve databundle again.

The gdrive links are in the error message above and:

• the content of the cutouts zip file shall be placed into the folder cutouts
• the content of the resources zip file shall be unzipped in the resources folder
• the content of the common zip files shall be placed into the data folder
• the content of landcover zip file shall be placed into data/landcover folder

Note: if you login in gdrive you should easily be able to access and download those files manually, but the automatic procedure unfortunately does't work due to download limits with no login

Let us know if you encounter further issue

[Haiping007]

Thank you! @davide-f

Actually I have downloaded the cutouts files (cutout-2013-era5.nc) in the first run with Kenya before , but it kept showing errors related to the files cutout-2013-era5.nc and profile_solar.nc. So I set a new folder of 'pypsa-earth' for Kenya and run from the beginning with set back to 'true' in config.yaml yesterday.

1. Today, I tried again the old 'pypsa-earth' of Kenya with all the files downloaded already. The first two runs look fine. But after I adjusted the number of cluster to 15 (10 and 20 were tried before and both fine), the 'ERROR:snakemake.logging:MissingInputException in rule cluster_network' showed up:

---

(pypsa-earth) C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth>snakemake -j 1 solve_all_networks
Set parameter Username
Academic license - for non-commercial use only - expires 2024-09-10
No parameters matching '_test' found
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\networkclustering.py:16: UserWarning: The namespace pypsa.networkclustering is deprecated and will be removed in PyPSA v0.24. Please use pypsa.clustering.spatial instead.
warnings.warn(
Building DAG of jobs...
WARNING:snakemake.logging:Building DAG of jobs...
MissingInputException in rule cluster_network in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 680:
Missing input files for rule cluster_network:
output: networks/elec_s_15.nc, resources/bus_regions/regions_onshore_elec_s_15.geojson, resources/bus_regions/regions_offshore_elec_s_15.geojson, resources/bus_regions/busmap_elec_s_15.csv, resources/bus_regions/linemap_elec_s_15.csv
wildcards: simpl=, clusters=15
affected files:
data/costs.csv
ERROR:snakemake.logging:MissingInputException in rule cluster_network in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 680:
Missing input files for rule cluster_network:
output: networks/elec_s_15.nc, resources/bus_regions/regions_onshore_elec_s_15.geojson, resources/bus_regions/regions_offshore_elec_s_15.geojson, resources/bus_regions/busmap_elec_s_15.csv, resources/bus_regions/linemap_elec_s_15.csv
wildcards: simpl=, clusters=15
affected files:
data/costs.csv

2. Then I tried to set the number of cluster to 20 for testing, the 'ERROR:snakemake.logging:MissingInputException in rule prepare_network' showed up. Don't know how to fix them?

---

(pypsa-earth) C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth>snakemake -j 1 solve_all_networks
Set parameter Username
Academic license - for non-commercial use only - expires 2024-09-10
No parameters matching '_test' found
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\networkclustering.py:16: UserWarning: The namespace pypsa.networkclustering is deprecated and will be removed in PyPSA v0.24. Please use pypsa.clustering.spatial instead.
warnings.warn(
Building DAG of jobs...
WARNING:snakemake.logging:Building DAG of jobs...
MissingInputException in rule prepare_network in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 751:
Missing input files for rule prepare_network:
output: networks/elec_s_20_ec_lcopt_Co2L-3H.nc
wildcards: simpl=, clusters=20, ll=copt, opts=Co2L-3H
affected files:
data/costs.csv
ERROR:snakemake.logging:MissingInputException in rule prepare_network in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 751:
Missing input files for rule prepare_network:
output: networks/elec_s_20_ec_lcopt_Co2L-3H.nc
wildcards: simpl=, clusters=20, ll=copt, opts=Co2L-3H
affected files:
data/costs.csv

[davide-f]

I'm unsure on the error, but please make sure to use the latest version of the repository.
I've been running Kenya recently and it seems working too. I believe it may be a version-related issue.
We recently released a new pypsa-earth version too. That may solve the issue.

Regarding the issue you are presenting, it seems that some input files are missing (data/costs.csv) that file may have been deleted by mistake maybe

[Haiping007]

Hi, @davide-f
thank you for your advise.
I found there is 'costs.csv' but in the folder 'resource'. So I copied it into the folder 'data'.
Now it is working!
Thank you!