Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem with retrieve_data_bundle #812

Open
2 tasks done
hazemakhalek opened this issue Jul 25, 2023 · 4 comments
Open
2 tasks done

Problem with retrieve_data_bundle #812

hazemakhalek opened this issue Jul 25, 2023 · 4 comments
Labels
bug Something isn't working

Comments

@hazemakhalek
Copy link
Collaborator

Checklist

  • I am using the current main branch or the latest release. Please indicate.
  • I am running on an up-to-date pypsa-earth environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

The workflow fail to download the data through the rule retrieve_databundle_light.
Steps to reproduce the error:

  1. Clone the main branch of the repo.
  2. run snakemake -j 1 solve_all_networks

Error Message

If applicable, paste any terminal output to help illustrating your problem.
In some cases it may also be useful to share your list of installed packages: conda list.

rule retrieve_databundle_light:
    output: data/eia_hydro_annual_generation.csv, data/ssp2-2.6/2030/era5_2013/Africa.nc, data/eez/eez_v11.gpkg, data/costs.csv, data/gebco/GEBCO_2021_TID.nc, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/custom_powerplants.csv, data/hydro_capacities.csv, data/natura.tiff, cutouts/cutout-2013-era5.nc, data/landcover
    log: logs/retrieve_databundle.log
    jobid: 9
    benchmark: benchmarks/retrieve_databundle_light
    reason: Missing output files: data/eia_hydro_annual_generation.csv, data/ssp2-2.6/2030/era5_2013/Africa.nc, data/gebco/GEBCO_2021_TID.nc, data/eez/eez_v11.gpkg, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/custom_powerplants.csv, data/hydro_capacities.csv, data/natura.tiff, cutouts/cutout-2013-era5.nc
    resources: tmpdir=/tmp
INFO:snakemake.logging:rule retrieve_databundle_light:
    output: data/eia_hydro_annual_generation.csv, data/ssp2-2.6/2030/era5_2013/Africa.nc, data/eez/eez_v11.gpkg, data/costs.csv, data/gebco/GEBCO_2021_TID.nc, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/custom_powerplants.csv, data/hydro_capacities.csv, data/natura.tiff, cutouts/cutout-2013-era5.nc, data/landcover
    log: logs/retrieve_databundle.log
    jobid: 9
    benchmark: benchmarks/retrieve_databundle_light
    reason: Missing output files: data/eia_hydro_annual_generation.csv, data/ssp2-2.6/2030/era5_2013/Africa.nc, data/gebco/GEBCO_2021_TID.nc, data/eez/eez_v11.gpkg, data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif, data/custom_powerplants.csv, data/hydro_capacities.csv, data/natura.tiff, cutouts/cutout-2013-era5.nc
    resources: tmpdir=/tmp

INFO:snakemake.logging:
Can't find the repo path.
INFO:__main__:Retrieving data for 2 countries.
WARNING:__main__:DISCLAIMER LICENSES: the use of PyPSA-Earth is conditioned 
         to the acceptance of its multiple licenses.
         The use of the code automatically implies that you accept all the licenses.
         See our documentation for more information. 
         Link: https://pypsa-earth.readthedocs.io/en/latest/introduction.html#licence
INFO:__main__:Bundles to be downloaded:
	bundle_data_earth
	bundle_cutouts_africa
	bundle_natura_earth
	bundle_landcover_africa
INFO:__main__:Downloading bundle bundle_data_earth - Host zenodo
INFO:__main__:Downloading resource 'common' from cloud 'https://sandbox.zenodo.org/record/1183583/files/data_v0.1.zip?download=1'
  0%|                                                   | 0/100 [00:00<?, ?it/s]WARNING:__main__:Failed download resource 'common' from cloud 'https://sandbox.zenodo.org/record/1183583/files/data_v0.1.zip?download=1'.
  0%|                                                   | 0/100 [00:00<?, ?it/s]
INFO:__main__:Downloading bundle bundle_data_earth - Host gdrive
Downloading 1jN5iV_iaRa5lc1XOaEtletKPUgPNOVuf into /tempfile.zip... 
WARNING:__main__:Error in downloading bundle bundle_data_earth - host gdrive
ERROR:__main__:Bundle bundle_data_earth cannot be downloaded
INFO:__main__:Downloading bundle bundle_cutouts_africa - Host zenodo
INFO:__main__:Downloading resource 'cutouts' from cloud 'https://sandbox.zenodo.org/record/1223907/files/cutouts_africa.zip?download=1'
  0%|                                                   | 0/100 [00:00<?, ?it/s]WARNING:__main__:Failed download resource 'cutouts' from cloud 'https://sandbox.zenodo.org/record/1223907/files/cutouts_africa.zip?download=1'.
  0%|                                                   | 0/100 [00:00<?, ?it/s]
INFO:__main__:Downloading bundle bundle_cutouts_africa - Host gdrive
Downloading 1WHv5Dm1GtrDZj-AxJZd3T-NMIBXty3eV into /tempfile.zip... 
WARNING:__main__:Error in downloading bundle bundle_cutouts_africa - host gdrive
ERROR:__main__:Bundle bundle_cutouts_africa cannot be downloaded
INFO:__main__:Downloading bundle bundle_natura_earth - Host zenodo
INFO:__main__:Downloading resource 'natura' from cloud 'https://sandbox.zenodo.org/record/1183583/files/natura_global.zip?download=1'
  0%|                                                   | 0/100 [00:00<?, ?it/s]WARNING:__main__:Failed download resource 'natura' from cloud 'https://sandbox.zenodo.org/record/1183583/files/natura_global.zip?download=1'.
  0%|                                                   | 0/100 [00:00<?, ?it/s]
INFO:__main__:Downloading bundle bundle_natura_earth - Host gdrive
Downloading 1WmAAwY0TAcHg8YZyAGof0bPewFacm7my into /tempfile.zip... 
WARNING:__main__:Error in downloading bundle bundle_natura_earth - host gdrive
ERROR:__main__:Bundle bundle_natura_earth cannot be downloaded
INFO:__main__:Downloading bundle bundle_landcover_africa - Host zenodo
INFO:__main__:Downloading resource 'landcover' from cloud 'https://sandbox.zenodo.org/record/1183583/files/landcover_africa.zip?download=1'
  0%|                                                   | 0/100 [00:00<?, ?it/s]WARNING:__main__:Failed download resource 'landcover' from cloud 'https://sandbox.zenodo.org/record/1183583/files/landcover_africa.zip?download=1'.
  0%|                                                   | 0/100 [00:00<?, ?it/s]
INFO:__main__:Downloading bundle bundle_landcover_africa - Host gdrive
Downloading 1qnJ0HAWZ4z_9q3PtDEUrFmZWBZYg-WNW into /tempfile.zip... 
WARNING:__main__:Error in downloading bundle bundle_landcover_africa - host gdrive
ERROR:__main__:Bundle bundle_landcover_africa cannot be downloaded
INFO:__main__:Bundle successfully loaded and unzipped:
	bundle_data_earth
	bundle_cutouts_africa
	bundle_natura_earth
	bundle_landcover_africa
Waiting at most 5 seconds for missing files.
WARNING:snakemake.logging:Waiting at most 5 seconds for missing files.
MissingOutputException in rule retrieve_databundle_light in file /nfs/home/haz43975/pypsa-earth-EG/Snakefile, line 147:
Job 9  completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
data/eia_hydro_annual_generation.csv
data/ssp2-2.6/2030/era5_2013/Africa.nc
data/eez/eez_v11.gpkg
data/costs.csv
data/gebco/GEBCO_2021_TID.nc
data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif
data/custom_powerplants.csv
data/hydro_capacities.csv
data/natura.tiff
cutouts/cutout-2013-era5.nc
data/landcover
ERROR:snakemake.logging:MissingOutputException in rule retrieve_databundle_light in file /nfs/home/haz43975/pypsa-earth-EG/Snakefile, line 147:
Job 9  completed successfully, but some output files are missing. Missing files after 5 seconds. This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait:
data/eia_hydro_annual_generation.csv
data/ssp2-2.6/2030/era5_2013/Africa.nc
data/eez/eez_v11.gpkg
data/costs.csv
data/gebco/GEBCO_2021_TID.nc
data/copernicus/PROBAV_LC100_global_v3.0.1_2019-nrt_Discrete-Classification-map_EPSG-4326.tif
data/custom_powerplants.csv
data/hydro_capacities.csv
data/natura.tiff
cutouts/cutout-2013-era5.nc
data/landcover
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-07-25T124343.200107.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake/log/2023-07-25T124343.200107.snakemake.log
@hazemakhalek hazemakhalek added the bug Something isn't working label Jul 25, 2023
@hazemakhalek
Copy link
Collaborator Author

I solved the issue by changing the way the current path is fetched.

in retrieve_databundle_lightline 631

changed from rootpath = os.getcwd() to rootpath = os.path.dirname(os.path.realpath(__file__))

If the error is reproducible for other people I will issue a PR

@davide-f
Copy link
Member

Hello! Thanks @hazemakhalek for notifying, that's quite weird...
Unfortunately I cannot reproduce, how did you experience that?

@hazemakhalek
Copy link
Collaborator Author

I think Leon caught the issue. This probably happens when you rename the repo after you download it. If that's true, then it's better to either incorporate my suggestion or improve the error message to be more descriptive. @davide-f

@davide-f
Copy link
Member

So, my feeling is that I would get rid of the set_path_to_root function. I think that's the main issue.
Probably, we could revise that function to take as input a integer number stating how many folders you want to go backwards and use that.

@martacki martacki mentioned this issue Sep 5, 2023
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@davide-f @hazemakhalek