-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't download data when runing .sh file #6
Comments
@bryandunn614, the Are you sure you need the CMIP6 data for your project? It is very large and doesn't lead to a substantial performance boost (see our paper). Good luck with the SEAS5 access request. You can also use the SEAS5 performance metrics that we computed by downloading the paper generated data. This would save you a lot of time. |
FYI I have updated the README to make it more clear that the parallel bash scripts run python processes in the background, and state the log file folder paths: 4f5fb27 |
Hello @tom-andersson and @JimCircadian : Rotating wind data in data/cmip6/EC-Earth3/r2i1p1f1 I have checked all data/cmip6 subfiles every folder just contains a single siconca_latlon.nc file,no uas_EASE_cmpr.nc is generated I think because download_cmip6_data.py doesn't generate or download uas_EASE_cmpr.nc correctly.All I modify to the project code is just comment out line 310 of download_cmip6_data.py(query['data_node'] = data_node),referring to another issue raised by @xinaesthete Thank you for your help |
nco should be installed to run the download_seas5_forecasts.py successfully I tried to find what's wrong with the code,but everything seems to be fine.So I just search on chatgpt |
Hi @bryandunn614.
The issue with downloading MRI-ESM2.0 data not returning files has now been fixed (#4). Can you try |
Regarding the I ran I did look into using There will be a workaround to avoid a large memory footprint during regridding (e.g. by looping over the data in years) but I don't have time to implement this unfortunately. Your best bet will be to move to a HPC where memory is not a constraint. |
Hello @tom-andersson :
I have checked the disk space,it finished downloading and did not generate anything on the disk space I don't think it is normal that single ./download_seas5_forecasts_in_parallel.sh can take up 190GB memory space |
Hi @bryandunn614 190 GB sounds correct given what is happening computationally. For each of the 6 lead times, the SEAS5 lat/lon forecast is 1440x360x240x25 which is roughly 13 GB when loaded uncompressed into memory. I ran You can modify the SEAS5 parallel download script to run the commands in sequence rather than in parallel by changing the |
Thank you @tom-andersson for your reply,
I am wondering whether it is caused by run 5 programs parallelly(I only run 5 EC-Earth3 download commands) caused running out of memory(190GB memory ) which I checked is not this kind of situation or this is because r14i1p1f1 and r12i1p1f1 should be changed to download data successfully.This program just used up my single node usage time (10 hours) and did not report failed.The download speed should be fast,because I just connect to the supercomputer server node based in Australia to run the command. Thank your for help! |
Hi @bryandunn614, I can't reproduce your error:
It's possible there was a connection issue during the download, either from the ESGF data node or your HPC. Did you try running the download for the missing variables again? |
Hello @tom-andersson I don't think this issue can be ignored
If I just ignored the issue I will have a error when runnin the biascorrect_seas5_forecasts.py command: I have successfully run the above commands,but ran into this problem. |
Hello Mr.Andersson:
./download_era5_data_in_parallel.sh
./download_cmip6_data_in_parallel.sh
./rotate_wind_data_in_parallel.sh
./download_seas5_forecasts_in_parallel.sh
I am still waiting for ECMWF to approve my access data application,but I already set up a CDS account and populate .cdsapirc file according to the guidance.
it seems that
./download_era5_data_in_parallel.sh
./download_cmip6_data_in_parallel.sh doesn't download anything.
I am confused why these two .sh files do not work.
Then when I execute python3 icenet/biascorrect_seas5_forecasts.py
![图片2](https://private-user-images.githubusercontent.com/129260579/239776927-f71ecd7e-0a40-417b-a836-d2c33c4548f9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIzNjA2MzYsIm5iZiI6MTcyMjM2MDMzNiwicGF0aCI6Ii8xMjkyNjA1NzkvMjM5Nzc2OTI3LWY3MWVjZDdlLTBhNDAtNDE3Yi1hODM2LWQyYzMzYzQ1NDhmOS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDczMFQxNzI1MzZaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kMjFlNWJiYWIyZTNhNTU1OThiOTk1ZTM2MzMzMjk4OGExZmEwMzA5YmNmMWY4YzA3YWY4OWY4OWYwMDdmMzAxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.0NQJSdn3DsNagrUw38l4NhmR5emyFeSEQh6POh1Rfp0)
I know because I haven't got access to ECMWF data,but it seems that I couldn't download era5 and cmip6 data either.
gen_masks.py/download_sic_data.py can work and download data correctly
Thank you for your help
The text was updated successfully, but these errors were encountered: