New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pre-processing stage key error #6
Comments
Hi everyone, I ran into the exact same error with the example code but have no idea how to solve it. Does someone know what is causing the error? |
I am searching for more than one week that what can cause this error and the most recommendation was data corruption which is not our case I think. |
I ran into the same problem running the parallel_copy_small_set.py file, and I was able to fix the issue by adding two lines in the writetofile() function:
The two lines I added are: This initializes the destination file with a shape of (timepoint, # features, latitude, longitude). The code then appends data for each time point, so in the 13 day example, you obtain a dataset with size (52,20,720,1440). Hope this fix also works for you! |
Thanks @Melissa3248. I would recommend pre-populating the empty hdf5 files. Before running the parallel_copy_small_set.py file. I will push a script to do that. It should just do the following: time_steps = 52 look through the h5py docs for more details https://docs.h5py.org/en/stable/high/dataset.html |
Thanks @Melissa3248 and @jdppthk for your solutions. Melissa her solution (with a small adjustment in the empty numpy array) only worked for the first variable, from the second variable ('v10') I then get the following error: File "/home/teun/Documents/TUD/FourCastNet/FourCastNet-0.0.0/data_process/parallel_copy_small_set_Melissa.py", line 119, in For me the pre-populating proposed by jdppthk worked! |
Glad it works for you. Closing this issue. |
First of all, thank you for your GREAT code, it is a real game changer.
I had an issue with running the pre-processing stage by running the exact datasets used inside the written example code in this stage (13 days) and still got the error which is: KeyError "Unable to open object (object 'fields' doesn't exist)"
I don't know what causes this problem because in my point of view everything must be ok with the code and datasets.
The text was updated successfully, but these errors were encountered: