Inference results using ERA5 data

[Ting-Yun-Huang]

Hi, thanks for releasing the strong foundation model!

I am trying to get the inference result of the released AuroraSmall checkpoint using ERA5 data.
I download the ERA5 data from Climate Data Store, which has 4 surf-vars and 5 atmos-vars with 8 pressure levels (differenct from the settings in the paper).
After that, I follow the guidance in the document to build the inference pipeline.
Also, I implement a RMSE metric by myself to evaluate the results.

However, I get a pretty bad performance on some of the variables.
I have known that the result is sensitive to different combinations of input variables, and I believe that the optimized inference result need to be generated by HRES-T0 dataset.
I would like to check if there is anything else I miss during my implementation that leads to the bad performance.
Or could you provide the inference result (the RMSE value of each variable, ex. 2t, 10u, t850, u500, ...) using a year (maybe 2020) of ERA5 data as a reference value?

[wesselb]

Hey @Ting-Yun-Huang! Thanks for opening an issue.

First of all, I should say that the small model should not be used for anything other than debug purposes. We did not carefully evaluate the model, so there are no guarantees about its performance.

That said, it also should not fail horribly. Did you include all the variables required for the main pretrained model? See this page. (Specifically, note that certain static variables are required.)

Could you visualise the predictions on your side to make sure that it at least visually looks reasonable?

[Ting-Yun-Huang]

Hi @wesselb,

Here are my previous input variables:
surf vars: 2t, 10u, 10v, msl
static vars: lsm, slt, z
atmos vars: t, u, v, q, z
Pressure levels (hPa): 50, 150, 300, 500, 700, 850, 925, 1000
region: global (720*1440)
Only the pressure levels are different from the website.
Belows are my previous results:

model	leadtime	data	2t	10u	10v	msl	t_50	t_150	t_300	t_500	t_700	t_850	t_925	t_1000	u_50	u_150	u_300	u_500	u_700	u_850	u_925	u_1000	v_50	v_150	v_300	v_500	v_700	v_850	v_925	v_1000	q_50	q_150	q_300	q_500	q_700	q_850	q_925	q_1000	z_50	z_150	z_300	z_500	z_700	z_850	z_925	z_1000
released global AS	lead6step1	8level-Global-202101	0.744729	0.637743	0.656857	55.78411	2.720657	3.369173	2.660106	4.137735	1.380356	0.80525	0.830077	1.134452	4.312917	4.208494	4.512932	3.465234	1.919414	1.103513	1.096128	1.018436	3.646721	4.077079	4.157751	3.321467	1.81774	1.096865	1.090493	0.976778	7.64E-08	1.37E-06	7.02E-05	0.000355	0.000569	0.000492	0.000489	0.000412	908.9406	643.2916	812.0091	623.0675	321.1534	60.53449	63.35052	68.89322

As shown in the table, z_50, z_150, z_300, z_500, and z_700 get extremely high values, which is kind of wierd for me.

But after I try to inference by ERA5 data with the same 13 pressure levels on the website, all the variables get reasonable values now. So I think the problem of my previous result is just caused by the different input pressure levels.
Thanks for your fast reply!

[wesselb]

Hey @Ting-Yun-Huang!

I'm glad that you got it to work. :)

But after I try to inference by ERA5 data with the same 13 pressure levels on the website, all the variables get reasonable values now. So I think the problem of my previous result is just caused by the different input pressure levels.

Yes, you will certainly need to use the 13 pressure levels stated on the website.

If you want to use a different collection of pressure levels, then that is also possible, but you will need to do some light fine-tuning.

[randy-chase]

tangential question for @wesselb, the output of aurora is (720 by 1440), but the input is (721 by 1440), do we chop off the first index or the last index to compare truth ERA5 to aurora outputs

[wesselb]

@randy-chase the final latitude is chopped off, corresponding to the south pole.