Speeding up MERRA data access

MERRA datasets are subdivided into daily files. This means that requests over long time periods must make hundreds of serial server requests. The time to connect to the server is often much greater than the time to download the dataset.

following the instructions here for creating cookie files: if you get a bunch of MERRA URLs using a modified MERRA subsetter, you can download them in (kind-of) parallel with the following:

cat ../merra-wishlist.txt | xargs -n 1 -P 8 wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition

where -P 8 uses 8 threads (could use more - I haven't tried)

then you need to merge them together

cdo mergetime MERRA2_*00.inst3_3d_asm_Np* globsim_merra2.inst3_3d_asm.nc
cdo mergetime MERRA2_*00.inst6_3d_ana_Np* globsim_merra2.inst6_3d_ana_Np.nc
cdo mergetime MERRA2_*00.inst3_3d_asm_Np* globsim_merra2.inst3_3d_asm_Np.nc
cdo mergetime MERRA2_*00.inst1_2d_asm_Nx* globsim_merra2.inst1_2d_asm_Nx.nc
cdo mergetime MERRA2_*00.tavg1_2d_flx_Nx* globsim_merra2.tavg1_2d_flx_Nx.nc
cdo mergetime MERRA2_*00.tavg1_2d_rad_Nx* globsim_merra2.tavg1_2d_rad_Nx.nc
cdo mergetime MERRA2_*00.tavg1_2d_slv_Nx* globsim_merra2.tavg1_2d_slv_Nx.nc

probably want to include the time range here. You could and delete the old ones