Conversion of llc4320 data

[hoang-dt]

Hi @dbalouek and @Zhang690683220 ,

As discussed earlier, I think one of the immediate goals is to convert a large dataset from NASA to OpenViSUS (or idx2 later).

The LLC4320 dataset is supposedly 4PB uncompressed and can be obtained by the following command (on NAS):

/home4/bcnelson/MITgcm/extract/latest/uncompress4320 -L 1 10368-19584 U

The above command extracts field U, time steps 10368 to 19584 , for a total of 64 time steps (each increases the timestep label by 144, so 19584 = 10368 + 144 * 64).
The full details can be found in data-descriptions.

Once the raw data is extracted, it can be converted to OpenViSUS (see conversion). @aashishpanta0 should know the details of this conversion.

We can also transfer parts of the data from NAS to SCI for easier testing if needed (see data transferring).

[aashishpanta0]

I can definitely start with the conversion, but it would be great if we could start testing the parallel conversions for these datasets @Zhang690683220

[bozhang-hpc]

Where can I access this dataset? I'm still working on one of the NASA CMIP6 .nc file.
BTW, I didn't try MPI on the NASA AWS. So it's not clear if we can do it parallel.

[aashishpanta0]

The extracted data are available at : /nobackupp19/vpascucc/llc4320_raw. The total size is 393TB for 1116 timesteps and 90 depths.
The converted IDX data are in the NAS server at: /nobackupp19/vpascucc/llc4320_raw.