Maximum data size for each CMOR output file.

[karakita-MRI]

Thank you for your valuable time.
This is question, not issue.
Is there the upper limit of each CMOR output file to submit CMIP7?
If so, we are going to prepare several separated files by a certain period,
not store all period in one file.

Tatsuro KARAKI(MRI)

[taylor13]

There is a draft data requirements document available here. You'll find in it:

Data representing a long time-series, as typically produced by a coupled model simulation, will often be split into several files, which should neither be too large (to be unwieldy) nor too small (as to create vexing I/O performance and archive management issues). Monthly data, for example, might be divided into multi-decade file segments. File sizes of order 10 GB can usually be routinely handled by modern file and cloud systems, whereas substantially larger files may become problematic for transfer (especially over flaky or unreliable networks), tape archival, replication, and some analysis workflows.

That guidance is not very specific, but I would not go much bigger than 10 GB per file. (In CMIP6 we recommended 2 GB.)

Does that answer your question?

[karakita-MRI]

Thank you for offering quick and clear answer to us.
I had not got this document, so your reply was very helpful.
CMOR3 written in Python is user-friendly, and splitting into several files may be relatively easy.

Thank you so much.

Tatsuro KARAKI(MRI)

[taylor13]

@durack1 has noted that if you expect folks to download data and process it on laptops, then files larger than, say, 2 GB will likely prove to be too big. Internet connections in many parts of the world are unreliable and will likely disrupt transfer of very large files from the CMIP7 archives. Furthermore, its not rare that a computer will have 8 GB of memory, which makes it almost impossible to work with datasets larger than a significant fraction of that. Even if a model has 1/4 deg latxlon resolution, you can store two years of uncompressed monthly 3-d (20 level) data in a 2 GB files. So, if you want folks to download and analyze your output, I would store it in files no larger than 2 GB. Of course if you have a very high resolution model, then perhaps you only expect power users to get at the data, and larger files might be acceptable to you.

(Others might advise otherwise.)

[karakita-MRI]

Thank you for very important advice.
Data providers will worry about Internet connections, too,
because they cannot necessarily make use of powerful and fast-flowing traffic on the network.

Alternatively, we have a plan that the time-slice of all variables is fixed at one year, except 6hrLev3D variables exceeding 16 GB / year, such that reanalysis data providers do.
Then, one year data is stored in a ~100MB file.
Does this time-slice treatment create vexing I/O performance and archive management issues, referred to in a draft data requirements document?

Thank you for your valuable time.

Tatsuro KARAKI(MRI)

[durack1]

Thanks for the question @karakita-MRI. Ideally, we'd like to optimise for the smallest number of files aiding their storage on ESGF/HPC systems, whilst also allowing for the broadest use across a plethora of use cases - from download and laptop analysis to targeted large-scale HPC system analysis.

I've interrogated the CMIP6 grids for two representative 3D variables, thetao 3D ocean and cl 3D atmospheric variable, using the CMIP6 grids, the following numbers can be estimated:

3D ocean thetao, 172 yr (1850–2021 - the length of the CMIP7 historical experiment) at monthly temporal frequency - Compression factor ~0.42 (masked ocean compresses well; Noting for CMIP, bytes_per_value = 4 = float32 is the standard output precision). Note, the numbers below assume standard netcdf deflate algorithms and not the heavier lossy bit-rounding routines which lose information, but also reduce file sizes more aggressively.

CMIP6 grid (x,y,z)	Model	Raw data size	Compressed data estimate	1 yr data estimate
288×180×40	GISS-E2-1-G	17 GB	~7 GB
360×180×60	E3SM, FIO	32 GB	~13 GB
360×291×45	CanESM5	39 GB	~16 GB
410×301×53	notebook mean	54 GB	~23 GB
362×292×75	EC-Earth3 , CMCC, CNRM	66 GB	~28 GB
802×404×40	MPI-ESM1-2-HR	107 GB	~45 GB	~0.25 GB
1442×1051×50	CMCC-CM2-HR4	626 GB	~263 GB
1440×1205×75	HadGEM3-GC31-MM	1.07 TB	~451 GB	~2.5 GB

3D atmos cl, 172 yr at monthly temporal frequency - Compression factor ~0.64.

CMIP6 grid (x,y,z)	Model	Raw data size	Compressed data estimate	1 yr data estimate
128×64×49	CanESM5 (T63)	3.3 GB	~2.1 GB
192×96×47	MPI-ESM1-2-LR	7.2 GB	~4.6 GB
360×180×33	GFDL-CM4	17.7 GB	~11 GB
288×192×32	CESM2	14.6 GB	~9 GB
256×128×91	CNRM, EC-Earth3	24.6 GB	~16 GB
288×192×59	notebook mean @1°	26.9 GB	~17 GB
192×144×85	ACCESS-CM2, UKESM	19.4 GB	~12 GB	~0.05 GB
288×192×70	CESM2-WACCM	32 GB	~21 GB
384×192×95	MPI-ESM1-2-HR	57.8 GB	~37 GB	~0.2 GB

For the most basic computing/download environments (laptop with 8 GB of RAM), 2-4 GB file sizes would be optimal, which is where the estimate from #981 (comment) was based. Realistically, analysis of data the size of HadGEM3-GC31-MM would likely not be possible on such a system, and it will be interesting to see what the final grids being used for CMIP7 registered models resolve to.

Note to self, discussion around ~80 GB CMIP5 data files is at #880 (comment)

[cofinoa]

One small observation.

I think it may be useful to separate two different concerns.

For file management, file size is clearly an important parameter: downloads, transfers, replication, checksums, archive/tape management, storage quotas, and user convenience all depend on it.

For interactive analysis, however, I am not sure the total file size should be considered the limiting factor. One of the main purposes of netCDF4/HDF5 chunked storage is precisely to avoid loading an entire dataset into memory. Assuming chunk-aware software, the limiting factors become the working set, chunk layout, access pattern, and chunk-cache behaviour, rather than the total size of the file itself.

In other words, I think file-size recommendations are well justified for operational and archive-management reasons, but memory limitations during analysis are more directly related to how the data are accessed than to the total size of the file.

[taylor13]

If a subset of data in a file can be reasonably easily obtained, then I guess breaking large files into smaller ones isn't necessary for that purpose. But users still need to successfully transfer full files from the data centers to their local machines, so large files may still be problematic. Might be good to try to learn how difficult in practice it is to transfer data to various places in the world.

[matthew-mizielinski]

Should it be useful here is a rough outline of how we (MOHC) determine chunk size of output files:

• avoid file sizes above 20 GB if at all possible (balance between small file size causing management overhead and reliability of data transfer mechanisms, which can struggle with large files. 10-20 GB is also the optimum file size for our tape archives)
• chunk files into one of the following temporal extents; 100/50/20/10/5/2/1 years or 6/3/1 months. Assignment of chunks is based on the "shape of the data" (number of levels, grid points and frequency). We have a different chunk size for data on all model levels compared to one pressure level at the same frequency.
• align "chunks" with 1st Jan 1850. E.g. for the amip simulation starting in 1978 and a variable that we have marked as using 10 year chunking we would produce files covering 197801-197912, 198001-198912, 199001-199912, etc.

This system isn't perfect (compression can significantly alter file sizes, so different variables with the same shape can have different sizes), but it is at least deterministic.

[karakita-MRI]

Thank you for providing very important discussions.
As a result, I can grasp more accurate intents about file sizes and numbers, more than those written in data requirements documents.

Summarizing opinions provided by Maintainers, I’m feeling like data size of order 1GB––or one-digit number GB––may be better, in terms of aiding both the download by broadest users and the storage on ESGF.
(That is, recommendation of 2 GB in CMIP6 is perhaps relaxed in CMIP7, but concerns remains as for file size of order 10 GB, although written so in the document. )

On the other hand, in ESGF total file size exceeding order 100GB can be seen frequently, so 10-20GB is reasonably approval as the optimum file size.

(welcome further opinions.)