HDF5 chunks size

[mkruckow]

We deal often with big data, hence we usually read and/or process HDF5 data in chunks.
Usually this results in an optimizing problem between available RAM (with too large chunks we run out of memory) and time for the processing (with too small chunks our runtime gets limited by the many IO requests or CPU in case of heavy processing).

The questions are:

1. What shell we use as defaults?
2. How shell we communicate with a user about inefficient usage?
3. Can we partially automatize a more efficient usage? E.g. by monitoring the usage via psutils?

---

Here an example, which I encountered when doing population synthesis and will probably be relevant for other users:
I was using the export_selection function to get a subpopulation of 13336 systems out of 10^6. The default history_chunksize is 1. Running it took a long time without getting any information. After switching on the verbose of the population, I at least got some information from the tqdm which otherwise is suppressed.
I made a test for my case with different chunk sizes:

chunk size	time (in seconds)	final file size (in bytes)
(default) 1	14811.339	49946344
10	1609.308	49946344
100	2760.898	49946344
1000	759.604	49946344
10000	561.194	50003280

Some remarks: the first increase shows a clear linear improvement without any drawback. On the 100, there it looked like the filesystem was busy (the rereading of the original population took there longer, too) and slowing down the writing, otherwise I'd expect it to be faster. But clearly at some point we don't get a linear scaling anymore because other limitations kick in.
Finally, I'm not sure, why the largest chuck size (close to the number of indexes) creates a differently sized output file. I guess, there is mainly some differences in the meta data, but we'd need to make sure that there won't be any issues with the data quality for different chunk sizes.

[maxbriel]

This is a good topic to have a discussion on and your questions are a good guideline.

As an additional comment, I made the decision to set it to 1 for the history_chunk, because people will use track interpolation. This will make it difficult to keep the history of multiple binaries in memory.

However, the writing of 1 binary drastically increases the IO time.
Due to pandas handling of format=table and data_columns=True, the write time drastically increases with each append.