Inclusion of a sample tablite.hdf5 file?

[rhs3i]

Would it be out of the question to include an accompanying tablite.hdf5 file in the repo, possible alongside one of the tests, or from a selected point in the demo?

[root-11]

Hello @rhs3i

The hdf5 file will always be in tmp as:

H5_STORAGE = pathlib.Path(tempfile.gettempdir()) / "tablite.hdf5"

May I ask what you're trying to achieve?

[rhs3i]

Certainly. I'm the author of H5s, a scanner for HDF5. The first objective is to verify the scanner renders tablite HDF5 well. HDF5 that is intended to be primarily machine-read can stress a visual model in ways perhaps not considered, but the graphical constructs should still hold up when inspecting these files. The screenshots and links to the visual vocabulary of the scanner can give some illustration.

The second objective is to get a quick bit of insight as to whether H5s can augment usage of tablite in an interesting way, but that would be a future topic.

[root-11]

Hi Robert,
As you can see from the usage of tempdir the HDF5 files are generally used as a volatile database where data is stored in a hierarchy best described as:

1. Tables have columns
2. Columns have pagehandlers
3. Page handlers have pages.
4. Pages are of type: (a) Simple (int,float), (b) String (str, utf-8), (c) Mixed (non simple datatypes), (d) Sparse (lots of Nones)

In the tablite.hdf5-file you will therefore find that the Pages contain all the data, whilst the dataset (hdf groups) for Tables and Columns are empty and only have metadata in the attrs-field.

The details are explained here in the HDF5 group webinar: https://youtu.be/OoHVIKAD854?t=1415

[rhs3i]

Ah. Thank you for the correction and apologies for the time-wasting. I did sit for your HDF5 webinar (thank you), but I misunderstood the design, thinking that once the HDF5 backing-store had been created and stored all the computational deltas, it would persist beyond program execution and be used by a subsequent downstream tablite processor. But your presentation was clear--the re-import/reload example you showed (39:19) was from within a single program session. Scanning a volatile HDF5 datastore might have some utility in a debugging capacity, but that's another matter and may not be very useful.

Appreciate your time in getting me straightened out on this.

[root-11]

No problem. Happy I could help.