HnSparseTreeReader - Reader with THnSparse mapping and content stored in TTree

[mvala]

The THnSparse class in ROOT provides a powerful and flexible way to store multi-dimensional histograms, especially when dealing with sparse data sets. However, directly managing very large THnSparse objects, particularly in terms of memory footprint and persistent storage, can present challenges. To address this, the HnSparseTreeReader emerges as a solution that leverages the robust data management capabilities of TTree while retaining the essential mapping and binning information provided by THnSparse. I built such ROOT object that is cappable of doing this (https://gitlab.com/ndmspc/ndmspc/-/blob/main/core/NHnSparseBase.h).

The HnSparseTreeReader should acts as an intermediary, allowing users to read the multi-dimensional structure and binning of their sparse data using the familiar THnSparse interface, but then efficiently store the actual bin content and associated weights within a TTree. This approach offers several significant advantages: improved scalability for extremely large datasets, reduced memory consumption during analysis or visualization by only loading relevant branches, and enhanced flexibility in how data is accessed and processed.

This issue will focus on the design, implementation of the HnSparseTreeReader using jsroot library.

Currently i get tree storage and mapping like this (i can store THnSparse in UserInfo if needed)

TFile *f = TFile::Open("root://eos.ndmspc.io//eos/ndmspc/scratch/hnst_cernstaff.root");
TTree *storage = (TTree*) gFile->Get("hnst")
THnSparse *mapping = ((Ndmspc::NBinning*)storage->GetUserInfo()->At(0))->GetContent()

Can you give me hint where to start?

Thanks

[linev]

If data stored in the TTree object in the ROOT file - one can try to implement efficient reading of such data in JSROOT.

JSROOT implements full support of TTree data reading. Minimal info is in tutorial:

https://github.com/root-project/jsroot/blob/master/docs/JSROOT.md#ttree-api

There are several demos - for plain HTML and for node.js.

https://jsroot.gsi.de/dev/api.htm#ttree_tselector_src
https://jsroot.gsi.de/dev/demo/read_tree.htm

For the case of THnSparse I recommend to use 'staged' approach.

On the first run one scan TTree object only for key branches to identify which TTree entry/entries contains interesting data.
And then on the second run one reads data only from these entries.
If entries are specified - JSROOT will reads only correspondent data from the file.

Output of the first stage can be preserved in the memory - so one can easily identify new data if selection changes.

Such staged approach was implemented recently in the treeDraw function of JSROOT.
And it can give enormous gain - when condition branch much smaller than data branch.

So I recommend to investigate in this direction.

[linev]

I extend TTree docu mentioning staged approach.
And provide macro which shows how manipulation with entries list can be used. Here is it:

https://github.com/root-project/jsroot/blob/master/demo/node/tree_staged.js

[mvala]

Thanks for support, i will have a look.