Output format update/discussion

[mvicenzi]

Reporting the issue from WenjieWu-Sci/FLArE#48.

mvicenzi
opened on Aug 3, 2024

The default output format defined in AnalysisManager is getting inadequate given that the simulation no longer contains just FLArE. For example, no information is saved from other sensitive volumes by default. This is in contrast with our push to make this toolkit available/usable by the entire FPF community.

We should revisit our plans for the output file format to be more inclusive of all detectors.
We should revisit our use of sensitive detectors: how are we planning to implement digitization? I would push to have it in an indipendent/downstream package.
We should revisit our "reconstruction" code/variables. My suggestion would be to save the recorded G4hits in the output (maybe divided according to SD?) and move every reco script outside of Geant4.

WenjieWu-Sci
on Aug 5, 2024

I completely agree. It has been in my mind for a while, but I didn't really think it through in terms of how to implement it, there are many alternative detector configurations. It requires much flexibility to the output variables.
I thought about saving the G4hits, but there are so many of them given the high energies and it results in a very large file size. I did some test before, which seems impossible to save all the hits. But maybe we can merge hits geometrically nearby?

mvicenzi
on Aug 22, 2024

Yes, it's not easy to create a flexible system given all the configurations and their possible output variables. My feeling is that it's better to keep things simple by outputting low-level info (G4hits), so that all the hit merging and hit digitization steps can be performed afterwards by detector-specific tools. However, it's true that it may not be possible to save everything... From your tests, which detector is more challeging? For example, the FLArE output could end up just being the pixelated projections (time and charge for each pixel) instead of the full 3D set of hits.

Maybe we can start by adding some infrastructure and options to save hits only from specific sensitive volumes?
It might end-up being impossible to dump all of them, but if someone is interested in a specific subdetector they can do that more easily?

I was looking at what the edep-sim output format looks like from here. For each event, there are three objects:

Primaries: The GEANT4 primary particles (A vector of TG4PrimaryVertex)

Trajectories: The GEANT4 particle trajectories (A vector of TG4Trajectory)

SegmentDetectors: The energy deposition information (A map keyed by sensitive detector name, containing a vector of TG4HitSegments).

We already have in place a vector of primaries, although I'm not sure how it would handle multiple vertexes. The hits are saved as a map using the sensitive detectors as keys, so we could potentially replicate something similar (specifying which ones to save in a macro parameter?). Regarding trajectories, they're most likley not needed unless we want to do some fancy visualization.

OlivierSalin
on Feb 20

Hi Matteo,

As we discussed in the FLArE meeting, the ACTS input for the hits might be a good reference to format the G4 output. We can find the exact format that is compatible with ACTS in those functions:
https://github.com/acts-project/acts/blob/main/Examples/Io/Root/src/RootSimHitWriter.cpp

Here you can find an example of Hits input file: https://cernbox.cern.ch/s/YorN7RqA0yWerdC

Do you think it would be straightforward to match this output ?

Best,
Olivier

benw22022
2 weeks ago

Hi All,

I've started working on some changes to the code to make it more useful from a FASER2 perspective on my fork (I'll make a PR to this repo when it's ready). For us I think what we need is a HepMC parser (this is the format that our LLP generator makes) and to change the FASER2 tracking elements from the scintillating bar-like design to simple layers of plastic. I think @mvicenzi suggestion of just keeping the hits info and keeping the digitisation and reco components separate is a good one. I can also try and look at implementing the ACTS SimHit format as Olivier suggested.
Cheers,

Ben

mvicenzi
2 weeks ago

Hi @benw22022,

Thank you for your input. Feel free to open PRs as you get things done! It would be great if you could split things in different PRs (HepMC parser, geometry changes, etc..) so that they are a bit easier to review.

I was planning to put together a new output structure, where hits are grouped by sensitive volume which can be saved or not depending on a detector names list in the configuration file. This probably doesn't solve the size problem, but at least you can switch off detectors you are not interested in. The ACTS SimHit should be easily reproducible, but it's unclear to me what to make of all these ids:

m_outputTree->Branch("volume_id", &m_volumeId);
m_outputTree->Branch("boundary_id", &m_boundaryId);
m_outputTree->Branch("layer_id", &m_layerId);
m_outputTree->Branch("approach_id", &m_approachId);
m_outputTree->Branch("sensitive_id", &m_sensitiveId);

We can easily assign a sensitive volume ID, but what's all the rest? Are all of them necessary?

benw22022
2 weeks ago

Hi @mvicenzi ,

Thanks, will do! I think I have the HepMC parser mostly sorted - will try and open a PR for that soon.
Honestly, I've not got much of a clue what all the IDs are, from what I understand they're related to how ACTS labels its geometry elements and indicates which volumes are connected to others. We can probably try and set them to zero for now and see if that works? Otherwise I think it might take some trial and error and perhaps some expert help - might be challenging since they'll need to match up with the ACTS implementation of the detector geometry

[benw22022]

Just to let people know, I've been working on the output formatting for Acts on my fork here - hoping to have a PR soonish once I validate the output works as expected in Acts

[mvicenzi]

I've given some thought to the output format, and also took a look at #8 (nice work, @benw22022 !).
I believe I have a proposal for a final structure that offers enough flexibility to accommodate all detectors.

I propose having three common TTrees:

• An event TTree that stores high-level event information for each event ID. This should include generator-level info: the type of generator used (genie, foresee\hepmc) and the physics input to that generator (e.g., decay or neutrino kinematics). It might require some thought on what and how to store information in a consistent format, given the differences between generators.
• A primaries TTree that stores the primary particles for each vertex in the event. The structure can follow what was laid out in Output reformatting for Acts tracking  #8 and is required by ACTS. In principle, the same info could be stored in the event tree, but I assume ACTS requires its own tree. While FPFNeutrino and FPFParticle help organize things, I’d argue that a flat ROOT tree is much easier to interpret downstream, so I advocate against using custom classes.
• An optional trajectories TTree that stores tracks for all particles in the event. This should be disabled by default but made available for use with FPFDisplay. Given my plan for saving hits, we’ll need this additional extended output to provide the right input for a proper event display.

In addition to these common trees, we also need to store the hits from the event. For this, I believe we should stick with the sensitive volume approach that @benw22022 is already using in #8. Essentially:

• Each detector defines its own sensitive detector classes, along with the corresponding hit classes (LArBoxSD + LArBoxHit, FASER2TrackerSD + FASER2TrackerHit, etc.), and assigns them to the appropriate logical volumes.
• These classes should be tailored to store whatever information each subdetector needs, including geometry data (e.g., volumeID/copyNumber) to track the specific physical volume that was crossed.
• The "active" sensitive detectors are listed in the SDNamelist within GeometricalParameters. This list is then used by AnalysisManager to determine which additional trees should be saved in the output.
• To keep things organized, each detector -- if enabled via its SD(s) in the list -- should save its output TTrees in a subdirectory of the main ROOT file. Generally, one SD = one output TTree in that directory, but each detector can retain flexibility in how it stores its output. The format of these trees should be ACTS-compatible whenever possible, but alternative formats can be used if needed (e.g., histograms/images for FLArE?).

In summary, an FPFSim output file would look like this:

TFile filename.root
|- TTree event
|- TTree primaries
|- TTree trajectorires [OPTIONAL]
|- TDirectory flare
  |- TTree LArBoxSDTree
  |- TTree BabyMINDSDTree
...
|- TDirectory faser2
  |- TTree TrackerSDTree
  |- TTree VetoSDTree
...

I don’t expect all of this to be implemented in #8, obviously. We’ll certainly need another PR to de-FLArE-ize the current event tree and move relevant parts into a dedicated FLArE-only tree. That said, I believe this should be our final goal.

Any comments on this structure? @OlivierSalin @WenjieWu-Sci @benw22022

[benw22022]

Hi @mvicenzi, I like this idea! I think there is some flexibility in how we format the output trees for Acts - I found that I needed to implement my own ROOT file readers for Acts anyway so we could probably customise those in case we want to make the output format more versatile

[WenjieWu-Sci]

Hi @mvicenzi, I like the thoughts. Regarding the usage of FPFNeutrino and FPFParticle, the motivation was to pass information in a more organized way. These information don't have to be saved in the files as the format of custom classes. I agree flat ROOT Ntuple is more convenient for downstream processing. But I think we can still use this format in AnalysisManager or whatever algorithms that use this information on the fly.
Out of curiosity, to produce the event display, what are the file sizes with all hits saved?

[mvicenzi]

Out of curiosity, to produce the event display, what are the file sizes with all hits saved?

FPFDiplay currently takes as input the trajectories TTree. For 5 numu genie events, I get on average ~45 MB/event. This saves all the tracks, including those outside of the sensitive volumes. I haven't checked how it compares with saving all hits.