ADD: Support for loading version 2 of the HDF hit info by fhagemann · Pull Request #100 · cositools/nuclearizer

fhagemann · 2026-02-26T07:58:15Z

This is my attempt to implement a HDF loader for the newest version 2 of the HDF hit into.
I'm not claiming that this is perfect, but it seems to give something reasonable at least..
It also fixes #57.

I developed this based on a file that was taken with HDF v2.3 and GSE v7.2.2 on the FM assembly at SSL.

The changes between versions 1 and 2 addressed in this PR are:

where the HDFVersion is saved in the file

Version 1 Version 2

HDFVersion is its own dataset within the file. HDFVersion is an attribute of the Events dataset.

🗂️ HDF5.File (version 1)
  ├─ 🏷️ CLASS
  ├─ 🏷️ FILTERS
  ├─ 🏷️ PYTABLES_FORMAT_VERSION
  ├─ 🏷️ TITLE
  ├─ 🏷️ VERSION
  ├─ 🔢 Buffers
  ├─ 🔢 Config
  ├─ 🔢 GSEVersion
  ├─ 🔢 HDFVersion
  │  ├─ 🏷️ CLASS
  │  ├─ 🏷️ FIELD_0_FILL
  │  ├─ 🏷️ FIELD_0_NAME
  │  ├─ 🏷️ NROWS
  │  ├─ 🏷️ TITLE
  │  └─ 🏷️ VERSION
  ├─ 🔢 Headers
  └─ 🔢 Hits

  🗂️ HDF5.File (version 2)
  ├─ 🔢 ACSHits
  ├─ 🔢 Buffers
  ├─ 🔢 DIBCoincidence
  ├─ 🔢 DetectorHits
  ├─ 🔢 DetectorLiveTime
  ├─ 🔢 EventIndices
  ├─ 🔢 Events
  │  ├─ 🏷️ BackendVersion
  │  ├─ 🏷️ Config
  │  ├─ 🏷️ Firmware
  │  ├─ 🏷️ GSEVersion
  │  └─ 🏷️ HDFVersion
  ├─ 🔢 FEEHits
  ├─ 🔢 H&S
  ├─ 🔢 Headers
  ├─ 🔢 RTBHousekeeping
  ├─ 🔢 SCBHousekeeping
  └─ 🔢 SinglesCounts

--> If no dataset HDFVersion exists, check for the existence of Events and attribute HDFVersion therein.

Hit and event information are now stored in different datasets

Version 1	Version 2
Both, hit and event information was stored in the dataset `Hits`.	The hit information is saved in the dataset `FEEHits` and the event information is saved in the dataset `Events`. Hit entries in `FEEHits` have a field `event_index` which denotes which "row" in the `Events` dataset correspond to the given hit.

For clarification:

Hit information = strip ID, energy data, timing data, hit type, timing type
Event information = event ID, timecode, GSE timecode, SPW timecode, number of hits, ...

--> use the same buffer logic for hit information, but read in the event information as a whole.

Right now, the handling of SPWTimeCode (spacewire time code) is commented out to avoid undeclared variable warnings.

I would be very grateful for feedback! :)

zoglauer · 2026-02-26T16:37:43Z

Thanks for starting this and taking it off my todo list, but there are a few more things to do:
(1) Based on info in the JSON ASIC setup file, we have to switch between strip maps - that is what Clio was supposed to implement in python and we just replicate it - but she thinks there is still a bug in her code - that is what we talked about on Tuesday.
(2) We need all the other info too: Live time, ACS hit info, etc.

fhagemann · 2026-02-26T16:40:56Z

Thanks for the feedback, I will try to look into this.
Should this all become one big PR (this PR) and review all the code at the end?
Or do you want to tackle this in multiple PRs, each doing once change at a time, to keep reviews manageable?

zoglauer · 2026-02-26T16:41:40Z

I don't have v2.3 but in previous 2.x versions, there was an EventIndices table "Maps each packet in Events to a [start_index, end_index) range in each hit table that contains the hits from that event. An event at index i in the
Events table corresponds to the indices listed at index i in this table."
If it is still there in version 2.3, I don't see it in your code, to do the right linking

zoglauer · 2026-02-26T16:53:00Z

I think we have to do (1) and the EventIndices if they are still in version 2.3 - because otherwise the output is not correct. And then we can do the rest of the content in a later PR

fhagemann · 2026-03-05T21:24:06Z

With the latest commit, I seem to be able to read the polarities from all 3 primary/secondary asics for all 16 detectors in the Config attribute of Events:

Geometry materials loaded!
Detector ID 0:
  primary: HV HV HV 
  secondary: LV LV LV 
Detector ID 1:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 2:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 3:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 4:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 5:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 6:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 7:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 8:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 9:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 10:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 11:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 12:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 13:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 14:
  primary: LV LV LV 
  secondary: LV LV LV 
Detector ID 15:
  primary: LV LV LV 
  secondary: LV LV LV

I still need to implement using this information for strip mapping.

fhagemann · 2026-03-05T22:04:17Z

From my understanding: instead of "switching" between strip maps, the ASIC polarities are used to update if an ASIC is on HV or LV side, by ignoring that entry in the strip map and calculating it from

detector ID,
if it's the primary/secondary ASIC, and
the ASIC number
This is what StripMap::UpdateASICPolarities would now be doing.

fhagemann · 2026-03-09T23:37:07Z

Ok, the code in this PR still "only" allows reading FEEHits,
but it should be written in a way that is easy to extend to all other fields (ACSHits, H&S, DetectorLiveTime, ...)

There is a slight difference between v2.0-v2.1 and v2.2+, (v2.2+ added spacewire timing via spw_timecode), so I also split the v2 support into v2.0 and v2.2 support.

fhagemann · 2026-03-09T23:42:25Z

+      // Catch a bug in the HDF5 data (v1)
+      if (EventID == 0 && StripID == 0 && ADCs == 0) {
+        IsZeroDataBug = true;
+        if (g_Verbosity >= c_Error) cout<<m_XmlTag<<": ZERO-DATA-BUG: Found empty data set. Ignoring event."<<endl;
+        continue;
+      } else {
+        if (IsZeroDataBug == true) { // We are now out of the bug and need to recover
+          // Clear the strip hits - everything else gets overwritten later
+          while (Event->GetNStripHits() > 0) {
+            MStripHit* H = Event->GetStripHit(0);
+            delete H;
+            Event->RemoveStripHit(0);
+          }
+        }
+        IsZeroDataBug = false;
+      }


Is this something we would also expect in HDF v2?
The file structure has changed completely..

fhagemann · 2026-03-09T23:43:43Z

+    } else if (m_HDFStripHitVersion >= MHDFStripHitVersion::V2_0)  {
      Event->SetTI(TimeCode);
-    }
-
-    if (m_StripMap.HasReadOutID(StripID) == true) {
-      MStripHit* H = new MStripHit();
-      H->SetDetectorID(m_StripMap.GetDetectorID(StripID));
-      H->SetStripID(m_StripMap.GetStripNumber(StripID));
-      H->IsLowVoltageStrip(m_StripMap.IsLowVoltage(StripID));
-      H->SetADCUnits(ADCs);
-      H->SetTAC(TACs);
-
-
-      // Set boolean flags based on HitType and TimingType
-      H->IsGuardRing(HitType == 2);
-      if (H->IsGuardRing() == true) {
-        Event->SetGuardRingVeto(true);
-      }
-
-      H->IsNearestNeighbor(HitType == 1);
-      H->HasFastTiming(TimingType == 1);
-
-      // If the user does not want to include Nearest Neighbors in the data, then this is where we remove them
-      // NOTE: at some point we will want to remove this code and always include nearest neighbor data
-      if (m_IncludeNearestNeighbor == false && HitType == 1) {
-        delete H; // Clean up the memory we just allocated
-        // Increase counters
-        NStripHits = static_cast<unsigned int>(NumberOfHits);
-        StripHitIndex++;
-        continue;
+      if (m_HDFStripHitVersion >= MHDFStripHitVersion::V2_2) {
+        Event->SetCL(SPWTimeCode);


I'm right now setting the TimeCode (which is GSETimeCode if existent) to TI and the SPWTimeCode to CL.
Is this how it should be?

I don't know what it actually means.

Those are the times we currently have (Carolyn want to scrub it eventually):
//! TODO Scrub all clock/time variables for COSI SMEX
//! Set the Frame Counter of this event
void SetFC(unsigned int FC) { m_FC = FC; }
//! Return the Frame Counter of this event
unsigned int GetFC() const { return m_FC; }

//! set and get Unix clock time
void SetTI(unsigned long long TI) { m_TI = TI;}
unsigned long long GetTI() const { return m_TI;}

//! set and get clock tick
void SetCL(uint64_t CL) { m_CL = CL;}
uint64_t GetCL() const { return m_CL;}

//! Set and get the Modified Julian Date of this event
void SetMJD(double MJD) { m_MJD = MJD; }
double GetMJD() const { return m_MJD; }

//! Set and get the UTC time of this event
void SetTimeUTC(const MTime& TimeUTC) { m_EventTimeUTC = TimeUTC; }
MTime GetTimeUTC() const { return m_EventTimeUTC; }

I didn't touch that part of the code with my latest commit, but this goes along with #69.
We should eventually decide how/where we want to store that information (also for DEE output).

fhagemann · 2026-03-09T23:45:17Z

I tried parsing files in v1.1, v2.1 and v2.3, and I'm not getting any errors at least.
I asked @robin4730 (huge thanks!!) to look into if after the updates, the ROA files still look reasonable/identical.

zoglauer · 2026-03-17T18:28:20Z

Generally looks good, just a bunch of small things.

zoglauer · 2026-03-23T17:22:18Z

Put you name into the copyright notices!

fhagemann · 2026-03-23T17:28:34Z

Done (I also added my name to the copyright notices of MStripMap -- even though the changes in this PR might be minor, there were some bigger changes I made in #103)

zoglauer · 2026-03-23T17:49:21Z

Can you check if you initialize or clear everything in the Initialize function?
I don't see m_ASICPolarities there

fhagemann · 2026-03-23T18:07:56Z

That should also be done now. The only thing I don't initialize is m_HDFStripHitVersion (this is set at the beginning of OpenHDF5File, but I could also move it to Initialize to have everything initialized there)

And m_StripMap is also handled by m_StripMap::Open, but I could also explicitly clear it in Initialize if you want.

zoglauer · 2026-03-24T16:50:45Z

OK. Feel free to merge.

fhagemann added 2 commits February 25, 2026 23:24

Add support for loading version 2.2 of the HDF hit info

41a0f20

Fix event counter when reading HDF files

f703756

fhagemann linked an issue Feb 26, 2026 that may be closed by this pull request

MModuleLoaderMeasurementsHDF Number of Events is 0 #57

Closed

fhagemann force-pushed the hdf5v2 branch from f6dc635 to 618acfd Compare March 5, 2026 21:18

Read ASIC polarities from JSON config string

fbb28e1

fhagemann force-pushed the hdf5v2 branch from 618acfd to fbb28e1 Compare March 5, 2026 21:21

fhagemann added 2 commits March 5, 2026 13:30

Save LV/HV polarity as bool

ef88e7b

Apply ASIC polarities (if existent) to strip map

2e2b200

Bug fix

cf2093c

fhagemann force-pushed the hdf5v2 branch from 4fb4e5c to cf2093c Compare March 6, 2026 00:21

fhagemann changed the title ~~Add support for loading version 2.2 of the HDF hit info~~ Add support for loading version 2 of the HDF hit info Mar 9, 2026

Allow more HDF5 hit versions

75af370

fhagemann force-pushed the hdf5v2 branch from 3e9bc98 to 75af370 Compare March 9, 2026 23:39

fhagemann commented Mar 9, 2026

View reviewed changes

fhagemann changed the title ~~Add support for loading version 2 of the HDF hit info~~ ADD: Support for loading version 2 of the HDF hit info Mar 10, 2026

ckierans requested a review from zoglauer March 11, 2026 01:55

zoglauer reviewed Mar 17, 2026

View reviewed changes

Comment thread src/MStripMap.cxx Outdated

zoglauer reviewed Mar 17, 2026

View reviewed changes

Comment thread include/MModuleLoaderMeasurementsHDF.h Outdated

zoglauer reviewed Mar 17, 2026

View reviewed changes

Comment thread include/MModuleLoaderMeasurementsHDF.h Outdated

Comment thread src/MModuleLoaderMeasurementsHDF.cxx Outdated

Comment thread src/MModuleLoaderMeasurementsHDF.cxx Outdated

Comment thread src/MModuleLoaderMeasurementsHDF.cxx Outdated

fhagemann force-pushed the hdf5v2 branch from 0cc71a7 to cce5443 Compare March 20, 2026 21:48

fhagemann added the enhancement New feature or request label Mar 23, 2026

Implement feedback

74f5356

fhagemann force-pushed the hdf5v2 branch from cce5443 to 74f5356 Compare March 23, 2026 17:13

fhagemann requested a review from zoglauer March 23, 2026 17:14

Include my name in copyright notices

37bf5ef

fhagemann force-pushed the hdf5v2 branch from 8df312a to 37bf5ef Compare March 23, 2026 17:27

Update MModuleLoaderMeasurementsHDF::Initialize

ae68597

fhagemann force-pushed the hdf5v2 branch from 9ea956f to ae68597 Compare March 23, 2026 18:10

fhagemann merged commit 3621805 into cositools:develop/em Mar 24, 2026

fhagemann deleted the hdf5v2 branch March 24, 2026 20:03

ckierans mentioned this pull request Apr 14, 2026

MModuleLoaderMeasurementsHDF Number of Events is 0 #57

Closed

Conversation

fhagemann commented Feb 26, 2026

Uh oh!

zoglauer commented Feb 26, 2026

Uh oh!

fhagemann commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zoglauer commented Feb 26, 2026

Uh oh!

zoglauer commented Feb 26, 2026

Uh oh!

fhagemann commented Mar 5, 2026

Uh oh!

fhagemann commented Mar 5, 2026

Uh oh!

fhagemann commented Mar 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fhagemann Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fhagemann Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

zoglauer Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

fhagemann Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

fhagemann commented Mar 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zoglauer commented Mar 17, 2026

Uh oh!

zoglauer commented Mar 23, 2026

Uh oh!

fhagemann commented Mar 23, 2026

Uh oh!

zoglauer commented Mar 23, 2026

Uh oh!

fhagemann commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zoglauer commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fhagemann commented Feb 26, 2026 •

edited

Loading

fhagemann commented Mar 23, 2026 •

edited

Loading