Bug in offsets readout, unexpected fall back to 0 #39

tamasgal · 2021-07-13T15:13:03Z

I just discovered a serious bug in the data and offset readout while I was working with larger ROOT files. I think it's mostly affecting large datasets, but I am not sure.

With small sample file, we have no issues, everything is read out correctly and I can assemble the hits.

Just as a side note: a UnROOT.KM3NETDAQHit consists of 10 bytes and per event we have a 10 bytes header.

julia> f = ROOTFile("/home/tgal/Dev/UnROOT.jl/test/samples/km3net_online.root")ROOTFile with 10 entries and 54 streamers.
/home/tgal/Dev/UnROOT.jl/test/samples/km3net_online.root
├─ E
│  ├─ "Evt"
│  └─ ""
├─ KM3NET_TIMESLICE
│  ├─ "KM3NET_TIMESLICE"
│  └─ ""
├─ KM3NET_TIMESLICE_L0
│  ├─ "km3net_timeslice_L0"
│  └─ ""
├─ KM3NET_TIMESLICE_L1
│  ├─ "km3net_timeslice_L1"
│  └─ ""
├─ KM3NET_TIMESLICE_L2
│  ├─ "km3net_timeslice_L2"
│  └─ ""
├─ KM3NET_TIMESLICE_SN
│  ├─ "km3net_timeslice_SN"
│  └─ ""
├─ KM3NET_EVENT
│  ├─ "KM3NET_EVENT"
│  └─ ""
└─ KM3NET_SUMMARYSLICE
   ├─ "KM3NET_SUMMARYSLICE"
   └─ ""


julia> data, offsets = UnROOT.array(f, "KM3NET_EVENT/KM3NET_EVENT/snapshotHits"; raw=true)
(UInt8[0x40, 0x00, 0x03, 0xc6, 0x00, 0x09, 0x00, 0x00, 0x00, 0x60  …  0x30, 0x40, 0xa9, 0x7d, 0x0a, 0x3c, 0x21, 0xc9, 0x03, 0x17], Int32[0, 970, 2220, 3010])

julia> length(data)
3010

julia> length(offsets)
4

julia> offsets
4-element Vector{Int32}:
    0
  970
 2220
 3010

julia> sum(diff(offsets))
3010

julia> minimum(diff(offsets))
790

julia> UnROOT.splitup(data, offsets, UnROOT.KM3NETDAQHit, skipbytes=10)
3-element Vector{Vector{UnROOT.KM3NETDAQHit}}:
 [UnROOT.KM3NETDAQHit(806451572, 0x0a....

Now comes the fun part. Here is a large file with 10410 events. In UnROOT, the length of the offsets indicates 10411 events, one more than expected!

julia> f = ROOTFile("/data/irods/mc/atm_muon/KM3NeT_00000044/v6.1/trigger/mcv6.1.mupage_10G.sirene.jterbr00007209.2548.root")
ROOTFile with 17 entries and 76 streamers.
/data/irods/mc/atm_muon/KM3NeT_00000044/v6.1/trigger/mcv6.1.mupage_10G.sirene.jterbr00007209.2548.root
├─ KM3NET_SUMMARYSLICE
│  ├─ "KM3NET_SUMMARYSLICE"
│  └─ ""
├─ KM3NET_SUMMARYSLICE
│  ├─ "KM3NET_SUMMARYSLICE"
│  └─ ""
├─ E
│  ├─ "Evt"
│  └─ ""
├─ E
│  ├─ "Evt"
│  └─ ""
├─ KM3NET_TIMESLICE_SN
│  ├─ "km3net_timeslice_SN"
│  └─ ""
├─ KM3NET_TIMESLICE_SN
│  ├─ "km3net_timeslice_SN"
│  └─ ""
├─ KM3NET_TIMESLICE
│  ├─ "KM3NET_TIMESLICE"
│  └─ ""
├─ KM3NET_TIMESLICE_L0
│  ├─ "km3net_timeslice_L0"
│  └─ ""
├─ KM3NET_TIMESLICE_L1
│  ├─ "km3net_timeslice_L1"
│  └─ ""
├─ KM3NET_TIMESLICE_L2
│  ├─ "km3net_timeslice_L2"
│  └─ ""
└─ KM3NET_EVENT
   ├─ "KM3NET_EVENT"
   └─ ""


julia> data, offsets = UnROOT.array(f, "KM3NET_EVENT/KM3NET_EVENT/snapshotHits"; raw=true)
(UInt8[0x40, 0x00, 0x05, 0xec, 0x00, 0x09, 0x00, 0x00, 0x00, 0x97  …  0x30, 0x37, 0xb3, 0xec, 0x10, 0x29, 0x2f, 0xc9, 0x04, 0x06], Int32[0, 1520, 2000, 2460, 2920, 3740, 4840, 5610, 6120, 6960  …  2982630, 2983100, 2983870, 2984680, 2986230, 2986700, 2987130, 2987610, 2988370, 2989030])

julia> length(data)
7936390

julia> length(offsets)
10412

julia> sum(diff(offsets))  # this does not give 7936390 due to the jump, see below
2989030

julia> minimum(diff(offsets))
-4947360

julia> argmin(diff(offsets))
6553

julia> offsets[6550:6560]
11-element Vector{Int32}:
 4944770
 4945250
 4945910
 4947360
       0
    1080
    1720
    2330
    2810
    4270
    5370

I first compared the length of events (number of hits) at the beginning of the data and the end of the data and they all match! It's just this weird jump in the middle of the offsets.

Luckily the number of bytes in data matches the expected number of bytes when cross-checking with uproot (it's n_events * 10bytes + n_hits * 10bytes == 7936390`)

Now I have to understand why the offset starts to count from 0 and how to solve it. For now, UnROOT simply iterates over the diff(offsets) and it worked, but apparently it's not the full story ;)

The text was updated successfully, but these errors were encountered:

tamasgal · 2021-07-13T15:16:06Z

I just checked with km3io/uproot and there, the offset is continous:

>>> import km3io

>>> f = km3io.OnlineReader("mcv6.1.mupage_10G.si
... rene.jterbr00007209.2548.root")

>>> f.events.snapshot_hits._offsets
array([     0,    151,    198, ..., 783089, 783164, 783229])

tamasgal · 2021-07-13T15:24:51Z

Digging deeper, it's related to how we handle multiple entries in fBasketSeek.

For the small file, fBasketSeek only contains a single entry:

julia> f = ROOTFile("/home/tgal/Dev/UnROOT.jl/test/samples/km3net_online.root");

julia> f["KM3NET_EVENT/KM3NET_EVENT/snapshotHits"].fBasketSeek
┌ Warning: Can't automatically create LazyBranch for branch KM3NET_EVENT/KM3NET_EVENT/snapshotHits. Returning a branch object
└ @ UnROOT ~/.julia/packages/UnROOT/o16Nw/src/root.jl:96
10-element Vector{Int64}:
 1606921
       0
       0
       0
       0
       0
       0
       0
       0
       0

Whereas for the larger file we have two entries, in two different parts of the ROOT file:

julia> f["KM3NET_EVENT/KM3NET_EVENT/snapshotHits"].fBasketSeek┌ Warning: Can't automatically create LazyBranch for branch KM3NET_EVENT/KM3NET_EVENT/snapshotHits. Returning a branch object
└ @ UnROOT ~/.julia/packages/UnROOT/o16Nw/src/root.jl:96
10-element Vector{Int64}:
 324654693
 531785725
         0
         0
         0
         0
         0
         0
         0
         0

It's just the stitching of the two different parts, which is buggy, so that will be an easy fix, I think. Let's see... ;)

Moelf · 2021-07-13T15:31:46Z

https://github.com/tamasgal/UnROOT.jl/blob/74735c939e4127ee527270b21be9cd19626566f9/src/root.jl#L280

maybe this? But this should only push the last offset for each basket

tamasgal · 2021-07-13T15:33:36Z

I am going through the code. The problem is also the extra entry in the offsets, since it has an additional entry when the basket is changed.

Are you handy with basket option tweaking in ROOT? We need a small test file which uses multiple branches to test this 😕 I am currently looking how to set the basket size but have not found anything useful yet.

Moelf · 2021-07-13T15:35:35Z

if there are 10 elements in a basket, we do expect 11 offsets. Because you index the raw bytes for ith element with offset[i]:offset[i+1] right?

tamasgal · 2021-07-13T15:35:55Z

Yes that's true, but I have one more ;)

tamasgal · 2021-07-13T15:37:02Z

Just to clarify: the number of offset-elements is correct (11) for the small file with 10 entries. For the large file it's 10412 instead of 10411 (corresponding to 10410 elements).

Moelf · 2021-07-13T15:37:12Z

ah ok, I think the fix is when we dump out the entire branch's raw bytes, for each basket we need to add a "global offset".

and that's probably also where you want to remove the extra offset that is duplicate between two baskets

tamasgal · 2021-07-13T15:37:50Z

Yep, we just need to go ahead and count up. I am trying to figure out where to do it, the code has changed a lot 😆

Moelf · 2021-07-13T15:39:01Z

I think just this one function:
https://github.com/tamasgal/UnROOT.jl/blob/74735c939e4127ee527270b21be9cd19626566f9/src/root.jl#L223

if what I imagine happening is causing the issue.

To enable accessing baskets out of order, each baskets return data and local offsets (i.e always starts 0 and have one extra ending), thus indexing and reconstructing within a basket is locally trivial. So if we want a branch's raw bytes, we should trim and stitch them together.

tamasgal · 2021-07-13T15:44:06Z

Yes, it should be ripped apart (divide and conquer). For now I'll do a quick fix and later we can discuss how to proceed.

Moelf · 2021-07-13T15:51:29Z

wait, I think we can just fix it by something along the line of:

Seeks = branch.fBasketSeek;
    for i in eachindex(Seeks)
        if  i<length(Seeks) && Seeks[i+1]!=0
            pop!(offsets)
        end
        data, offset = readbasketseek(f, branch, Seeks[i])
        append!(datas, data)
        append!(offsets, offset)
    end

tamasgal · 2021-07-13T15:56:46Z

Oh well, look at my hack 😆

tamasgal added the bug Something isn't working label Jul 13, 2021

tamasgal mentioned this issue Jul 13, 2021

Preliminary fix of offset-readout #40

Merged

tamasgal closed this as completed in a937bc6 Jul 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug in offsets readout, unexpected fall back to 0 #39

Bug in offsets readout, unexpected fall back to 0 #39

tamasgal commented Jul 13, 2021

tamasgal commented Jul 13, 2021

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021 •

edited

tamasgal commented Jul 13, 2021

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021 •

edited

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021 •

edited

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021 •

edited

tamasgal commented Jul 13, 2021

Bug in offsets readout, unexpected fall back to 0 #39

Bug in offsets readout, unexpected fall back to 0 #39

Comments

tamasgal commented Jul 13, 2021

tamasgal commented Jul 13, 2021

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021 • edited

tamasgal commented Jul 13, 2021

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021 • edited

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021 • edited

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021 • edited

tamasgal commented Jul 13, 2021

Moelf commented Jul 13, 2021 •

edited

Moelf commented Jul 13, 2021 •

edited

Moelf commented Jul 13, 2021 •

edited

Moelf commented Jul 13, 2021 •

edited