Reading a file with Branches and Leafs #323

TheFibonacciEffect · 2024-03-23T11:33:40Z

I originally posted this on the julia discourse https://discourse.julialang.org/t/reading-root-file-with-branches/112014. However I believe this might actually be a bug in the package so I am posting my issue here as well.

Here is a link to the file I would like to read:
https://polybox.ethz.ch/index.php/s/1aNZCiDSO1f2Eig

Original Post on julia Discourse

This is what I had originaly posted on the julia discourse

I have a root file with a tree, branches and leaves:
When I try to read it like using UnROOT like this:

using UnROOT
f = ROOTFile("out_files/pienu00000-00.root")

tree = LazyTree(f,"sim")

I get a bounds error:

ERROR: BoundsError: attempt to access 7-element Vector{Any} at index [-1]
Stacktrace:
  [1] getindex(A::Vector{Any}, i1::Int64)
    @ Base ./essentials.jl:13
  [2] streamerfor(f::ROOTFile, branch::UnROOT.TBranchElement_10)
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/root.jl:160
  [3] UnROOT.JaggType(f::ROOTFile, branch::UnROOT.TBranchElement_10, leaf::UnROOT.TLeafElement)
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/utils.jl:64
  [4] auto_T_JaggT(f::ROOTFile, branch::UnROOT.TBranchElement_10; customstructs::Dict{String, Type})
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/root.jl:365
  [5] auto_T_JaggT
    @ ~/.julia/packages/UnROOT/PnXmk/src/root.jl:360 [inlined]
  [6] LazyBranch(f::ROOTFile, b::UnROOT.TBranchElement_10)
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:117
  [7] LazyBranch(f::ROOTFile, s::String)
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:134
  [8] LazyTree(f::ROOTFile, tree::UnROOT.TTree, treepath::String, branches::Vector{String}; sink::Type{LazyTree})
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:450
  [9] LazyTree
    @ ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:432 [inlined]
 [10] LazyTree(f::ROOTFile, s::String, branches::Vector{String}; kwargs::@Kwargs{})
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:393
 [11] LazyTree(f::ROOTFile, s::String, branches::Vector{String})
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:390
 [12] LazyTree(f::ROOTFile, s::String; kwargs::@Kwargs{})
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:461
 [13] LazyTree(f::ROOTFile, s::String)
    @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:460
 [14] top-level scope
    @ /popos/home/caspar/Documents-old/code/semesterprojekt-pioneer/analysis/analyze.jl:4

However reading root files that do not have branches works just fine.

This is how the file looks like within unroot:

julia> f
ROOTFile with 2 entries and 27 streamers.
out_files/pienu00000-00.root
├─ sim (TTree)
│  ├─ "info"
│  ├─ "init"
│  ├─ "track"
│  ├─ "decay"
│  ├─ "ghost"
│  ├─ "ghostface"
│  └─ "upstream"
└─ PIMCRunHeader (PIMCRunHeader)

In C I can read it by branch.leaf:

    events = (TTree*)fp->Get("sim");
    unsigned int nEvents = events -> GetEntriesFast();
    int pions = events -> GetEntries("ghost.pdgid==211");

So I also tried using
tree = LazyTree(f,"sim", "ghost.pdgid")

But I get another error:

ERROR: MethodError: no method matching LazyBranch(::ROOTFile, ::Missing)

Closest candidates are:
  LazyBranch(::ROOTFile, ::Union{UnROOT.TBranch, UnROOT.TBranchElement})
   @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:116
  LazyBranch(::ROOTFile, ::AbstractString)
   @ UnROOT ~/.julia/packages/UnROOT/PnXmk/src/iteration.jl:134

While it works fine when having a flat root file:

f = ROOTFile("data/data.root")
tree = LazyTree(f,"events","NJet")

Method where the error occurs

The error occurs here, from my understanding I have a branch with branch.fID < -1

streamerfor(f::ROOTFile, branch::TBranch) = missing
function streamerfor(f::ROOTFile, branch::TBranchElement)
    fID = branch.fID
    # According to ChatGPt: When fID is equal to -1, it means that the
    # TBranch object has not been registered yet in the TTree's list of
    # branches. This can happen, for example, when a TBranch object has been
    # created, but has not been added to a TTree with the TTree::Branch()
    # method.
    #
    # TODO: For now, we force it to be 0 in this case, until someone complains.
    if fID == -1
        fID = 0
    end
    next_streamer = streamerfor(f, branch.fClassName)
    if ismissing(next_streamer)
        return missing
    else
        return next_streamer.streamer.fElements.elements[fID + 1]  # one-based indexing in Julia
    end
end

Which is located at ~/.julia/dev/UnROOT/src/root.jl

The text was updated successfully, but these errors were encountered:

tamasgal · 2024-03-23T19:02:47Z

Just a very quick answer with a workaround, to allow you to proceed with your research. You can use the array() function to load the full data into memory. Since your file is small, laziness is not really needed, but your mileage may vary ;)

Anyways this is an example:

julia> f = ROOTFile("/Users/tamasgal/Downloads/pienu00002-02.root")
ROOTFile with 2 entries and 27 streamers.
/Users/tamasgal/Downloads/pienu00002-02.root
├─ sim (TTree)
│  ├─ "info"
│  ├─ "init"
│  ├─ "track"
│  ├─ "decay"
│  ├─ "ghost"
│  ├─ "ghostface"
│  └─ "upstream"
└─ PIMCRunHeader (PIMCRunHeader)


julia> UnROOT.array(f, "sim/ghost/ghost.pdgid")
1200-element ArraysOfArrays.VectorOfVectors{Int32, Vector{Int32}, Vector{Int32}, Vector{Tuple{}}}:
 Int32[-14, 14]
 Int32[14]
 Int32[-14, 14]
 Int32[14]
 0-element view(::Vector{Int32}, 7:6) with eltype Int32
 Int32[-14]
 Int32[211]

I will have a closer look at the file. LazyTree and LazyBranch are sometimes a bit hard to use with nested data.

tamasgal · 2024-03-23T19:24:44Z

OK, the first problem is that you are using the wrong path to the subbranch. I don't blame you, this is just something which is probably (read obviously) not very well documented ;)

If the structure is the following:

julia> f["sim/ghost"]
ghost
├─ ghost.fUniqueID
├─ ghost.fBits
├─ ghost.ghostID
├─ ghost.trackID
├─ ghost.stepID
├─ ghost.pdgid
├─ ghost.xpos
├─ ghost.ypos
├─ ghost.zpos
├─ ghost.time
├─ ghost.xmom
├─ ghost.ymom
└─ ghost.zmom

the path to the subbranch is "sim/ghost/ghost.pdgid". A LazyTree would then be

LazyBranch(f, "sim", ["sim/ghost/ghost.pdgid"])

or using a regex:

julia> LazyTree(f, "sim", [r"ghost/ghost\.(.*)" => s"\1"])

However (now comes the real problem), you will face the following error:

ERROR: MethodError: no method matching UInt32()

Closest candidates are:
  UInt32(::HTTP.WebSockets.Mask)
   @ HTTP ~/.julia/packages/HTTP/1EWL3/src/WebSockets.jl:62
  UInt32(::Char)
   @ Base char.jl:127
  UInt32(::Float64)
   @ Base float.jl:884
  ...

Stacktrace:
 [1] LazyBranch(f::ROOTFile, b::UnROOT.TBranchElement_10)
   @ UnROOT ~/Dev/UnROOT.jl/src/iteration.jl:124)

which is there because we never had a flat array and we use VectorOfVectors and do not cover the case for a flat array 🙈

I'll try to provide a fix asap.

tamasgal · 2024-03-23T19:34:46Z

Fix on the way in #324

tamasgal · 2024-03-23T19:45:21Z

UnROOT 0.10.26 will allow you

julia> t = LazyTree(f, "sim", [r"ghost/ghost\.(.*)" => s"\1"])
 Row │ ypos             ghostID          time             stepID           fUniqueID        xmom             pd ⋯
     │ SubArray{Float3  SubArray{Int32,  SubArray{Float3  SubArray{Int32,  SubArray{UInt32  SubArray{Float3  Su ⋯
─────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────
 1   │ [982.0, 6.83]    [110000, 11000   [3230.0, 36.4]   [0, 0]           [0, 0]           [-25.6, -29.5]   [- ⋯
 2   │ [2590.0]         [110000]         [29.7]           [0]              [0]              [21.9]           [1 ⋯
 3   │ [258.0, -2230.   [110000, 11000   [774.0, 48.0]    [0, 0]           [0, 0]           [-44.3, 22.1]    [- ⋯
 4   │ [-1850.0]        [110000]         [60.6]           [0]              [0]              [14.1]           [1 ⋯
 5   │ []               []               []               []               []               []               [] ⋯
 6   │ [512.0]          [110000]         [587.0]          [0]              [0]              [-11.5]          [- ⋯
 7   │ [-11.0]          [110000]         [40.8]           [0]              [0]              [-2.61]          [2 ⋯
 8   │ [2890.0]         [110000]         [28.7]           [0]              [0]              [-8.56]          [1 ⋯
 9   │ []               []               []               []               []               []               [] ⋯
 10  │ [28.1]           [110000]         [41.0]           [0]              [0]              [-0.453]         [2 ⋯
 11  │ [2660.0]         [110000]         [799.0]          [0]              [0]              [-4.84]          [- ⋯
 12  │ [12.0]           [110000]         [41.0]           [0]              [0]              [2.28]           [2 ⋯
 13  │ [33.0]           [110000]         [41.8]           [0]              [0]              [-4.04]          [2 ⋯
 14  │ [-0.84]          [110000]         [40.5]           [0]              [0]              [-5.09]          [2 ⋯
 15  │ [664.0]          [110000]         [311.0]          [0]              [0]              [-3.08]          [1 ⋯
 16  │ []               []               []               []               []               []               [] ⋯
 17  │ [-1220.0]        [110000]         [640.0]          [0]              [0]              [7.4]            [1 ⋯
 18  │ [-2290.0, -269   [110000, 11000   [1410.0, 1410.   [0, 0, 0]        [0, 0, 0]        [0.0214, 0.01,   [2 ⋯
 19  │ [906.0]          [110000]         [8140.0]         [0]              [0]              [41.8]           [- ⋯
 20  │ [3430.0]         [110000]         [4960.0]         [0]              [0]              [-13.3]          [- ⋯
 21  │ []               []               []               []               []               []               [] ⋯
 22  │ [5.71]           [110000]         [41.0]           [0]              [0]              [6.6]            [2 ⋯
 23  │ []               []               []               []               []               []               [] ⋯
 24  │ [1330.0]         [110000]         [1250.0]         [0]              [0]              [-24.4]          [1 ⋯
 25  │ [-1050.0]        [110000]         [23.5]           [0]              [0]              [10.1]           [1 ⋯
 26  │ [-280.0, -2400   [110000, 11000   [1150.0, 39.1]   [0, 0]           [0, 0]           [0.412, -13.6]   [2 ⋯
 27  │ []               []               []               []               []               []               [] ⋯
 28  │ [-12.8]          [110000]         [1150.0]         [0]              [0]              [10.7]           [- ⋯
 29  │ [2770.0, 2080.   [110000, 11000   [1610.0, 20.5]   [0, 0]           [0, 0]           [20.8, -15.4]    [- ⋯
 30  │ [262.0, -866.0   [110000, 11000   [36.3, 35.7]     [0, 0]           [0, 0]           [-17.5, 5.21]    [1 ⋯
 31  │ []               []               []               []               []               []               [] ⋯
  ⋮  │        ⋮                ⋮                ⋮                ⋮                ⋮                ⋮            ⋱
                                                                                  7 columns and 1169 rows omitted

TheFibonacciEffect · 2024-03-23T20:40:30Z

Thank you so much for the timely response! Even on a saturday, I appreciate that a lot.

tamasgal · 2024-03-23T21:09:13Z

Science never sleeps ;)

Let me know if you have other issues. The next weeks will be very busy for me though 🙈

Moelf · 2024-03-25T22:15:55Z

huh, I didn't realize but the test file is kinda of large:

 test/samples/issue323.root | Bin 0 -> 5890205 bytes

is it possible to reduce it

TheFibonacciEffect · 2024-03-31T10:33:18Z

Sure there you go:
https://polybox.ethz.ch/index.php/s/PcYbFyA5WO2Txvt

Moelf · 2024-04-02T14:52:38Z

@TheFibonacciEffect that file seems to be empty:

julia> LazyTree("./test/samples/issue323_small.root", "sim", [r"ghost/ghost\.(.*)"])
 Row │ ghost_ymom       ghost_fBits      ghost_pdgid      ghost_trackID    ghost_ghostID    ghost_time       ghost_xmom       ghost_xpos       ghost_zmom      ⋯
     │ SubArray{Float3  SubArray{UInt32  SubArray{Int32,  SubArray{Int32,  SubArray{Int32,  SubArray{Float3  SubArray{Float3  SubArray{Float3  SubArray{Float3 ⋯
─────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                                                               4 columns omitted

TheFibonacciEffect · 2024-04-03T08:17:16Z

@Moelf
Sorry about that, this is also a way to solve large filesizes I guess 😅
Here this file should work: https://polybox.ethz.ch/index.php/s/i6KKEMt8A7UtVM0

julia> LazyTree("./issue323_small.root", "sim", [r"ghost/ghost\.(.*)"])
 Row │ ghost_ymom                   ghost_fBits                  ghost_pdgid                  ghost_trackID                ghost_ghostID                ghost_ ⋯
     │ SubArray{Float3              SubArray{UInt32              SubArray{Int32,              SubArray{Int32,              SubArray{Int32,              SubArr ⋯
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 1   │ [-3.08, -3.08, -3.08, -3.08  [50331648, 50331648, 503316  [-13, -13, -13, -13, -13, -  [3, 3, 3, 3, 3, 3, 3, 3, 3,  [110000, 110001, 110002, 11  [21.7, ⋯
 2   │ [-2.92, -2.92, -2.92, -2.92  [50331648, 50331648, 503316  [211, 211, 211, 211, 211, 2  [1, 1, 1, 1, 1, 1, 1, 1, 1,  [110000, 110001, 110002, 11  [22.8, ⋯
 3   │ [0.417, 0.417, 0.417, 0.417  [50331648, 50331648, 503316  [211, 211, 211, 211, 211, 2  [1, 1, 1, 1, 1, 1, 1, 1, 1,  [110000, 110001, 110002, 11  [21.8, ⋯
 4   │ [-10.8, -10.8, -10.8, -10.8  [50331648, 50331648, 503316  [-14, -14, -14, -14, -14, -  [4, 4, 4, 4, 4, 4, 4, 4, 4,  [110000, 110001, 110002, 11  [10100 ⋯
 5   │ [-21.3, -21.3, -21.3, -21.3  [50331648, 50331648, 503316  [14, 14, 14, 14, 14, 14, 14  [2, 2, 2, 2, 2, 2, 2, 2, 2,  [110000, 110001, 110002, 11  [13.2, ⋯
 6   │ [-15.2, -15.2, -15.2, -15.2  [50331648, 50331648, 503316  [211, 211, 211, 211, 211, 2  [1, 1, 1, 1, 1, 1, 1, 1, 1,  [110000, 110001, 110002, 11  [22.3, ⋯
 7   │ [2.34, 2.34, 2.34, 2.34, 2.  [50331648, 50331648, 503316  [-14, -14, -14, -14, -14, -  [4, 4, 4, 4, 4, 4, 4, 4, 4,  [110000, 110001, 110002, 11  [1140. ⋯
 8   │ [28.6, 28.6, 28.6, 28.6, 28  [50331648, 50331648, 503316  [14, 14, 14, 14, 14, 14, 14  [2, 2, 2, 2, 2, 2, 2, 2, 2,  [110000, 110001, 110002, 11  [20.3, ⋯
 9   │ [9.87, 9.87, 9.87, 9.87, 9.  [50331648, 50331648, 503316  [211, 211, 211, 211, 211, 2  [1, 1, 1, 1, 1, 1, 1, 1, 1,  [110000, 110001, 110002, 11  [22.1, ⋯
 10  │ [10.3, 10.3, 10.3, 10.3, 10  [50331648, 50331648, 503316  [14, 14, 14, 14, 14, 14, 14  [2, 2, 2, 2, 2, 2, 2, 2, 2,  [110000, 110001, 110002, 11  [10.3, ⋯
 11  │ [-24.1, -24.1, -24.1, -24.1  [50331648, 50331648, 503316  [14, 14, 14, 14, 14, 14, 14  [2, 2, 2, 2, 2, 2, 2, 2, 2,  [110000, 110001, 110002, 11  [15.0, ⋯
  ⋮  │              ⋮                            ⋮                            ⋮                            ⋮                            ⋮                      ⋱                                                                                                                               8 columns and 113 rows omitted

Thanks again for helping me out with the issue : )

tamasgal · 2024-04-05T20:38:06Z

Oh shoot, that's my bad, I misread the file size for whatever reason (thought it's 500kB) 🙈

tamasgal added bug Something isn't working help wanted Extra attention is needed labels Mar 23, 2024

tamasgal mentioned this issue Mar 23, 2024

Hotfix for flat vector of vectors #324

Merged

tamasgal closed this as completed in #324 Mar 23, 2024

tamasgal removed the help wanted Extra attention is needed label Mar 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reading a file with Branches and Leafs #323

Reading a file with Branches and Leafs #323

TheFibonacciEffect commented Mar 23, 2024 •

edited

tamasgal commented Mar 23, 2024

tamasgal commented Mar 23, 2024

tamasgal commented Mar 23, 2024

tamasgal commented Mar 23, 2024

TheFibonacciEffect commented Mar 23, 2024 •

edited

tamasgal commented Mar 23, 2024

Moelf commented Mar 25, 2024

TheFibonacciEffect commented Mar 31, 2024

Moelf commented Apr 2, 2024

TheFibonacciEffect commented Apr 3, 2024

tamasgal commented Apr 5, 2024

Reading a file with Branches and Leafs #323

Reading a file with Branches and Leafs #323

Comments

TheFibonacciEffect commented Mar 23, 2024 • edited

Original Post on julia Discourse

Method where the error occurs

tamasgal commented Mar 23, 2024

tamasgal commented Mar 23, 2024

tamasgal commented Mar 23, 2024

tamasgal commented Mar 23, 2024

TheFibonacciEffect commented Mar 23, 2024 • edited

tamasgal commented Mar 23, 2024

Moelf commented Mar 25, 2024

TheFibonacciEffect commented Mar 31, 2024

Moelf commented Apr 2, 2024

TheFibonacciEffect commented Apr 3, 2024

tamasgal commented Apr 5, 2024

TheFibonacciEffect commented Mar 23, 2024 •

edited

TheFibonacciEffect commented Mar 23, 2024 •

edited