You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
uproot much slower than RDataFrame for simple numerical branches on NFS
#1629
I'm reading ROOT files from an NFS mount in an ATLAS data analysis. Files contain a TTree (CollectionTree) with many branches of various types. In this benchmark I'm reading a single scalar float branch (HGamEventInfoAuxDyn.m_ll) and filling a 1D histogram.
Is it expected that RDataframe is so faster?
Benchmark
Note that in ROOT, I am not using ROOT.ROOT.EnableImplicitMT().
[1] UserWarning: Sending large graph of size 449.19 MiB. This may cause some slowdown. Consider loading the data with Dask directly or using futures or delayed objects to embed the data into the graph without repetition.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I'm reading ROOT files from an NFS mount in an ATLAS data analysis. Files contain a TTree (
CollectionTree) with many branches of various types. In this benchmark I'm reading a single scalar float branch (HGamEventInfoAuxDyn.m_ll) and filling a 1D histogram.Is it expected that RDataframe is so faster?
Benchmark
Note that in ROOT, I am not using
ROOT.ROOT.EnableImplicitMT().Single file 150k entries
uproot.openuproot.dask+ synchronous scheduler,steps_per_file=16uproot.dask+ threads scheduler,steps_per_file=16uproot.dask+ threads scheduler,steps_per_file=32RDataFrame20 files 1.7M events
uproot.iterateuproot.dask+ threads scheduler,steps_per_file=16uproot.dask+ threads scheduler,steps_per_file=16, open_files=Falseuproot.dask+ threads scheduler,steps_per_file=16+ LocalCluster(n_workers=20)uproot.dask,steps_per_file=16+ LocalCluster(n_workers=20)RDataFrameRDataFrame+ EnableImplicitMT[1]
UserWarning: Sending large graph of size 449.19 MiB. This may cause some slowdown. Consider loading the data with Dask directly or using futures or delayed objects to embed the data into the graph without repetition.Setup
uproot.open
Dask:
RDataFrame
RDataFrame 20 files
uproot.iterate 20 files
uproot.dask 20 files
Environment
x86_64-el9-gcc14-opt)All reactions