xrootd read support #150
xrootd read support #150
Conversation
|
if we don't want to change local read into some kind of buffered read, we would have to put some transparent smartness into waiting to see if we can make our own buffer easily: useful resources regarding design: |
|
maybe it's not that bad... julia> const r = @time ROOTFile("root://eospublic.cern.ch//eos/root-eos/cms_opendata_2012_nanoaod/Run2012B_DoubleMuParked.root")
18.410567 seconds (8.31 M allocations: 450.700 MiB, 1.19% gc time, 18.37% compilation time)
ROOTFile with 1 entry and 19 streamers.
root://eospublic.cern.ch//eos/root-eos/cms_opendata_2012_nanoaod/Run2012B_DoubleMuParked.root
└─ Events (TTree)
├─ "run"
├─ "luminosityBlock"
├─ "event"
├─ "⋮"
├─ "Electron_dxyErr"
├─ "Electron_dz"
└─ "Electron_dzErr"
julia> const t = LazyTree(r, "Events", r"Muon_*");
julia> @time let i = 0
s = 0.0
for evt in t
s += sum(evt.Muon_pt)*evt.nMuon
i+=1
i > 1000 && break
end
end
2.167068 seconds (1.52 M allocations: 78.707 MiB, 1.31% gc time, 23.45% compilation time) |
julia> @time ROOTFile("root://eospublic.cern.ch//eos/root-eos/cms_opendata_2012_nanoaod/Run2012B_DoubleMuParked.root")
5.010594 seconds (4.65 k allocations: 492.016 KiB)
ROOTFile with 1 entry and 19 streamers.
root://eospublic.cern.ch//eos/root-eos/cms_opendata_2012_nanoaod/Run2012B_DoubleMuParked.root
└─ Events (TTree)
├─ "run"
├─ "luminosityBlock"
├─ "event"
├─ "⋮"
├─ "Electron_dxyErr"
├─ "Electron_dz"
└─ "Electron_dzErr"with some hacks, down to 5s, still pretty slow, half of the time it's parsing |
|
I can have a look later! What is the access time with uproot? |
|
It could be that there are too many seeks, which have more overhead via a network connection. This means that local buffering might be needed… |
|
Here is the distribution of the streamer locations based on our test-file samples. Not much we can predict there 😕 julia> using UnROOT
julia> using UnicodePlots
julia> rootfiles = map(ROOTFile, filter(f->endswith(f, ".root"), readdir("test/samples/"; join=true)));
julia> histogram([r.header.fSeekInfo / filesize(r.filename) for r ∈ rootfiles])
┌ ┐
[0.0, 0.2) ┤███████████████████▍ 7
[0.2, 0.4) ┤████████████████████████████████████ 13
[0.4, 0.6) ┤ 0
[0.6, 0.8) ┤████████▍ 3
[0.8, 1.0) ┤██████████████████████▎ 8
└ ┘
Frequency |
|
Here is the distribution of streamer locations for all the ROOT files in the scikit-hep testdata repo: The positions are likely at the end of the files as we assumed and the outliers are probably due to the small filesizes. Here is a crappy scatterplot to demonstrate it, kind of ;) |
|
Ok I'm gonna read the last 5KB of the file as a heuristic. Thanks for the investigation! |
julia> r = @time ROOTFile("root://eospublic.cern.ch//eos/root-eos/cms_opendata_2012_nanoaod/Run2012B_DoubleMuParked.root");
2.114943 seconds (4.65 k allocations: 495.797 KiB)
julia> r = @time ROOTFile("root://eospublic.cern.ch//eos/root-eos/cms_opendata_2012_nanoaod/Run2012B_DoubleMuParked.root");
1.995430 seconds (4.65 k allocations: 495.797 KiB)much better now, thanks, we're similar to uproot speed: In [1]: import uproot as up
In [2]: %%time
...: up.open(
...: "root://eospublic.cern.ch//eos/root-eos/cms_opendata_2012_nanoaod/Run2012B_DoubleMuParked.root"
...:
...: )["Events"].keys()
CPU times: user 106 ms, sys: 4.58 ms, total: 110 ms
Wall time: 1.9 s
Out[2]:
['run',
'luminosityBlock', |
|
Ah nice, on par with |
|
I have reduced the number of IO s in a basket read from four to just one. Also I now recognize the usefulness of IO Context, so I'm adding a OffsetBuffer to mimic that....... |
|
I already planned to do the I/O completely with our own custom type so that we can do much more caching, but it gets complicated quite quickly. We basically have to do coordinate transformations and juggle around the cached data ;) Anyways, I think that needs to be done at some point if we want to have good support for network protocols. The filesystem already does a good job in caching file access but of course, network streams are different. |
|
I don't think we need to complicate things much. The read is semi-random (we can of course always assume the reader is doing a loop). But, I think it's reasonable to ask users to do chunked iteration over the network, it's probably their workflow in The Go library is thread-safe, and they don't have |
Codecov Report
@@ Coverage Diff @@
## master #150 +/- ##
==========================================
- Coverage 91.32% 90.66% -0.66%
==========================================
Files 11 11
Lines 1522 1554 +32
==========================================
+ Hits 1390 1409 +19
- Misses 132 145 +13
Continue to review full report at Codecov.
|
|
@tamasgal if you ever dig up your pure-Julia implementation, here's a piece of useful code:
recently, xrootd servers start to support pure HTTP(s) protocol, and there, they support token-based authentication (https://wlcg-authz-wg.github.io/wlcg-authz-docs/token-based-authorization/configuration/xrootd/), essentially, in that python script, you simply load your token content into the header of a HTTP request If you ever re-start your adventure, I suggest implement the HTTP one, should be much easier than TCP. per Jim:
|
|
more information...........it turns out HTTP is trivial, in the sense that:
|
like this.... network is inclusive on both ends, watch out |
julia> using Revise, UnROOT
[ Info: Precompiling UnROOT [3cd96dde-e98d-4713-81e9-a4a1b0235ce9]
julia> r = @time ROOTFile("https://scikit-hep.org/uproot3/examples/Zmumu.root")
5.213773 seconds (11.84 M allocations: 646.995 MiB, 3.41% gc time, 98.53% compilation time)
julia> r = @time ROOTFile("https://scikit-hep.org/uproot3/examples/Zmumu.root")
0.034877 seconds (5.13 k allocations: 533.125 KiB)
ROOTFile with 1 entry and 18 streamers.
https://scikit-hep.org/uproot3/examples/Zmumu.root
└─ events (TTree)
├─ "Type"
├─ "Run"
├─ "Event"
├─ "⋮"
├─ "phi2"
├─ "Q2"
└─ "M" |
|
here are some performance numbers : (note the difference between Local filejulia> const t2 = LazyTree("/home/akako/Downloads/Run2012B_DoubleMuParked.root", "Events");
Remote file:julia> const t = LazyTree("https://jiling.web.cern.ch/jiling/public/Run2012B_DoubleMuParked.root", "Events");This PRjulia> function g(t)
res = 0.0
for rang in Iterators.partition(1:10^7, 10^6)
res += sum(t.nMuon[rang])
end
res
end
g (generic function with 1 method)
julia> @time g(t)
5.442506 seconds (5.89 M allocations: 402.796 MiB, 5.91% gc time, 20.78% compilation time)
2.3621071e7
julia> @time g(t)
2.517748 seconds (48.84 k allocations: 110.399 MiB, 0.25% gc time)
2.3621071e7
julia> function f(t)
res = zeros(Threads.nthreads())
Threads.@threads for rang in collect(Iterators.partition(1:10^7, 10^6))
res[Threads.threadid()] += sum(t.nMuon[rang])
end
sum(res)
end
f (generic function with 1 method)
julia> @time f(t)
1.093996 seconds (49.68 k allocations: 110.422 MiB, 0.77% gc time)
2.3621071e7
julia> @time f(t)
1.200791 seconds (49.95 k allocations: 110.156 MiB, 0.60% gc time)
2.3621071e7 |
Compare to uproot branch readingIn [2]: t = up.open("https://jiling.web.cern.ch/jiling/public/Run2012B_DoubleMuParked.root")["Events"]
In [22]: b = t["nMuon"]
In [24]: %time tt = b.array(entry_start=0, entry_stop=10**5);
CPU times: user 25 ms, sys: 153 µs, total: 25.2 ms
Wall time: 620 ms
In [25]: %time tt = b.array(entry_start=0, entry_stop=10**5);
CPU times: user 18.4 ms, sys: 3.95 ms, total: 22.3 ms
Wall time: 404 msjulia> const t = LazyTree("https://jiling.web.cern.ch/jiling/public/Run2012B_DoubleMuParked.root", "Events");
julia> @time t.nMuon[1:10^5];
1.330934 seconds (996 allocations: 1.657 MiB)
julia> @time t.nMuon[1:10^5];
0.290066 seconds (718 allocations: 1.648 MiB)tree readingIn [30]: %time t.arrays(entry_start=0, entry_stop=10**4);
CPU times: user 562 ms, sys: 101 ms, total: 664 ms
Wall time: 7.01 s
Out[30]: <Array [{run: 194108, ... MET_phi: -1.43}] type='10000 * {"run": int32, "luminos...'>
In [31]: %time t.arrays(entry_start=0, entry_stop=10**4);
CPU times: user 451 ms, sys: 57.6 ms, total: 509 ms
Wall time: 6.77 s
Out[31]: <Array [{run: 194108, ... MET_phi: -1.43}] type='10000 * {"run": int32, "luminos...'>julia> @time t[1:10^4];
6.305692 seconds (5.84 M allocations: 388.173 MiB, 1.37% gc time, 35.34% compilation time)
julia> @time t[1:10^4];
1.696461 seconds (20.91 k allocations: 71.446 MiB, 5.39% gc time)
julia> @time t[1:10^4];
1.600297 seconds (18.96 k allocations: 71.124 MiB, 4.95% gc time) |
|
Very nice work! I tried out the PR... julia> r = ROOTFile("https://jiling.web.cern.ch/jiling/public/Run2012BC_DoubleMuParked_Muons.root");
julia> const t = LazyTree(r, "Events");
julia> @time display(t.nMuon)
61540413-element LazyBranch{UInt32, UnROOT.Nojagg, Vector{UInt32}}:
0x00000002
0x00000002
0x00000001
0x00000004
0x00000004
0x00000003
0x00000002
⋮
0x00000003
0x00000002
0x00000002
0x00000004
0x00000003
0x00000002
1.475081 seconds (2.21 k allocations: 4.208 MiB)And it takes 1.5s every time because only the last basket is cached. So the printout has to fetch the first and last basket each time 😛 . Not sure why it's so slow...maybe my connection? Next, I tried eos... credential issue? |
yeah, I mean, I don't see a way around it. Right now we have: julia> print(tree.nMuon)
61540413-element LazyBranch{UInt32, UnROOT.Nojagg, Vector{UInt32}}:
File: /home/akako/Downloads/Run2012BC_DoubleMuParked_Muons.root
Branch: nMuon
Description: nMuon/i
NumEntry: 61540413
Entry Type: UInt32but the REPL display (i.e. the
fixed now, turns out that file has a 20MB gap between some header and next header.... I changed the logic to read 3 times now instead of trying to read the whole "tail" in one go |
|
Wasn't suggesting to change anything but iirc there was some discussion somewhere about LRU vs MRU cache, and I think network delay strengthens the difference. A really weird approach (actually, is it?) might be to have a simultaneous LRU and MRU cache. Then people can printout/loop through/do the equivalent of |
|
there are two quick fixes:
|
|
Probably not worth it. I'm also a huge fan of seeing the first few and last few elements. |
|
technically we can backport some improvement......basically everything except xrootd can be back ported, let me know what you guys think.... |
|
merging this but let's not tag a new version yet |
* xrootd * bump Julia requirement to >=1.6
* xrootd * bump Julia requirement to >=1.6
I will eventually combine the Go pkg into the wrapper pkg...maybe even move both into UnROOT.
works but takes a millennium to read anything because everyread(io, ::Type{T})is a network(?) roundtrip