Writing files with Uproot 4

[jpivarski]

This thread is about writing files with Uproot 4. I've included here everyone who has asked about this, talked about this, or provided feedback on it in Uproot 3. You can "unwatch" this thread if you're not interested.

@reikdas @pcanal @sbinet @oshadura @nsmith- @goi42 @eduardo-rodrigues @chrisburr @hassec @dnadeau-lanl @lindsaybolz @atasattari @lkorley @bfis @masonproffitt @jonr667 @xaratustrah @HDembinski @kratsg @adamdddave @bixel @andrzejnovak @BrutalPosedon @VukanJ @marinang @douglasdavis @kgizdov @matthewfeickert @beojan @Justin-Tan @burneyy @anerokhi @lukasheinrich @henryiii

PR #320 was a first step—it only adds one test file:

https://github.com/scikit-hep/uproot4/blob/main/tests/test_0320-start-working-on-ROOT-writing.py

which doesn't actually use Uproot; it reads and writes to a file with basic Python commands (struct.pack and struct.unpack). It moves the root directory of the file from one seek position to another such that the file contents are effectively the same, but it has to update pointers and the list of free segments to do so (for three different files that started with different numbers of free segments). To be thorough, I put "X" chars over the original data, to be sure it can't be read anymore.

Then the same file can not only be read by ROOT, it can be updated by ROOT and the output of that second operation is still valid. Thus, it's a full cycle: the original file was created by ROOT → updated by Python code → read back and updated again by ROOT → read back again by ROOT and Uproot. The original contents and the newly added contents (a TObjString) are both readable in the final state.

My conclusion from this is that it will be possible to have an "update" function, not just "recreate", despite what I've said in the past about that being out of scope (e.g. scikit-hep/uproot3#381, scikit-hep/uproot3#460, scikit-hep/uproot3#530). I haven't written any of that new code, just determined that it will be possible.

I have all of @reikdas's Uproot 3 work to draw on, and I just came across this:

https://github.com/root-project/root/tree/master/io/doc/TFile

Many of the fundamental serialization types of ROOT are specified, including the free segments used above. I've gone through and verified these against what we use in Uproot—this is a fantastic resource I wish I'd known about before!

[jpivarski]

Just an update: we can now create ROOT files with nested subdirectories (and no other kinds of objects!).

Most importantly, ROOT can open these files and put objects in the subdirectories Uproot made, with Uproot → ROOT → Uproot round trips and no corruption. I'm fairly confident that the seek positions and map of free segments are being updated correctly because even adding a subdirectory requires significant movement of data as directory key-lists outgrow their original allocation. Also, ROOT has thankfully been verbose about any mistakes, even recoverable ones.

A sample test from PR #322:

https://github.com/scikit-hep/uproot4/blob/2b518fa7f01fa94f539710bfc73fead455ea2877/tests/test_0322-writablefile-infrastructure.py#L65-L107

Next, I'll work on defining streamers, which will make it possible to store actual objects (starting with TObjString for simplicity).

[jpivarski]

As of PR #329, updating files can go full circle: files made by ROOT can be updated by Uproot, the result of which can be updated by ROOT, etc.

Here, ROOT made the initial file containing a string, Uproot added a subdirectory, ROOT put another string inside that subdirectory, and Uproot-reading shows that everything's there in the end.

https://github.com/scikit-hep/uproot4/blob/95bd056afebf33a4b6cf8f01f023ee53d85ed87f/tests/test_0329-update-existing-root-files.py#L17-L43

[liaolibo-jay]

Hello, dear jpivarski,
I tried to input some information from NumPy to ROOT by uproot, but it always failed.
And in this file, uproot4/tests/test_0329-update-existing-root-files.py, has a module named uproot.writing, but I can't find this module anywhere. Do you have any complete tutorials?
thanks a lot.

[jpivarski]

I don't think there has been a deployed release with uproot.writing in it yet, but also check this thread for progress: as I've said here, you can't write new objects in a ROOT file yet. Next, I'll be working on TObjString, then histograms after that, then TTrees after that (the case that you need).

[henryiii]

And you can always use uproot3 in the meantime for the writing step.

[liaolibo-jay]

ok, thanks a lot

[agoose77]

This thread is about writing files with Uproot 4. I've included here everyone who has asked about this, talked about this, or provided feedback on it in Uproot 3. You can "unwatch" this thread if you're not interested.

@reikdas @pcanal @sbinet @oshadura @nsmith- @goi42 @eduardo-rodrigues @chrisburr @hassec @dnadeau-lanl @lindsaybolz @atasattari @lkorley @bfis @masonproffitt @jonr667 @xaratustrah @HDembinski @kratsg @adamdddave @bixel @andrzejnovak @BrutalPosedon @VukanJ @marinang @douglasdavis @kgizdov @matthewfeickert @beojan @Justin-Tan @burneyy @anerokhi @lukasheinrich @henryiii

PR #320 was a first step—it only adds one test file:

https://github.com/scikit-hep/uproot4/blob/main/tests/test_0320-start-working-on-ROOT-writing.py

which doesn't actually use Uproot; it reads and writes to a file with basic Python commands (struct.pack and struct.unpack). It moves the root directory of the file from one seek position to another such that the file contents are effectively the same, but it has to update pointers and the list of free segments to do so (for three different files that started with different numbers of free segments). To be thorough, I put "X" chars over the original data, to be sure it can't be read anymore.

Then the same file can not only be read by ROOT, it can be updated by ROOT and the output of that second operation is still valid. Thus, it's a full cycle: the original file was created by ROOT → updated by Python code → read back and updated again by ROOT → read back again by ROOT and Uproot. The original contents and the newly added contents (a TObjString) are both readable in the final state.

My conclusion from this is that it will be possible to have an "update" function, not just "recreate", despite what I've said in the past about that being out of scope (e.g. scikit-hep/uproot3#381, scikit-hep/uproot3#460, scikit-hep/uproot3#530). I haven't written any of that new code, just determined that it will be possible.

I have all of @reikdas's Uproot 3 work to draw on, and I just came across this:

https://github.com/root-project/root/tree/master/io/doc/TFile

Many of the fundamental serialization types of ROOT are specified, including the free segments used above. I've gone through and verified these against what we use in Uproot—this is a fantastic resource I wish I'd known about before!

It's very interesting to see such low level interaction with the root format. Thanks for sharing this!

[jpivarski]

Just to keep this thread updated, I managed to get back to Uproot file-writing and added TStreamerInfo handling in PR #341. It is still the case that you can't write any objects to the file (other than subdirectories of subdirectories), but TStreamerInfo is a prerequisite for objects because a type's serialization must be described for it to be readable.

As usual, everything is being tested in "volleys"—a file is updated by ROOT, then by Uproot, then by ROOT again, etc. to ensure that Uproot is giving ROOT data that it can not only understand, but manipulate.

[jpivarski]

Today's update: Uproot can now copy objects from one ROOT file into another, taking all the TStreamerInfos that would be necessary to interpret those objects.

https://github.com/scikit-hep/uproot4/blob/191274763de98ed9cf9ac1eb9474be53b087d1b5/tests/test_0345-bulk-copy-method.py#L34-L36

This was the first step because it's a literal byte-for-byte copy of the original data, without interpretation (other than getting its classname to make a dependency tree of all the TStreamerInfos, but the classname is in the TKey, so the object data do not need to be interpreted). In ROOT-speak, this is a "fast copy," because it doesn't require any decompression and recompression.

The

destination_file["name"] = object_to_write

syntax of Uproot 3 will be reimplemented, but it requires knowledge of how to serialize the object_to_write. The first of these will be TObjString, for simplicity, followed by histograms. (I'm heavily relying on @reikdas's code for that!)

The copy_from method will remain a better way to copy many objects, though, because it knows the full set of objects you want to copy. Its second argument is a name filter (like the one you can pass to ReadOnlyDirectory.keys), so you can pass a list of paths, paths with wildcards, regular expressions, or a function that returns True or False for each name. Therefore, it knows the full set of TStreamerInfos to copy, the full set of TDirectories to make, and how many items will go in each, so that the TDirectories are allocated with a large enough size to not need to be reallocated. Also, the data-reading is collected into a single HTTP/XRootD request, if the server supports multi-part GET or vector_read.

We'll probably want something similar for the categorical-axis histograms in Coffea and Boost-Histogram/Hist, so that they can be translated into a directory of conventional histograms without reallocating the directory as it grows in size.

[archiron]

Hi, I tried those lines to make a copy but tell if I'm wrong :
it only copy the name of the directory ?
I tried with : dest.copy_from(source, "DQMData/Run 1/EgammaV") to have a copy of EgammaV but there is only the name of the directory which was created.

[jpivarski]

It should be recursively copying all of the objects. I'm on the fence about whether to support copy_from or not: it was an early stepping-stone to writing data because it doesn't require Uproot to understand the contents of what it's copying, yet it exercised some of the machinery of writing. It might be useful for copying a lot of data quickly (because it doesn't interpret), but it's another code path that needs to be kept alive and answer questions about usage.

The tests that I put in when copy_from was first implemented still work—have you found a case that doesn't? If so, how does it differ from the tests? Sorry, I mean "test" (singular):

% fgrep copy_from tests/*.py
tests/test_0345-bulk-copy-method.py:            dest.copy_from(source, "subdir/hist")

Is this an operation that you need to do—fast bulk copying? If it's not something you actively need and the copy_from function is indeed broken, my quick fix would be to hide the function and only bring it back (with enough care to ensure that all the cases work) if someone says they need it.

Writing new data to a file—with examples farther down in this thread or in the official documentation, are fully supported.

[archiron]

I only want to copy all the branch (here DQMData/Run 1/EgammaV) into a new file.

Perhaps it's me that does not use uproot correctly. I use :

fileName1 = rootFilesPath + '/' + "DQM_V0001_R000000001__RelValZEE_13__CMSSW_10_6_20-106X_mc2017_realistic_v9-v1__DQMIO.root"
fileName2 = rootFilesPath + '/' + "result.root"

with up4.open(fileName1) as source:
    with up4.writing.recreate(fileName2) as dest:
        dest.copy_from(source, "DQMData/Run 1/EgammaV")

but the only result I have into the new file is empty directories : DQMData/Run 1/EgammaV.

[jpivarski]

Here's a reason why copy_from should perhaps be retired: it won't copy the data associated with a TTree. What it does is recursively copy self-contained objects, and TTrees refer to data outside of themselves (the TBaskets), so by its name, it gave a misleading impression that it would copy the TTrees when it can't.

To copy the TTrees, you'll need to iterate over the first file and write to the new file. Enough people have asked about copying TTrees that we ought to have more tools for doing that (exact copying, copying and renaming branches, dropping branches, adding branches...).

[jpivarski]

Today's update: PR #349 adds the ability to write TObjStrings from scratch (they don't need to come from another file), assigning the TStreamerInfo appropriately. With this update, ROOT files can be used as persistent dicts of strings:

https://github.com/scikit-hep/uproot4/blob/1ce55e34285be232f5255730e090b83189e03061/tests/test_0349-write-TObjString.py#L17-L29

Next is histograms.

[jpivarski]

After 3 months of not being able to get to this, I managed to start working on ROOT file writing in Uproot again. PR #405 writes a histogram. It will have to be generalized beyond TH1F version 3, but now I have a method for filling in the missing serialization code.

def test(tmp_path):
    original = os.path.join(tmp_path, "original.root")
    newfile = os.path.join(tmp_path, "newfile.root")

    f1 = ROOT.TFile(original, "recreate")
    h1 = ROOT.TH1F("h1", "title", 8, -3.14, 2.71)
    h1.SetBinContent(0, 0.0)
    h1.SetBinContent(1, 1.1)
    h1.SetBinContent(2, 2.2)
    h1.SetBinContent(3, 3.3)
    h1.SetBinContent(4, 4.4)
    h1.SetBinContent(5, 5.5)
    h1.SetBinContent(6, 6.6)
    h1.SetBinContent(7, 7.7)
    h1.SetBinContent(8, 8.8)
    h1.SetBinContent(9, 9.9)
    h1.Write()
    f1.Close()

    with uproot.open(original) as fin:
        h2 = fin["h1"]

        with uproot.recreate(newfile) as fout:
            fout["h1"] = h2

    f3 = ROOT.TFile(newfile)
    h3 = f3.Get("h1")
    assert h3.GetBinContent(0) == pytest.approx(0.0)
    assert h3.GetBinContent(1) == pytest.approx(1.1)
    assert h3.GetBinContent(2) == pytest.approx(2.2)
    assert h3.GetBinContent(3) == pytest.approx(3.3)
    assert h3.GetBinContent(4) == pytest.approx(4.4)
    assert h3.GetBinContent(5) == pytest.approx(5.5)
    assert h3.GetBinContent(6) == pytest.approx(6.6)
    assert h3.GetBinContent(7) == pytest.approx(7.7)
    assert h3.GetBinContent(8) == pytest.approx(8.8)
    assert h3.GetBinContent(9) == pytest.approx(9.9)

[jpivarski]

And now any histogram (descendants of TH1, excluding TH1K, TH2Poly, TProfile2Poly, and TGLTH3Composition) can be written to a new file (uproot.recreate) or added to an existing file (uproot.update): PR #405.

As a reminder, that's all of these:

So, all the 1, 2, and 3D histograms with profiles, but no TEfficiency. There are also functions like uproot.writing.to_th1x to create these writable objects from data, rather than from another file. The idea is that uproot.writing.to_writable, which is called every time this happens:

outfile = uproot.recreate("outfile.root")
outfile["name_of_histogram"] = xyz

will use these methods to attempt to convert xyz into something writable. The first xyz types that will be recognized are tuples of NumPy arrays, so that they can be auto-converted into Uproot histograms for writing. The second types that will be recognized are hist and boost-histogram objects (@henryiii and @amangoel185).

For PyROOT objects (conversion in both directions), I think we should do something more general, serializing and deserializing uproot.Model objects as TMessages, which PyROOT objects can also serialize and deserialize, such that any PyROOT object can become an Uproot object, and any writable Uproot object can become a PyROOT object ("writable" is a subset that only includes TObjString and histograms right now).

[jpivarski]

Moving along, PR #406 creates an empty TTree (with a TBranch, but no TBasket data). This has to follow a different code path from histograms and other "normal" objects because TTrees can create and manage other objects (TBaskets), and they have to be replaceable, if the number of TBaskets exceeds the originally allocated number. (A TTree has to communicate with the TDirectory that contains it to replace the TTree metadata with larger metadata. The TBaskets themselves never get copied or moved.)

    with uproot.recreate(newfile, compression=None) as fout:
        # internal interface; not wrapped up in a high-level function yet
        tree = fout._cascading.add_tree(
            fout._file.sink, "t1", "title", {"branch1": np.float64}
        )

    f2 = ROOT.TFile(newfile)
    t2 = f2.Get("t1")

    assert t2.GetName() == "t1"
    assert t2.GetTitle() == "title"

    assert t2.GetBranch("branch1").GetName() == "branch1"
    assert t2.GetBranch("branch1").GetTitle() == "branch1/D"

    assert t2.GetBranch("branch1").GetLeaf("branch1").GetName() == "branch1"
    assert t2.GetBranch("branch1").GetLeaf("branch1").GetTitle() == "branch1"

    assert t2.GetLeaf("branch1").GetName() == "branch1"
    assert t2.GetLeaf("branch1").GetTitle() == "branch1"

The TTree above is byte-for-byte identical with one made this way:

    f1 = ROOT.TFile(original, "recreate")
    f1.SetCompressionLevel(0)

    t1 = ROOT.TTree("t1", "title")
    d1 = array.array("d", [0.0])
    d2 = array.array("d", [0.0])
    t1.Branch("branch1", d1, "branch1/D")

    t1.Write()
    f1.Close()

[jpivarski]

Today's update (PR #408): we can now write NumPy arrays to a TTree in Uproot and read them back in ROOT.

    newfile = os.path.join(tmp_path, "newfile.root")

    b1 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    b2 = [0.0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8, 9.9]

    with uproot.recreate(newfile, compression=None) as fout:
        tree = fout.mktree("t", "title", {"b1": np.int32, "b2": np.float64})   # analogy with mkdir
        tree.extend({"b1": b1, "b2": b2})   # iterables are cast to arrays of the appropriate type
        tree.extend({"b1": b1, "b2": b2})

    # read back in ROOT
    f1 = ROOT.TFile(newfile)
    t1 = f1.Get("t")
    assert t1.GetEntries() == len(b1) + len(b1)
    assert [x.b1 for x in t1] == b1 + b1
    assert [x.b2 for x in t1] == b2 + b2

    # read back in Uproot
    with uproot.open(newfile + ":t") as t2:
        assert t2.num_entries == len(b1) + len(b1)
        assert t2["b1"].array().tolist() == b1 + b1
        assert t2["b2"].array().tolist() == b2 + b2

Still, there are limitations:

• Each extend makes a TBasket in every TBranch, and that number can't exceed 10. When a TTree exceeds its initial capacity for baskets (10), it will have to rewrite itself to become larger, and that's a TODO.
• Baskets are not compressed.
• We can't add data to TTrees from preexiting files (uproot.update) yet.
• No jagged arrays yet. Eventually, the syntax will be just

fout["treename"] = awkward_array
fout["treename"].extend(another_awkward_array)

but that's to come.

[jpivarski]

Uproot file-writing is in a usable state!

As of PR #409 (which is now merged into main), Uproot's file-writing code can be considered alpha quality: ready for testers, albeit with no documentation and you'd be the first to test it. Fortunately for documentation, there's not much to explain.

The interface for writing a TTree is

with uproot.recreate("filename.root") as output_file:
    output_file["tree_name"] = {"branch1": array1, "branch2": array2, "branch3": array3}

where array1, array2, array3, etc. are NumPy arrays of the same length (maybe different dtypes). This will create a TTree named "tree_name" with the array data as branches, the opposite of input_file["tree_name/branch1"].array().

Once you've created a TTree, you can add more data to it with

    output_file["tree_name"].extend({"branch1": array1, "branch2": array2, "branch3": array3})

The set of branch names needs to be the same (dtypes will be converted if they don't match the first batch), and again, the lengths of all of these arrays must be equal. The file must still be open. Each call to extend adds another TBasket to every TBranch, so be sure to extend the TTree in large batches, like MB at least, to ensure that the file has big enough TBaskets for good performance. (Appending length-1 arrays would lead to the worst possible read performance.)

The interface for writing a histogram is

    output_file["hist_name"] = np.histogram(some_data)

where the right-hand-side is some "recognized type," a type that can be converted into a histogram. The first such type is a 2-tuple of NumPy arrays—what np.histogram creates.

Fancy stuff:

• The name that you assign to can be a "deeply/nested/directory". If you want to make a directory without putting any objects in it, you can use output_file.mkdir("name") (and output_file.mktree("name", "title", {"branch": dtype}) for empty TTrees).
• uproot.recreate creates a new ROOT file (overwriting anything that previously existed with that name), and uproot.update opens a previously existing ROOT file, so that you can add more TTrees and histograms or delete objects. The original ROOT file could have been made by ROOT or by Uproot.
• You can delete objects (uproot.recreate or uproot.update mode) with del output_file["object_name"].

Temporary limitations:

• Input arrays for making TTrees have to be one-dimensional NumPy arrays. Multidimensional NumPy, single-jagged Awkward, and Pandas DataFrames are TODO items.
• Only 2-tuples of NumPy arrays are currently recognized as histograms. 3-tuples (TH2) and 4-tuples (TH3) are TODO items, as well as boost-histogram and hist objects.
• PyROOT objects are not yet recognized entities, but they will be. (They'll be serialized and deserialized through a TMessage.) The same for reading: all Uproot Models will get a to_pyroot() method to convert themselves into PyROOT objects (through a TMessage).
• Nothing is compressed yet, but in principle, the same compression libraries (zlib, lz4, lzma, zstd) can be used to compress as to decompress.

Permanent limitations:

• Once a histogram is written to the file, there is no modifying it. You can del output_file["hist_name"] and assign it again, or just assign it again and get a new cycle number (which does not delete the first with that name). If you try to look at output_file["hist_name"] after writing it, you'll get a read-only histogram object.
• When you add a TTree to a file, it is always a WritableTree; looking at it again with output_file["tree_name"] gives you the same writable object back. This writable object does not have all the methods and properties that a read-only TTree has.
• You cannot extract a read-only TTree from a file opened with uproot.recreate or uproot.update. TTrees are either write-only or read-only, depending on how the file was opened.

Probably permanent limitation:

• Can only write to local files. I don't know what it would take to write through XRootD, and I don't know if it batches data for performance. If it does any batching, that batching would have limited performance if you interleave reads and writes of a uproot.update file.

[jpivarski]

Update from PR #412: I fixed some of the "temporary limitations" above, including usability feedback from @alexander-held (on https://gitter.im/Scikit-HEP/uproot):

• rootfile["tree"] = {"branch": [ ... simple list ... ]} now converts the list to NumPy and writes it into the ROOT TTree as a branch. Any non-Mapping Iterable that can be converted into a NumPy array with integer or floating point type (i.e. no dtype("O")) will be converted into a NumPy array, and from there into a branch of a ROOT TTree. This includes multidimensional arrays.
• multidimensional NumPy arrays (i.e. not jagged arrays, since those aren't NumPy or have dtype("O"), which we strongly avoid)
• NumPy structured arrays. These aren't written to ROOT as leaf-lists but as simple branches; Uproot will only write fully split data.
• Pandas DataFrames; you can say rootfile["df"] = df and save your Pandas DataFrames in a ROOT file. The index has to be numeric (df.index.is_numeric()). That leaves open the possibility of identifying a DataFrame with an interval index as a histogram and a multiindex as a jagged array.
• jagged arrays (that will be a PR on its own)
• the NumPy forms (tuples of arrays) of 1D, 2D, and 3D histograms are all recognized, both the np.histogram2d format (flat tuple of entries and bin edges) and the np.histogramdd format (nested tuple for the bin edges). In particular, I tested all of the quantities for getting the stats box right.
• writing boost-histogram and hist objects to ROOT files. @henryiii and @amangoel185, I'd like to talk sometime to share knowledge on that; as a preview from my side, this is what's involved in recognizing the 1D NumPy histogram case. I don't think boost-histogram/hist would be much more complicated.
• writing generic PyROOT objects
• compression

Now just for showing off, behold:

>>> import pandas as pd
>>> df = pd.DataFrame({"x": [1, 2, 3, 4, 5], "y": [1.1, 2.2, 3.3, 4.4, 5.5]})
>>> df
   x    y
0  1  1.1
1  2  2.2
2  3  3.3
3  4  4.4
4  5  5.5
>>>
>>> import uproot
>>> with uproot.recreate("output.root") as rootfile:
...     rootfile["tree"] = df
... 
>>>
>>> import ROOT
>>> f = ROOT.TFile("output.root")
>>> t = f.Get("tree")
>>> t.Scan()
************************************************
*    Row   * index.ind *       x.x *       y.y *
************************************************
*        0 *         0 *         1 *       1.1 *
*        1 *         1 *         2 *       2.2 *
*        2 *         2 *         3 *       3.3 *
*        3 *         3 *         4 *       4.4 *
*        4 *         4 *         5 *       5.5 *
************************************************
5
>>> t.Print()
******************************************************************************
*Tree    :tree      :                                                        *
*Entries :        5 : Total =            2133 bytes  File  Size =       2144 *
*        :          : Tree compression factor =   1.00                       *
******************************************************************************
*Br    0 :index     : index/L                                                *
*Entries :        5 : Total  Size=        609 bytes  File Size  =        112 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    1 :x         : x/L                                                    *
*Entries :        5 : Total  Size=        589 bytes  File Size  =        108 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    2 :y         : y/D                                                    *
*Entries :        5 : Total  Size=        589 bytes  File Size  =        108 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*

(The DataFrame index becomes a TBranch named "index," unless overwritten by a DataFrame column named "index.")

Thus, a ROOT file is a handy way to store Pandas DataFrames.

[jpivarski]

PR #414 adds jagged array writing.

Jagged arrays present a problem of interface. Although we have a choice between writing them as dynamic length arrays (e.g. "Muon_pt[NMuon]/F") and writing them as std::vectors, dynamic length arrays are both more efficient on disk and faster for Uproot to read and write because each entry does not have a header, making it possible to just dump or load numerical data without inserting or removing std::vector headers at non-uniform places in the byte stream. So we want dynamic length arrays, but these require some other branch to contain a counter (the "NMuon/I" that goes with the "Muon_pt[NMuon]/F"). One array becomes two branches.

What do we name these extra branches? How do we keep them from colliding? How do we avoid creating a lot of them?

For the first two problems, I added counter_name and field_name arguments to WritableDirectory.mktree, which are functions that determine how branch names will be built. Their defaults are the NanoAOD conventions:

• counter_name = lambda counted: "N" + counted: i.e. an array named Muon gets a counter named NMuon
• field_name = lambda outer, inner: inner if outer == "" else outer + "_" + inner: i.e. a record array named Muon with fields pt, eta, phibecomes branchesMuon_pt, Muon_eta, Muon_phi`.

If you need different names or you need to avoid a name collision, you can pass your own lambda functions.

For the last problem of avoiding many of them, I made sure that we can write Awkward record arrays as well as numerical or jagged-numerical arrays. This is a direct extension of the rootfile["tree"] = df DataFrame writing described in the comment above. If you have a lot of jagged arrays with the same number of values per entry each, you can ak.zip them into a single array.

>>> import uproot
>>> import awkward as ak
>>> one = ak.Array([[1, 2, 3], [], [4, 5]])
>>> two = ak.Array([[1.1, 2.2, 3.3], [], [4.4, 5.5]])
>>> three = ak.Array([[100], [200, 200], [300, 300, 300]])
>>> with uproot.recreate("output.root", compression=None) as rootfile:
...     rootfile["tree1"] = ak.zip({"one": one, "two": two})
...     rootfile["tree2"] = {"A": ak.zip({"one": one, "two": two}), "B": three}
... 
>>> import ROOT
>>> f = ROOT.TFile("output.root")
>>> tree1 = f.Get("tree1")
>>> tree2 = f.Get("tree2")

The first TTree, "tree1", takes the record array to be the complete set of branches, adding only a counter branch named "N".

>>> tree1.Print()
******************************************************************************
*Tree    :tree1     :                                                        *
*Entries :        3 : Total =            2145 bytes  File  Size =       2161 *
*        :          : Tree compression factor =   1.00                       *
******************************************************************************
*Br    0 :N         : N/I                                                    *
*Entries :        3 : Total  Size=        550 bytes  File Size  =         81 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    1 :one       : one/L                                                  *
*Entries :        3 : Total  Size=        685 bytes  File Size  =        131 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    2 :two       : two/D                                                  *
*Entries :        3 : Total  Size=        685 bytes  File Size  =        131 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
>>> [x.N for x in tree1]
[3, 0, 2]
>>> [list(x.one) for x in tree1]
[[1, 2, 3], [], [4, 5]]
>>> [list(x.two) for x in tree1]
[[1.1, 2.2, 3.3], [], [4.4, 5.5]]

The second TTree, "tree2", takes the record array to be under the names "A" and "B". The counters named "NA" and "NB" distinguish the arrays with different numbers of items per entry.

>>> tree2.Print()
******************************************************************************
*Tree    :tree2     :                                                        *
*Entries :        3 : Total =            3349 bytes  File  Size =       3395 *
*        :          : Tree compression factor =   1.00                       *
******************************************************************************
*Br    0 :NA        : NA/I                                                   *
*Entries :        3 : Total  Size=        555 bytes  File Size  =         82 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    1 :A_one     : A_one/L                                                *
*Entries :        3 : Total  Size=        697 bytes  File Size  =        133 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    2 :A_two     : A_two/D                                                *
*Entries :        3 : Total  Size=        697 bytes  File Size  =        133 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    3 :NB        : NB/I                                                   *
*Entries :        3 : Total  Size=        555 bytes  File Size  =         82 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    4 :B         : B/L                                                    *
*Entries :        3 : Total  Size=        685 bytes  File Size  =        137 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
>>> [x.NA for x in tree2]
[3, 0, 2]
>>> [list(x.A_one) for x in tree2]
[[1, 2, 3], [], [4, 5]]
>>> [list(x.A_two) for x in tree2]
[[1.1, 2.2, 3.3], [], [4.4, 5.5]]
>>> 
>>> [x.NB for x in tree2]
[1, 2, 3]
>>> [list(x.B) for x in tree2]
[[100], [200, 200], [300, 300, 300]]

I know that TTrees can have nested branches and this could be a good application of nested branches (i.e. distinguish them as "A/N", "A/one", "A/two", "B/N", and "B/three"), but users of TTrees, particularly at the level of type-complexity that Uproot-writing will ever support, have been opting for naming conventions in non-nested branches. Also, that's much easier for me.

Now that this is done, all that's left is

• and (one more) refactor the writing.py and _writing.py modules into submodules; they're too large
• writing boost-histogram and hist objects to ROOT files
• writing generic PyROOT objects (see below)
• compression (see below)
• documentation

[jpivarski]

(The refactoring task is done in #415.)

[jrueb]

Really nice!
You wrote that the counter_name defaults to the NanoAOD convention, but in NanoAOD the prefix of the counter is always a lower case 'n', while it is upper case in uproot right now if I understand correctly. Where does the difference come from?

[jpivarski]

You're right! I don't know where I got that from. I'll change the default counter_name to lambda counted: "n" + counted.

[masonproffitt]

Is it possible to put an option in somewhere to write std::vectors instead of dynamic length arrays? Most existing ROOT analysis code that I've seen expects std::vectors, so having this as an option would be a huge benefit for compatibility with other code.

[jpivarski]

Ach, no, this was not added. And now I remember that it was requested, in #257!

While implementing, I was thinking about how writing std::vector would simplify the interface—you wouldn't need counter branches—but it would complicate the implementation and definitely slow down the execution time. I'll keep thinking about how to add this as an extra feature for when you're willing to make that trade-off, but it did not make it into this sprint, sorry.

[jpivarski]

PR #416 adds the ability to compress data written to ROOT files: objects like histograms and also TTree data. The compression setting can be specified when creating the file:

with uproot.recreate("filename.root", compression=uproot.ZLIB(5)) as file:
    ...

or it can be assigned to a file later:

file.compression = uproot.LZMA(9)

or it can be assigned to a tree (and therefore be different from the rest of the file):

tree = file.mktree("tree", {"branch1": numpy_dtype1, "branch2": awkward_type2})
tree.compression = uproot.LZ4(1)

or it can be set on a per-branch level:

tree["branch1"].compression = uproot.ZSTD(4)

or

tree.compression = {"branch1": uproot.ZSTD(4), "branch2": uproot.LZ4(1)}

Since WritableFile, WritableDirectory (which just passes it to its file), and WritableTree are all mutable, you can set the compression to one value, write some data, set it to another value, and write some more data. The compression that gets used is the configuration at the time of writing. Thus, it's even possible for different baskets of the same branch to be compressed differently, though I can't see how that's a good idea. As far as I can tell, it is allowed by ROOT, since ROOT uses a specialized header at the beginning of a TObject or TBasket to determine how to decompress some data; it doesn't seem to ever use the file-level or tree-level fCompress value to decide how to decompress. That's good.

[pcanal]

Thus, it's even possible for different baskets of the same branch to be compressed differently,

IRL this happens when (fast) merging TTrees from files written some time apart with different compression request.

[pcanal]

; it doesn't seem to ever use the file-level or tree-level fCompress value to decide how to decompress. That's good.

Indeed, each compressed buffer carries the compression algorithm used with it.

Another practical use is to compression different branches with different compression alogithm (for example, one for indices and one for data).

[jpivarski]

Hi @pcanal!

PR #420 adds PyROOT-Uproot interoperability:

• any PyROOT object can be converted to its Uproot equivalent through uproot.from_pyroot(obj), if Uproot would have been able to read instances of that class from a file (a rather broad set);
• any uproot.Model can be converted to its PyROOT equivalent through obj.to_pyroot(), if Uproot would have been able to write that class to a file (just TObjString and histograms—quite limited).

The reason for these caveats is because the data conversion goes through TMessages: they're serialized to bytes and back, though that all happens in memory (no disk). The main motivation for this is to be able to write PyROOT objects to the same files you're writing hist histograms as TH* or Awkward Arrays as TTrees (without having to reopen the file with PyROOT and doing it the conventional way—this is a convenience).

with uproot.recreate("filename.root") as file:
    file["hist"] = hist_object
    file["tree"] = {"branch1": jagged_array, "branch2": numpy_array}
    file["workspace"] = roofit_workspace

(When PyROOT objects are being written to files, they do not need to be readable by Uproot because the raw bytes from the TMessage are directly copied into the file, albeit with compression if you ask for it. Also, all of the appropriate TStreamerInfo is included in the output file, which we get from PyROOT by making a temporary TMemFile, once per class type. If you have to write a thousand histograms, only one TMemFile will be made for the TStreamerInfo, but a thousand TMessages will be made.)

The only things left on my to-do list are interpreting hist objects and documentation. @henryiii and @amangoel185, we should talk about the hist objects; I'll ping you on Slack. With the documentation finished, this will be released as 4.1.0.

[jpivarski]

As of PR #421, it is now documented. See the last 1/3rd of the Getting Started Guide.

[allenxiangxin]

Thanks for adding the write feature. Useful stuff.

[goi42]

As I accidentally stated here,

Thank you, @jpivarski, for putting this together. This was a really essential project for the interoperability of ROOT, and we’re all better off for your effort.

[jpivarski]

As a last update (in this sprint, at least), I've released Uproot 4.1.1 (PyPI) with several writing-related performance bugs and one bug-bug fixed, all in PRs #426 and #428. The GitHub pages have some measurements with the code used to make them.

A rule of thumb that comes out of this is that you want to call TTree.extend with no less than about 100 kB per array branch. At that point, the Pythonic overhead for writing TBasket metadata is in the noise:

This is also the scale at which reading data becomes efficient, too, but in writing, you get to control it.

@masonproffitt's question is buried in a thread above, but writing of std::vector types (as opposed to dynamically sized arrays, both of which are jagged) has not been implemented yet. This is also issue #257, which remains open.

[eguiraud]

Hi,

I was wondering that the status of uproot writing is w.r.t. inferring the right counters from the input tree.

For example here I'm reading Muon_pt and its counter nMuon from an input NanoAOD and I try to write them in a new tree with uproot, but it looks like uproot things Muon_pt should have a counter called nMuon_pt rather than just nMuon:

import uproot
fname = "https://xrootd-local.unl.edu:1094//store/user/AGC/nanoAOD/TT_TuneCUETP8M1_13TeV-powheg-pythia8/cmsopendata2015_ttbar_19980_PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1_00000_0000.root"
in_tree, out_file = uproot.open(fname + ":Events"), uproot.recreate("out.root")

def get_uproot_type(interpretation):
    try:
        return interpretation.to_dtype
    except:
        return interpretation.content.to_dtype

branch_dict = {name: get_uproot_type(branch.interpretation) for name, branch in in_tree.items() if name == "Muon_pt" or name == "nMuon"}
out_tree = out_file.mktree("Events", branch_dict)
iterator = in_tree.iterate(branch_dict.keys(), step_size="1 MB")
batch = next(iterator)
contents = {name: batch[name] for name in branch_dict.keys()}
out_tree.extend(contents)

yields ValueError: 'extend' was given data that do not correspond to any branch: 'nMuon_pt'.

Question 2. is whether there is a better, built-in way to convert the input branch types to the dtypes that works for ~all branches, I realize my get_uproot_type function is hacky at best.

(Question 3. is whether I'm completely misusing uproot here 😄 )

EDIT:
For the counter issue an ancillary question is why does uproot not get that information from TBranch directly?

[jpivarski]

Each TBranch is independently read from a ROOT file, producing an Awkward Array with independent offsets for each output array (i.e. each field of the output array of records):

>>> import uproot
>>> import awkward as ak
>>> in_tree = uproot.open("Run2012B_DoubleMuParked.root")["Events"]
>>> muon_pt_eta = in_tree.arrays(["Muon_pt", "Muon_eta"])
>>> muon_pt_eta.type.show(type=True)
type: 1000000 * {
    Muon_pt: var * float32,
    Muon_eta: var * float32
}

The key thing is that the type has Muon_pt: var * float32 and separately Muon_eta: var * float32. There's no assumption that they share the same "var". This could be discovered by checking to see if both TBranches reference the same counter TLeaf, but reading treats each input TBranch independently—it doesn't even look at the counters or require there to be counters (because all of the offsets can be found in the TBaskets).

Writing is different. We can't make variable array type data without also making their counters because there are ROOT functions that use these counters. So, given a jagged array, Uproot has to make two TBranches, one for the entire jagged array (including offsets) and another for the counter. The uproot.WritableDirectory.mktree function has some handles to control what the counter TBranch will be named,

• counter_name (callable of str → str) – Function to generate counter-TBranch names for Awkward Arrays of variable-length lists.

and the default is to put "n" at the beginning of the name (like NanoAOD).

This becomes problematic if you have a lot of arrays that should share the same counter TBranch, such as Muon_pt, Muon_eta, Muon_phi, Muon_.... Uproot has to somehow be informed that these arrays happen to have the same number of items in each list. The way this can be done is to make the separate arrays for each particle attribute into a single array of particles with attributes.

>>> muons = ak.zip({"pt": muon_pt_eta["Muon_pt"], "eta": muon_pt_eta["Muon_eta"]})
>>> muons.type.show()
1000000 * var * {
    pt: float32,
    eta: float32
}

Now there's only one "var", outside of the record-type. If the separate arrays didn't have the same number of items in each list, the ak.zip would fail.

This is an Awkward type that Uproot recognizes, and upon seeing it, Uproot will make all of the output TBranches share a single counter TBranch because it knows that it can do that. Now, though, it also needs to generate names for TBranches from fields of a record array, which is another uproot.WritableDirectory.mktree argument,

• field_name (callable of str → str) – Function to generate TBranch names for columns of an Awkward record array or a Pandas DataFrame.

and the default is to use "_" as a separator (like NanoAOD).

>>> out_file = uproot.recreate("/tmp/output.root")
>>> out_file["out_tree"] = {"Muon": muons}
>>> out_file["out_tree"].show()
name                 | typename                 | interpretation                
---------------------+--------------------------+-------------------------------
nMuon                | int32_t                  | AsDtype('>i4')
Muon_pt              | float[]                  | AsJagged(AsDtype('>f4'))
Muon_eta             | float[]                  | AsJagged(AsDtype('>f4'))

This asymmetry between reading and writing is because the data model of Awkward Arrays doesn't perfectly match the data model of TTree. To make them match, reading could get more aggressive about zipping TBranches that share a counter, but that would surprise long-time users of Uproot, who expect 1 TBranch → 1 array. (Also, that's a nice simplifying assumption.) Writing can't get less aggressive about unzipping records into TBranches with a shared counter because somehow the knowledge of which TBranches should share a counter has to be communicated. It could have been done with arguments to the function, but then if somebody had data like muons, above, they'd have to do even more work to express information that's already latent in the structure.

And anyway, data structures like muons, above, are more convenient for doing analysis, since you can slice all fields of an array of particles with one line, instead of doing it individually for each attribute.

[eguiraud]

Thank you for the explanation, Jim!

the default is to put "n" at the beginning of the name (like NanoAOD).

That's not the NanoAOD convention it, I think? E.g. Muon_pt's counter is not nMuon_pt, but nMuon (hence the error in my snippet above).
I guess you are implicitly assuming that when reading Muon_{pt,eta,...} they always get zipped into a Muon?
Based on your explanation, however, I would think that if you used the actual NanoAOD naming convention (something like lambda name: 'n' + name.rsplit('_')[0]), then the writing logic would automatically use the same counter for several separate branches (and I guess that if the same counter is requested multiple times it is still only written once).

So I think things could work out of the box for NanoAODs thanks to their regular naming conventions? In general I understand that I should explicitly inform mktree of how the counters are called.

[jpivarski]

See the last example: since the key of the dict in

>>> out_file["out_tree"] = {"Muon": muons}

is "Muon", the name of the counter branch becomes "nMuon", which is NanoAOD convention.

What you can't do is read data out of one file and write it into another file without reformatting with ak.zip. The information that the writing step would need (the fact that all of the arrays suggestively named "Muon_" ought to share a counter) is lost.

[eguiraud]

Yes I understand, as well as the will to prioritize the case of "zipped" SoAs like "Muon" over single columns like "Muon_pt".

The information that the writing step would need (the fact that all of the arrays suggestively named "Muon_" ought to share a counter) is lost.

About this specific point, what I wanted to say is that the writing side of uproot could easily recover that information from the column names (since uproot already bases its default counter_name on NanoAOD naming conventions). It could even support both SoAs like Muon and individual columns like Muon_pt if counter_name were able to distinguish the two cases, e.g. with a second argument that contains the type of the column:

def counter_name(name, type_ = None):
  if is_zip(type): # SoA
    return 'n' + name
  else:
    return 'n' + name.rsplit('_')[0]

[jpivarski]

If counter_name is the function above, then each Muon_pt, Muon_eta, Muon_... would each get a counter named nMuon. I'm not really sure what would happen if multiple input arrays generate counter branches with the same name, though in this case, they would also have the same value (unchecked, unlike the ak.zip method). It's possible that, the way it's implemented now, only one counter would get written.

I'd like to avoid, as much as possible, CMS-specific details getting into Uproot. The default values of the counter_name and field_name functions match NanoAOD only because there isn't a competing convention that I'm aware of and this is a style that's sometimes even adopted by non-CMS analyses. Splitting strings (i.e. assuming that they have an underscore) is a little further down that road I wanted to avoid than just appending "n" and joining by "_".

But there's plenty of room for specialization by building on top of Uproot. It was a design goal for Uproot to just be a base layer that other packages build on top of. This name-manipulation is a good candidate for that.

[eguiraud]

Fair enough, thank you for your patience!

At the end of the day my goal was to read some columns from a NanoAOD-like TTree and write them out in another -- based on this conversation my understanding is that it's not possible without re-injecting, between the reading and the writing part, some information about objects ("Muon is made of Muon_pt, Muon_eta, Muon_phi, ...") and their counters (i.e. some domain knowledge about the actual schema). Good to know!

[xaratustrah]

Hi @jpivarski, and everyone, thanks for updating on this! I was just wondering, is there an example of how to use uproot.behaviors.TH1.TH1? I have been using TH1 with uproot3 and uproot3_methods successfully here, btw. same with writing a tree, like here. Wondering if it would be possible to move on to uproot5?

[jpivarski]

The TH1 interface is documented here, though the most useful methods are the ones that take you out of Uproot into boost-histogram (TH1.to_boost) or hist (TH1.to_hist). The design strategy for Uproot v4 (and v5) is to get out of the business of providing interfaces for all the ROOT data types and instead pass them to libraries that specialize in each type of functionality.

These histogram objects are also recognized by Uproot's file-writing, as documented here. The blue box lists all of the types that are recognized on the right-hand side of file-writing assignment.

It should be possible to modify anything that worked in Uproot v3 to use the new interfaces in v4/v5. I can't think of anything that v3 could do that v4/v5 can't do—it should be a strict superset.