Writing arrays to ROOT files in Uproot 4

[jpivarski]

From @johnmcgowan1 in scikit-hep/uproot3#532:

Just a question, so maybe an Issue is not the right place to ask. Will uproot support the writing of ndarrays in the future? If so is there a timescale on which we can expect this?

And also @Skagamaster in Gitter:

Sorry for posting int he incorrect forum, and thanks for the input! My arrays aren't rectangular, so I'll have to go with the ListOffsetArray method (or pad or flatten, as suggested).

Uproot 3's ability to write NumPy arrays and simple Awkward Arrays (lists of booleans/numbers only) will be ported into Uproot 4 this spring. This will also include writing histograms (and TObjString, though that's primarily because it's a simple type for debugging). I can see that it's a frequently requested feature, but it will take a few months to complete. Everything in Uproot 4 is a rewrite of the corresponding part of Uproot 3 to use the new Awkward Array API and make Uproot more maintainable (e.g. what used to be two large files in Uproot 3 has been split into specialized classes, one per file, with improved debugging, ROOT class version handling, and all that). I'm going to do the same thing for @reikdas's file-writing classes. He put a lot of effort into understanding how to write correct ROOT files—which is a different problem from merely reading them—and that needs to be translated into more future-proof code, like all the rest of Uproot 3.

You can follow progress on its project board, though work won't begin in earnest until late February or early March (juggling with other deadlines). Originally, this was one of the items that I wanted to include before renaming uproot4 → uproot, but considering how long file-writing would take, I didn't want it to delay the name change. (The new version had already been in this opt-in state for about 5 months by that point.)

What will be included:

• Creating ROOT files that can be read and updated in ROOT, but probably not updating files made by ROOT for reasons described here: Is uproot.update not yet supported? uproot3#381 (comment).
• Writing histograms (descendants of TH1) and TObjStrings.
• Writing TTrees with arrays that don't have to be in memory all at once—i.e. progressively extending the file in chunks. (Those chunks becomes TBaskets).
• "Arrays" includes all NumPy arrays, including multidimensional and numeric dtypes (booleans and numbers), and maybe structured arrays → leaf-lists. It also includes Awkward lists of numeric types; that is, ListArray, ListOffsetArray, RegularArray, and NumpyArray from Awkward Array (and EmptyArray as a non-useful trivial case, for completeness).

For the time being, if you need to write arrays to ROOT files, you have a few options:

• Import both uproot and uproot3 in the same Python process* (this is why they have different names, so you can do that) and follow these instructions to write histograms and follow these instructions to write TTrees from NumPy arrays.

• If you have an Awkward Array of lists ("jagged array"), this is supported in Uproot 3 but not documented. Here is the pull request, which has some discussion of the API and points to the tests that demonstrate it. If your arrays are in Awkward 1.x format, convert them into Awkward 0.x using ak.to_awkward0.

• Read NumPy arrays into RDataFrame and use RDataFrame to write files (syntax is ROOT.RDF.MakeNumpyDataFrame(dict_of_numpy_arrays)).

• I'm planning to add round-trip transformations between arbitrary Awkward Arrays and RDataFrame, so this will be a long-term option (not just for writing files, but also for mixing Awkward/Coffea and C++ analysis in general).

• To use uproot and uproot3 in the same Python process, as well as awkward and awkward0, make sure to update all relevant packages: everything should be to the right of the transition line: