% ROOT Version 6.24 Release Notes % 2020-05-19
ROOT version 6.24/00 is scheduled for release in November 2020.
For more information, see:
The following people have contributed to this new version:
Guilherme Amadio, CERN/SFT,
Bertrand Bellenot, CERN/SFT,
Josh Bendavid, CERN/CMS,
Jakob Blomer, CERN/SFT,
Rene Brun, CERN/SFT,
Philippe Canal, FNAL,
Olivier Couet, CERN/SFT,
Massimiliano Galli, CERN/SFT,
Andrei Gheata, CERN/SFT,
Hadrien Grasland, IJCLab/LAL,
Enrico Guiraud, CERN/SFT,
Claire Guyot, CERN/SFT,
Jonas Hahnfeld, CERN/SFT,
Emmanouil Michalainas, CERN/SFT,
Stephan Hageboeck, CERN/SFT,
Sergey Linev, GSI,
Javier Lopez-Gomez, CERN/SFT,
Pere Mato, CERN/SFT,
Lorenzo Moneta, CERN/SFT,
Alja Mrak-Tadel, UCSD/CMS,
Axel Naumann, CERN/SFT,
Vincenzo Eduardo Padulano, CERN/SFT and UPV,
Danilo Piparo, CERN/SFT,
Fons Rademakers, CERN/SFT,
Jonas Rembser, CERN/SFT,
Andrea Sciandra, SCIPP-UCSC/Atlas,
Oksana Shadura, UNL/CMS,
Enric Tejedor Saavedra, CERN/SFT,
Christian Tacke, GSI,
Matevz Tadel, UCSD/CMS,
Vassil Vassilev, Princeton/CMS,
Wouter Verkerke, NIKHEF/Atlas,
Stefan Wunsch, CERN/SFT,
Anirudh Dagar, CERN-HSF/GSoC
RooAbsReal::evaluateBatch()
has been removed in favour of the faster evaluateSpan(). See section "RooFit Libraries" for instructions on how to useRooAbsReal::evaluateSpan()
.TTreeProcessorMT::SetMaxTasksPerFilePerWorker
has been deprecated in favour ofTTreeProcessorMT::SetTasksPerWorkerHint
.
Due to internal changes required to comply with the deprecation of Intel TBB's task_scheduler_init
and related
interfaces in recent TBB versions, as of v6.24 ROOT will not honor a maximum concurrency level set with
tbb::task_scheduler_init
but will require instead the usage of tbb::global_control
:
//tbb::task_scheduler_init init(2); // does not affect the number of threads ROOT will use anymore
tbb::global_control c(tbb::global_control::max_allowed_parallelism, 2);
ROOT::TThreadExecutor p1; // will use 2 threads
ROOT::TThreadExecutor p2(/*nThreads=*/8); // will still use 2 threads
Note that the preferred way to steer ROOT's concurrency level is still through
ROOT::EnableImplicitMT
or by passing the appropriate parameter to executors' constructors, as in
TThreadExecutor::TThreadExecutor
.
See the discussion at ROOT-11014 for more context.
A new way to set ROOT's "Dynamic Path" was added: the
environment variable ROOT_LIBRARY_PATH
. On Unix it should contain a colon
separated list of paths, on Windows a semicolon separated list. It is
intended to be cross platform and to be specific to ROOT (and thus not
interfere with the system's shared linker).
The final "Dynamic Path" is now composed of these sources in order:
ROOT_LIBRARY_PATH
environment variable- System specific shared linker environment variables like
LD_LIBRARY_PATH
,LIBPATH
, orPATH
. - Setting from rootrc
- ROOT's builtin library directory
- cling's LLVM is upgraded to version 9.0
- New interface to enable/disable optional cling features. Currently, it can be used to enable/disable support for redefinitions. See this issue for more information.
- Fix an uninitialized variable in global read-write lock which could have caused deadlocks or crashes in some rare cases.
- Default global read-write lock transitioned to new implementation based on TBB thread local storage when TBB is available on supported platforms (all except Windows). This gives an O(10%) performance improvement for some typical RDataFrame scenarios with 256 threads due to reduced lock contention.
- Exclusive use of the global lock is reduced or migrated to finer grained read and write locks in a few hotspots that occur during file opening/closing or task initialization in RDataFrame. This can lead to O(100x) improvements for some typical RDataFrame scenarios with 256 threads due to massively reduced lock contention.
TTree
now supports the inclusion of leaves of typeslong
andunsigned long
(and therefore alsostd::size_t
on most systems) also for branches in "leaflist mode". The corresponding leaflist letters are 'G' and 'g'.- when looping over a
TTree
with a friend with a larger number of entries,TTreeReader
now ends the event loop when the entries in the mainTTree
are exhausted, consistently with other interfaces. See #6518 for more details. TTreeProcessorMT::SetMaxTasksPerFilePerWorker
is now deprecated in favor of the more flexible and newly introducedTTreeProcessorMT::SetTasksPerWorkerHint
. See the relevant entries in our reference guide for more information.- The name of the sub-branches of a split collection no longer have 2 consecutive dots if the top level branche name has a trailing dot. The name of the collection's index leaf also no longer include the dot. For example for "t." the names where "t." and "t..fValue" and are now "t" and "t.fValue".
- Introduce
ROOT::RDF::RunGraphs
, which allows to compute the results of multipleRDataFrame
s (or better, multiple independent computation graphs) concurrently while sharing the same thread pool. The computation may be more efficient than running theRDataFrame
s sequentially if an analysis consists of several computation graphs that individually do not fully utilize the available resources. See e.g. this tutorial for an example usage. RDataFrame
now supports reading friendTTree
s with aTTreeIndex
, aka "indexed friends". More details at ROOT-9559.- Experimental logging capabilities have been added to
RDataFrame
. To activate logging, define the following variable before creating theRDataFrame
object:auto verbosity = ROOT::Experimental::RLogScopedVerbosity(ROOT::Detail::RDF::RDFLogChannel(), ROOT::Experimental::ELogLevel.kInfo);
. - With ROOT-10023 fixed,
RDataFrame
can now read and write certain branches containing unsplit objects, i.e.TBranchObjects
. More information is available at ROOT-10022. - CSV files can now be opened and processed from remote HTTP(S) locations
RDataFrame
results produced by the same action in different computation graphs can now be merged thanks to the new interface provided byROOT::Detail::RDF::RMergeableValue
, introduced in #5552. A feature originally requested with ROOT-9869, it helps streamlineRDataFrame
workflows in a distributed environment. Currently only a subset ofRDataFrame
actions have their corresponding mergeable class, but in the future it will be possible to extend it to any action through the creation of a newRMergeableValue
derived class.
Snapshot
now respects the basket size and split level of the original branch when copying branches to a newTTree
.Snapshot
now writes branches coming from friendTTree
s even if they have the same name as branches in the main tree (friendname_
is prepended to the name of the output branches). More details at #7181.- Just-in-time compilation of string expressions passed to
Filter
andDefine
now generates functions that take fundamental types by const value (rather than by non-const reference as before). This will break code that was assigning to column values in string expressions: this is an intended side effect as we want to prevent non-expert users from performing assignments (=
) rather than comparisons (==
). Expert users can resort to compiled callables if they absolutely have to assign to column values (not recommended). See ROOT-11009 for further discussion. - For some
TTrees
,RDataFrame::GetColumnNames
might now returns multiple valid spellings for a given column. For example, leaf"l"
under branch"b"
might now be mentioned as"l"
as well as"b.l"
, while only one of the two spellings might have been recognized before. - Certain RDF-related types in the
ROOT::Detail
andROOT::Internal
namespaces have been renamed, most notablyRCustomColumn
is nowRDefine
. This does not impact code that only makes use of entities in the public ROOT namespace, and should not impact downstream code unless it was patching or reusing internalRDataFrame
types.
- A critical issue has been fixed that could potentially result in wrong data being silently read in multi-thread runs when an input
TChain
contained more than oneTTree
coming from the same input file. More details are available at #7143. - The start-up time of event loops with large computation graphs with many just-in-time-compiled expressions (e.g. thousands of string
Filter
s andDefine
s) has been greatly reduced. See the corresponding pull request for more details.
The full list of bug fixes for this release is available below.
ROOT 6.24 introduces ROOT.RDF.Experimental.Distributed
, an experimental python package that enhances RDataFrame with distributed computing capabilities. The new package allows distributing RDataFrame applications through one of the supported distributed backends. The package was designed so that different backends can be easily plugged in. Currently the Apache Spark backend is supported and support for Dask is coming soon. The backend submodules of this package expose their own RDataFrame
objects. The only needed change in user code is to substitute ROOT.RDataFrame
calls with such backend-specific RDataFrame
s. For example:
import ROOT
# Point RDataFrame calls to the Spark specific RDataFrame
RDataFrame = ROOT.RDF.Experimental.Distributed.Spark.RDataFrame
# It still accepts the same constructor arguments as traditional RDataFrame
df = RDataFrame("mytree","myfile.root")
# Continue the application with the traditional RDataFrame API
The main goal of this package is to support running any RDataFrame application distributedly. Nonetheless, not all RDataFrame operations currently work with this package. The subset that is currently available is:
- AsNumpy
- Count
- Define
- Fill
- Filter
- Graph
- Histo[1,2,3]D
- Max
- Mean
- Min
- Profile[1,2,3]D
- Snapshot
- Sum
with support for more operations coming in the future.
Any distributed RDataFrame backend inherits the dependencies of the underlying software needed to distribute the applications. The Spark backend for example has the following runtime dependencies (ROOT will build just fine without, but the feature will be unavailable without these packages):
Tests for the Spark backend can be turned ON/OFF with the new build option test_distrdf_pyspark
(OFF by default).
- Update the definitions of the physical constants using the recommended 2018 values from NIST.
- Use also the new SI definition of base units from 2019, where the Planck constant, the Boltzmann constant, the elementary electric charge and the Avogadro constant are exact numerical values. See https://en.wikipedia.org/wiki/2019_redefinition_of_the_SI_base_units. Note that with this new definition the functions
TMath::HUncertainty()
,TMath::KUncertainty()
,TMath::QeUncertainty()
andTMath::NaUncertainty()
all return a0.0
value. - Due to some planned major improvements to
RVec
, the layout ofRVec
objects will change in a backward-incompatible way between v6.24 and v6.26. Because of this, we now print a warning if an application is reading or writing aROOT::RVec
object from/to a ROOT file. We assume this is an exceedingly rare case, as the ROOT interface typically used to manipulateRVec
s isRDataFrame
, andRDataFrame
performs an on-the-flyRVec <-> std::vector
conversion rather than writingRVec
s to disk. Note that, currently,RVecs
written e.g. in aTTree
cannot be read back using certain ROOT interfaces (e.g.TTreeReaderArray
,RDataFrame
and the experimentalRNTuple
). All these limitations will be lifted in v6.26. - Portable implementation of the RANLUX++ generator, see RanluxppEngine and our blog post.
- Introducing TMVA PyTorch Interface, a method to use PyTorch internally with TMVA for deep learning. This can be used as an alternative to PyKeras Interface for complex models providing more flexibility and power.
- Extension / updates of the doxygen reference guide.
- Allow for removing RooPlot from global directory management, see RooPlot::AddDirectory and RooPlot::SetDirectory
- Hash-assisted finding of elements in RooWorkspace. Large RooWorkspace objects were slow in finding elements. This was improved using a hash map.
- Stabilise RooStats::HypoTestInverter. It can now tolerate a few failed fits when conducting hypothesis tests. This is relevant when a few points in a parameter scan don't converge due to numerical or model instabilities. These points will be skipped, and HypoTestInverter can continue.
- Tweak pull / residual plots. ROOT automatically zoomed out a bit when a pull / residual plot is created. Now, the axis range of the original plot is transferred to the residual plot, so the pulls can be drawn below the main plot.
RooFit's BatchMode
has been around
since ROOT 6.20, but to
fully use vector extensions of modern CPUs, a manual compilation of ROOT was necessary, setting the required compiler flags.
Now, RooFit comes with dedicated computation libraries, each compiled for a specific CPU architecture. When RooFit is loaded for the first time, ROOT inspects the CPU capabilities, and loads the fastest supported version of this computation library. This means that RooFit can now use vector extensions such as AVX2 without being recompiled, which enables a speed up of up to 4x for certain computations. Combined with better data access patterns (~3x speed up, ROOT 6.20), computations with optimised PDFs speed up between 4x and 16x.
The fast BatchMode
now also works in combination with multi processing (NumCPU
) and with binned data (RooDataHist
).
See Demo notebook in SWAN, EPJ Web Conf. 245 (2020) 06007, arxiv:2012.02746.
The library that contains the optimised computation functions is called RooBatchCompute
. The PDFs contained in this library are highly optimized, and there is currently work in progress for further optimization using CUDA and multi-threaded computations. If you use PDFs that are not part of the official RooFit, you are very well invited to add them to RooFit by submitting a ticket or a pull request.
For PDFs that are not part of RooFit, it is possible to benefit from batch computations without vector extensions. To do so, consult the RooBatchCompute readme.
In case you have created a custom PDF which overrides evaluateBatch()
, please follow these steps to update your code to the newest version:
- Change the signature of the function both in the source and header file:
- RooSpan<double> RooGaussian::evaluateBatch(std::size_t begin, std::size_t batchSize) const
+ RooSpan<double> evaluateSpan(RooBatchCompute::RunContext& evalData, const RooArgSet* normSet) const
- Include
RunContext.h
andBracketAdapter.h
. - Use
getValues()
instead ofgetValBatch()
to retrieve a RooSpan for the data of every value.
- auto xData = x.getValBatch(begin, batchSize);
+ auto xData = x->getValues(evalData,normSet);
- Retrieve the number of events by getting the maximum size of the input spans.
size_t nEvents=0;
for (auto& i:{xData,meanData,sigmaData})
nEvents = std::max(nEvents,i.size());
- Create the output batch by calling
RunContext::makeBatch()
- auto output = _batchData.makeWritableBatchUnInit(begin, batchSize);
+ auto output = evalData.makeBatch(this, nEvents);
- DO NOT use
RooSpan::isBatch()
andRooSpan::empty()
methods! Instead, distinguish between scalar (RooSpan of size 1) and vector (RooSpan of size>1) parameters as shown below.
- const bool batchX = !xData.empty();
+ const bool batchX = xData.size()>1;
- Append
RooBatchCompute::
to the classes that have been moved to the RooBatchCompute Library:RooSpan
,BracketAdapterWithMask
,BracketAdapter
,RunContext
. Alternatively, you can write
using namespace RooBatchCompute;
- Replace
_rf_fast_<function>
withRooBatchCompute::fast_<function>
and includeRooVDTHeaders.h
(if applicable).
- output[i] = _rf_fast_exp(arg*arg * halfBySigmaSq);
+ output[i] = RooBatchCompute::fast_exp(arg*arg * halfBySigmaSq);
When RooFit performs binned fits, it takes the probability density at the bin centre as a proxy for the probability in the bin. This can lead to a bias. To alleviate this, the new class RooBinSamplingPdf has been added to RooFit. Also see arxiv:2012.02746.
When making residual or pull distributions with RooPlot::residHist
or RooPlot::pullHist
, the histogram is now compared with the curve's average values within a given bin by default, ensuring that residual and pull distributions are valid for strongly curved distributions.
The old default behaviour was to interpolate the curve at the bin centres, which can still be enabled by setting the useAverage
parameter of RooPlot::residHist
or RooPlot::pullHist
to false
.
When a function in RooFit is undefined (Poisson with negative mean, PDF with negative values, etc), RooFit can now pass information about the "badness" of the violation to the minimiser. The minimiser can use this to compute a gradient to find its way out of the undefined region. This can drastically improve its ability to recover when unstable fit models are used, for example RooPolynomial.
For details, see the RooFit tutorial rf612_recoverFromInvalidParameters.C and arxiv:2012.02746.
RooDataHist was partially modernised to improve const-correctness, to reduce side effects as well as its memory footprint, and to make
it ready for RooFit's faster batch evaluations.
Derived classes that directly access protected members might need to be updated. This holds especially for direct accesses to _curWeight
,
_curWeightErrLo
, etc, which have been removed. (It doesn't make sense to write to these members from const functions when the same information
can be retrieved using an index access operator of an array.) All similar accesses in derived classes should be replaced by the getters get_curWeight()
or better get_wgt(i)
, which were also supported in ROOT <v6.24. More details on what happened:
- Reduced side effects. This code produces undefined behaviour because the side effect of
get(i)
, i.e., loading the new weight into_curWeight
is not guaranteed to happen beforeweight()
is called:
processEvent(dataHist.get(i), dataHist.weight()); // Dangerous! Order of evaluation is not guaranteed.
With the modernised interface, one would use:
processEvent(dataHist.get(i), dataHist.weight(i));
To modernise old code, one should replace patterns like h.get(i); h.func()
by h.func(i);
. One may #define R__SUGGEST_NEW_INTERFACE
to switch on
deprecation warnings for the functions in question.
Similarly, the bin content can now be set using an index, making prior loading of a certain coordinate unnecessary:
for (int i=0 ; i<hist->numEntries() ; i++) {
- hist->get(i) ;
- hist->set(hist->weight() / sum);
+ hist->set(i, hist->weight(i) / sum, 0.);
}
- More const correctness.
calcTreeIndex()
doesn't rely on side effects, any more. Instead of overwriting the internal coordinates with new values:
// In a RooDataHist subclass:
_vars = externalCoordinates;
auto index = calcTreeIndex();
// Or from the outside:
auto index = dataHist.getIndex(externalCoordinates); // Side effect: Active bin is now `index`.
coordinates are now passed into calcTreeIndex without side effects:
// In a subclass:
auto index = calcTreeIndex(externalCoordinates, fast=<true/false>); // No side effect
// From the outside:
auto index = dataHist.getIndex(externalCoordinates); // No side effect
This will allow for marking more functions const, or for lying less about const correctness.
- RooDataHist now supports fits with RooFit's faster
BatchMode()
. - Lower memory footprint. If weight errors are not needed, RooDataHist now allocates only 40% of the memory that the old implementation used.
The public member function RooDataHist::sum()
has three overloads.
Two of these overloads accept a sumSet
parameter to not sum over all variables.
These two overloads previously behaved inconsistently when the correctForBinSize
or inverseBinCor
flags were set.
If you use the RooDataHist::sum()
function in you own classes, please check that it can still be used with its new logic.
The new and corrected bin correction behaviour is:
correctForBinSize
: multiply counts in each bin by the bin volume corresponding to the variables insumSet
inverseBinCor
: divide counts in each bin by the bin volume corresponding to the variables not insumSet
So far, the Crystal Ball distribution has been represented in RooFit only by the RooCBShape
class, which has a Gaussian core and a single power-law tail on one side.
This release introduces RooCrystalBall
, which implements some common generalizations of the Crystal Ball shape:
- symmetric or asymmetric power-law tails on both sides
- different width parameters for the left and right sides of the Gaussian core
The new RooCrystalBall
class can substitute the RooDSCBShape
and RooSDSCBShape
, which were passed around in the community.
- Add the method
AddPoint
toTGraph(x,y)
andTGraph2D(x,y,z)
, equivalent toSetPoint(g->GetN(),x,y)
andSetPoint(g->GetN(),x,y,z)
- Option
E0
draws error bars and markers are drawn for bins with 0 contents. Now, combined with options E1 and E2, it avoids error bars clipping.
Now when THttpServer creates FastCGI engine, 10 worker threads used to process requests received via FastCGI channel. This significantly increase a performance, especially when several clients are connected.
If THttpServer created for use with webgui widgets (RBrowser, RCanvas, REve), it only will provide access to the widgets via websocket connection - any other kind of requests like root.json or exe.json will be refused completely. Combined with connection tokens and https protocol, this makes usage of webgui components in public networks more secure.
Bearer tokens are part of WLCG capability-based infrastructure with capability-based scheme which uses an infrastructure that describes what the bearer is allowed to do as opposed to who that bearer is. Token discovery procedure are developed according WLCG Bearer Token Discovery specification document (https://github.com/WLCG-AuthZ-WG/bearer-token-discovery/blob/master/specification.md). Short overview:
- If the
BEARER_TOKEN
environment variable is set, then the value is taken to be the token contents. - If the
BEARER_TOKEN_FILE
environment variable is set, then its value is interpreted as a filename. The contents of the specified file are taken to be the token contents. - If the
XDG_RUNTIME_DIR
environment variable is set, then take the token from the contents of$XDG_RUNTIME_DIR/bt_u$ID
(this additional location is intended to provide improved security for shared login environments as$XDG_RUNTIME_DIR
is defined to be user-specific as opposed to a system-wide directory.). - Otherwise, take the token from
/tmp/bt_u$ID
.
- central factory methods to handle browsing, editing and drawing of different classes
- simple possibility to extend RBrowser on user-defined classes
- support of web-based geometry viewer
- better support of TTree drawing
- server-side handling of code editor and image viewer widgets
- rbrowser content is fully recovered when web-browser is reloaded
- load of widgets code only when really required (shorter startup time for RBrowser)
- update all used libraries
d3.js
,three.js
,MathJax.js
, openui5 - change to Promise based interface for all async methods, remove call-back arguments
- change scripts names, core scripts name now
JSRoot.core.js
- unify function/methods naming conventions, many changes in method names
- provide central code loader via
JSROOT.require
, supporting 4 different loading engines - many nice features and many bug fixes; see JSROOT v6 release notes
- a new cmake variable,
CMAKE_INSTALL_PYTHONDIR
, has been added: it allows customization of the installation directory of ROOT's python modules - the developer build option
asserts
is introduced to enable/disable asserts via theNDEBUG
C/CXX flag. Asserts are always enabled forCMAKE_BUILD_TYPE=Debug
anddev=ON
. The previous behavior of the builds set via theCMAKE_BUILD_TYPE
variable has not changed. CMAKE_CXX_STANDARD
, i.e. the C++ standard ROOT is built with, now defaults to the compiler default (or C++11 if the compiler default is older than that) rather than always defaulting to C++11. In turn this means that v6.24 is the first ROOT release for which ROOT's pre-compiled binaries are not compiled with C++11 but with the default standard in use by the default system compiler. On Ubuntu 20.04, for example, the v6.24 pre-compiled binaries are now compiled with C++14 rather than C++11 as it happened for previous ROOT versions. Also see ROOT-10692.
The following builtins have been updated:
- VecCore 0.7.0
- Deprecate
TTree.AsMatrix
in this release and mark for removal in v6.26. Please use insteadRDataFrame.AsNumpy
.