[PyROOT exp] Introduce MakeNumpyDataFrame to read numpy arrays with RDF #3669

stwunsch · 2019-04-10T07:58:47Z

This PR supersedes #3424.

The reference counting is greatly improved and the data is kept alive until the datasource dies, which gets delete at the end of the lifetime of the computational graph.

See here for the use-case:

import ROOT
import numpy

data = {
    "x": numpy.array([1, 2, 3]),
    "y": numpy.array([4, 5, 6])
}

df = ROOT.ROOT.RDF.MakeNumpyDataFrame(data)
df = df.Define("z", "x + y")

print(df.Mean("z").GetValue()) # Returns 7.0

The feature plays well along with the RDataFrame.AsNumpy feature:

import ROOT

df = ROOT.ROOT.RDataFrame(10).Define("x", "(int)rdfentry_")
data = df.AsNumpy()

df2 = ROOT.ROOT.RDF.MakeNumpyDataFrame(data)
df2.Snapshot("tree", "file.root")

TODO:

Figure out how to install the header needed for the NumyDataSource
How to call the header (current name MakeNumpyDataFrame.hxx)? We should put it in a scope.

phsft-bot · 2019-04-10T07:59:02Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, ROOT-ubuntu18.04-i386/cxx14, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-04-10T08:13:22Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, ROOT-ubuntu18.04-i386/cxx14, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-04-10T08:40:29Z

Build failed on mac1014/cxx17.
See console output.

Failing tests:

And 463 more

stwunsch · 2019-04-10T12:35:19Z

@phsft-bot build with -Dpyroot_experimental=ON

phsft-bot · 2019-04-10T12:35:25Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, ROOT-ubuntu18.04-i386/cxx14, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-04-10T22:50:47Z

Build failed on ROOT-ubuntu18.04-i386/cxx14.
See console output.

Failing tests:

dpiparo

this PR is great. It allows to close the circle between cpp and py. I mark this as "Request Changes" because I think it requires some more discussion, especially the factory function part. Other than that, well done @stwunsch !!

dpiparo · 2019-04-11T10:52:31Z

bindings/pyroot_experimental/PyROOT/python/ROOT/pythonization/_rdataframe.py

@@ -96,3 +97,7 @@ def pythonize_rdataframe(klass, name):
        klass.AsNumpy = RDataFrameAsNumpy

    return True
+
+# Add MakeNumpyDataFrame feature as free function to the ROOT module


why can't this be a regular binding to a C++ factory function?

Because the input is a PyObject* and we need to process it with ROOT.AsRVec internally, we need a free function in the module.

dpiparo · 2019-04-11T10:52:54Z

bindings/pyroot_experimental/PyROOT/src/MakeNumpyDataFrame.hxx

+   }
+
+protected:
+   std::string AsString() { return "RVec data source"; };


maybe a better name can be found?

Thanks, gonna fix this!

dpiparo · 2019-04-11T10:54:19Z

bindings/pyroot_experimental/PyROOT/src/MakeNumpyDataFrame.hxx

+// Note that we have to return the object on the heap so that the interpreter
+// does not clean it up during shutdown and causes a double delete.
+template <typename... ColumnTypes>
+RDataFrame* MakeNumpyDataFrame(PyObject* pyRVecs,


this one would be an unicum: we do not write C++ code which depends on Python, unless this is a pythonisation. Is there any way we can exploit to write a proper cpp function and then use pyroot bindings to it?

We need to call a Py_DECREF in the destructor of the data-source, which dictates the lifetime of a dataframe graph. I do not see any other way to make the refcount properly than having this custom datasource. However, the compliation (or dependecy) is at runtime since the source is jitted.

phsft-bot · 2019-04-12T12:55:22Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, ROOT-ubuntu18.04-i386/cxx14, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-04-12T13:01:22Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, ROOT-ubuntu18.04-i386/cxx14, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-04-12T13:03:24Z

Build failed on mac1014/cxx17.
See console output.

Errors:

CMake Error at /build/jenkins/workspace/root-pullrequests-build_2/build/CMakeFiles/CMakeTmp/CMakeLists.txt:15 (add_executable):
CMake Error at /build/jenkins/workspace/root-pullrequests-build_2/build/CMakeFiles/CMakeTmp/CMakeLists.txt:15 (add_executable):
CMake Error at /usr/local/Cellar/cmake/3.13.4/share/cmake/Modules/CheckSymbolExists.cmake:90 (try_compile):
CMake Error at /build/jenkins/workspace/root-pullrequests-build_2/rootspi/jenkins/root-build.cmake:844 (message):

phsft-bot · 2019-04-12T13:33:16Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, ROOT-ubuntu18.04-i386/cxx14, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-04-13T00:59:58Z

Build failed on ROOT-ubuntu18.04-i386/cxx14.
See console output.

Failing tests:

phsft-bot · 2019-04-17T08:04:23Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-04-17T08:08:37Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-04-17T09:18:32Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

etejedor

Looks great! Thanks @stwunsch !

phsft-bot · 2019-04-17T09:36:35Z

Build failed on ROOT-fedora27/noimt.
See console output.

Failing tests:

projectroot.test.test_stressgraphics_interpreted

phsft-bot · 2019-04-17T09:50:54Z

Starting build on ROOT-performance-centos7-multicore/default, ROOT-fedora27/noimt, ROOT-fedora29/python3, ROOT-ubuntu16/rtcxxmod, mac1014/cxx17, windows10/default
How to customize builds

phsft-bot · 2019-04-19T04:32:52Z

Build failed on windows10/default.
See console output.

stwunsch requested review from amadio and etejedor as code owners April 10, 2019 07:58

stwunsch mentioned this pull request Apr 10, 2019

[PyROOT exp] Add MakeNumpyDataFrame feature #3424

Closed

stwunsch self-assigned this Apr 10, 2019

stwunsch added the needs review label Apr 10, 2019

dpiparo reviewed Apr 11, 2019

View reviewed changes

stwunsch force-pushed the impl_rdfnumpy2 branch from 9fabc91 to bcefa3d Compare April 12, 2019 12:53

stwunsch force-pushed the impl_rdfnumpy2 branch from ce55f9e to 154c6d5 Compare April 12, 2019 13:31

stwunsch force-pushed the impl_rdfnumpy2 branch from 154c6d5 to 520c0bc Compare April 17, 2019 08:04

stwunsch force-pushed the impl_rdfnumpy2 branch from 520c0bc to 86a0563 Compare April 17, 2019 08:08

stwunsch force-pushed the impl_rdfnumpy2 branch from 86a0563 to edd5b9b Compare April 17, 2019 09:18

etejedor approved these changes Apr 17, 2019

View reviewed changes

[PyROOT exp] Introduce MakeNumpyDataFrame to read numpy arrays with RDF

23102dd

stwunsch force-pushed the impl_rdfnumpy2 branch from edd5b9b to 23102dd Compare April 17, 2019 09:50

stwunsch merged commit 989619c into root-project:master Apr 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PyROOT exp] Introduce MakeNumpyDataFrame to read numpy arrays with RDF #3669

[PyROOT exp] Introduce MakeNumpyDataFrame to read numpy arrays with RDF #3669

stwunsch commented Apr 10, 2019 •

edited

phsft-bot commented Apr 10, 2019

phsft-bot commented Apr 10, 2019

phsft-bot commented Apr 10, 2019

stwunsch commented Apr 10, 2019

phsft-bot commented Apr 10, 2019

phsft-bot commented Apr 10, 2019

dpiparo left a comment

dpiparo Apr 11, 2019

stwunsch Apr 12, 2019

dpiparo Apr 11, 2019

stwunsch Apr 12, 2019

dpiparo Apr 11, 2019

stwunsch Apr 12, 2019

phsft-bot commented Apr 12, 2019

phsft-bot commented Apr 12, 2019

phsft-bot commented Apr 12, 2019

phsft-bot commented Apr 12, 2019

phsft-bot commented Apr 13, 2019

phsft-bot commented Apr 17, 2019

phsft-bot commented Apr 17, 2019

phsft-bot commented Apr 17, 2019

etejedor left a comment

phsft-bot commented Apr 17, 2019

phsft-bot commented Apr 17, 2019

phsft-bot commented Apr 19, 2019

[PyROOT exp] Introduce MakeNumpyDataFrame to read numpy arrays with RDF #3669

[PyROOT exp] Introduce MakeNumpyDataFrame to read numpy arrays with RDF #3669

Conversation

stwunsch commented Apr 10, 2019 • edited

phsft-bot commented Apr 10, 2019

phsft-bot commented Apr 10, 2019

phsft-bot commented Apr 10, 2019

Failing tests:

stwunsch commented Apr 10, 2019

phsft-bot commented Apr 10, 2019

phsft-bot commented Apr 10, 2019

Failing tests:

dpiparo left a comment

Choose a reason for hiding this comment

dpiparo Apr 11, 2019

Choose a reason for hiding this comment

stwunsch Apr 12, 2019

Choose a reason for hiding this comment

dpiparo Apr 11, 2019

Choose a reason for hiding this comment

stwunsch Apr 12, 2019

Choose a reason for hiding this comment

dpiparo Apr 11, 2019

Choose a reason for hiding this comment

stwunsch Apr 12, 2019

Choose a reason for hiding this comment

phsft-bot commented Apr 12, 2019

phsft-bot commented Apr 12, 2019

phsft-bot commented Apr 12, 2019

Errors:

phsft-bot commented Apr 12, 2019

phsft-bot commented Apr 13, 2019

Failing tests:

phsft-bot commented Apr 17, 2019

phsft-bot commented Apr 17, 2019

phsft-bot commented Apr 17, 2019

etejedor left a comment

Choose a reason for hiding this comment

phsft-bot commented Apr 17, 2019

Failing tests:

phsft-bot commented Apr 17, 2019

phsft-bot commented Apr 19, 2019

stwunsch commented Apr 10, 2019 •

edited