Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GSoC][TMVA][SOFIE] Serialisation of RModel #8666

Merged
merged 6 commits into from
Aug 10, 2021

Conversation

sanjibansg
Copy link
Contributor

@sanjibansg sanjibansg commented Jul 14, 2021

Description

This PR implements serialisation of RModel by adding a custom streamer in class RModel and modifying the definition of structure TMVA::Experimental::SOFIE::InitializedTensor.

Checklist:

  • tested changes locally

Tests

  • Tests for Serialisation of Custom Models

Example usage

Writing the RModel object

using namespace TMVA::Experimental;
TFile file("model.root","CREATE");
SOFIE::RModelParser_ONNX parser;
SOFIE::RModel model = parser.Parse("Linear_2.onnx");
model.Write("model");
file.Close();

Reading the RModel object

using namespace TMVA::Experimental;
TFile file("model.root","READ");
SOFIE::RModel *model;
file.GetObject("model",model);
file.Close();
model->Generate();
model->OutputGenerated("model.hxx");

@phsft-bot
Copy link
Collaborator

Can one of the admins verify this patch?

@jalopezg-git
Copy link
Collaborator

@phsft-bot build

@phsft-bot
Copy link
Collaborator

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

@phsft-bot
Copy link
Collaborator

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

  • [2021-07-14T09:35:30.879Z] stderr: error: could not read '.git/rebase-apply/head-name': No such file or directory

Failing tests:

@phsft-bot
Copy link
Collaborator

Build failed on windows10/cxx14.
Running on null:C:\build\workspace\root-pullrequests-build
See console output.

@bellenot
Copy link
Member

bellenot commented Jul 14, 2021

@sanjibansg please fork also roottest to prevent timeouts on the Windows nodes. Thanks

@lmoneta
Copy link
Member

lmoneta commented Jul 21, 2021

@sanjibansg : can you please rebase it to fix the conflict file , happening after merging your other PR on restructuring Sofie. Thank you

@sanjibansg
Copy link
Contributor Author

@sanjibansg : can you please rebase it to fix the conflict file , happening after merging your other PR on restructuring Sofie. Thank you

@lmoneta Resolved the merge conflict.

@lmoneta
Copy link
Member

lmoneta commented Aug 4, 2021

@phsft-bot build just on ROOT-ubuntu16/nortcxxmod with flags -Dtmva-sofie=On

@phsft-bot
Copy link
Collaborator

Starting build on ROOT-ubuntu16/nortcxxmod with flags -Dtmva-sofie=On
How to customize builds

@phsft-bot
Copy link
Collaborator

Build failed on ROOT-ubuntu16/nortcxxmod.
Running on sft-ubuntu-1604-1.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

  • [2021-08-04T15:13:35.602Z] CMake Error at /mnt/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1045 (message):

@lmoneta
Copy link
Member

lmoneta commented Aug 5, 2021

@phsft-bot build just on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On

@phsft-bot
Copy link
Collaborator

Starting build on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On
How to customize builds

@phsft-bot
Copy link
Collaborator

Build failed on ROOT-ubuntu2004/default.
Running on root-ubuntu-2004-3.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Errors:

  • [2021-08-05T11:58:02.761Z] CMake Error at /home/sftnight/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1045 (message):

Copy link
Member

@lmoneta lmoneta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a small comment on the names of member function and data members

tmva/sofie/inc/TMVA/SOFIE_common.hxx Outdated Show resolved Hide resolved
tmva/sofie/test/CMakeLists.txt Outdated Show resolved Hide resolved
@lmoneta
Copy link
Member

lmoneta commented Aug 6, 2021

@phsft-bot build just on ROOT-ubuntu18.04/default with flags -Dtmva-sofie=On

@phsft-bot
Copy link
Collaborator

Starting build on ROOT-ubuntu18.04/default with flags -Dtmva-sofie=On
How to customize builds

@phsft-bot
Copy link
Collaborator

Build failed on ROOT-ubuntu18.04/default.
Running on sft-ubuntu-1804-2.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

  • [2021-08-06T09:10:24.228Z] CMake Error at /mnt/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1045 (message):

@lmoneta
Copy link
Member

lmoneta commented Aug 10, 2021

@phsft-bot build just on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On

@phsft-bot
Copy link
Collaborator

Starting build on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On
How to customize builds

@phsft-bot
Copy link
Collaborator

Build failed on ROOT-ubuntu2004/default.
Running on root-ubuntu-2004-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Errors:

  • [2021-08-10T08:07:34.691Z] FAILED: tmva/sofie/test/CMakeFiles/emitFromONNX.dir///sofie_parsers/src/RModelParser_ONNX.cxx.o
  • [2021-08-10T08:07:35.415Z] /home/sftnight/build/workspace/root-pullrequests-build/root/tmva/sofie_parsers/src/RModelParser_ONNX.cxx:185:15: error: ‘class onnx::ModelProto’ has no member named ‘ParseFromIstream’; did you mean ‘ParseFromString’?
  • [2021-08-10T08:07:35.415Z] FAILED: tmva/sofie/test/CMakeFiles/emitFromROOT.dir///sofie_parsers/src/RModelParser_ONNX.cxx.o
  • [2021-08-10T08:07:36.011Z] /home/sftnight/build/workspace/root-pullrequests-build/root/tmva/sofie_parsers/src/RModelParser_ONNX.cxx:185:15: error: ‘class onnx::ModelProto’ has no member named ‘ParseFromIstream’; did you mean ‘ParseFromString’?

Warnings:

  • [2021-08-10T08:07:40.475Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 32] from the object at ‘onnx::TensorProto_Segment_default_instance’ is out of the bounds of referenced subobject ‘onnx::TensorProto_Segment::begin’ with type ‘long int’ at offset 16 [-Warray-bounds]
  • [2021-08-10T08:07:40.475Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Tensor_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Tensor::shape’ with type ‘onnx::TensorShapeProto*’ at offset 16 [-Warray-bounds]
  • [2021-08-10T08:07:40.475Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Map_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Map::value_type’ with type ‘onnx::TypeProto*’ at offset 16 [-Warray-bounds]
  • [2021-08-10T08:07:40.475Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [49, 56] from the object at ‘onnx::SparseTensorProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::SparseTensorProto::values’ with type ‘onnx::TensorProto*’ at offset 40 [-Warray-bounds]
  • [2021-08-10T08:07:40.475Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [73, 80] from the object at ‘onnx::TrainingInfoProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::TrainingInfoProto::initialization’ with type ‘onnx::GraphProto*’ at offset 64 [-Warray-bounds]
  • [2021-08-10T08:07:40.475Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 32] from the object at ‘onnx::TensorProto_Segment_default_instance’ is out of the bounds of referenced subobject ‘onnx::TensorProto_Segment::begin’ with type ‘long int’ at offset 16 [-Warray-bounds]
  • [2021-08-10T08:07:40.475Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Tensor_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Tensor::shape’ with type ‘onnx::TensorShapeProto*’ at offset 16 [-Warray-bounds]
  • [2021-08-10T08:07:40.475Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Map_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Map::value_type’ with type ‘onnx::TypeProto*’ at offset 16 [-Warray-bounds]
  • [2021-08-10T08:07:40.476Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [49, 56] from the object at ‘onnx::SparseTensorProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::SparseTensorProto::values’ with type ‘onnx::TensorProto*’ at offset 40 [-Warray-bounds]
  • [2021-08-10T08:07:40.476Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [73, 80] from the object at ‘onnx::TrainingInfoProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::TrainingInfoProto::initialization’ with type ‘onnx::GraphProto*’ at offset 64 [-Warray-bounds]

And 1 more

@lmoneta
Copy link
Member

lmoneta commented Aug 10, 2021

@phsft-bot build just on ROOT-ubuntu18.04/default with flags -Dtmva-sofie=On

@phsft-bot
Copy link
Collaborator

Starting build on ROOT-ubuntu18.04/default with flags -Dtmva-sofie=On
How to customize builds

@phsft-bot
Copy link
Collaborator

Build failed on ROOT-ubuntu18.04/default.
Running on sft-ubuntu-1804-2.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Errors:

  • [2021-08-10T10:26:16.753Z] FAILED: tmva/sofie_parsers/CMakeFiles/ROOTTMVASofieParser.dir/src/RModelParser_ONNX.cxx.o
  • [2021-08-10T10:26:17.011Z] /mnt/build/workspace/root-pullrequests-build/root/tmva/sofie_parsers/src/RModelParser_ONNX.cxx:185:15: error: ‘class onnx::ModelProto’ has no member named ‘ParseFromIstream’; did you mean ‘ParseFromString’?

Warnings:

  • [2021-08-10T10:26:14.987Z] Warning in <CheckModuleValid>: warning: Couldn't find in the following specified headers in the module ROOTTMVASofieParser:

@lmoneta
Copy link
Member

lmoneta commented Aug 10, 2021

@phsft-bot build just on ROOT-ubuntu18.04/default, ROOT-ubuntu2004/default with flags -Dtmva-sofie=On

@phsft-bot
Copy link
Collaborator

Starting build on ROOT-ubuntu18.04/default, ROOT-ubuntu2004/default with flags -Dtmva-sofie=On
How to customize builds

@phsft-bot
Copy link
Collaborator

Build failed on ROOT-ubuntu2004/default.
Running on root-ubuntu-2004-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Warnings:

  • [2021-08-10T11:25:00.049Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 32] from the object at ‘onnx::TensorProto_Segment_default_instance’ is out of the bounds of referenced subobject ‘onnx::TensorProto_Segment::begin’ with type ‘long int’ at offset 16 [-Warray-bounds]
  • [2021-08-10T11:25:00.049Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Tensor_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Tensor::shape’ with type ‘onnx::TensorShapeProto*’ at offset 16 [-Warray-bounds]
  • [2021-08-10T11:25:00.049Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Map_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Map::value_type’ with type ‘onnx::TypeProto*’ at offset 16 [-Warray-bounds]
  • [2021-08-10T11:25:00.049Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [49, 56] from the object at ‘onnx::SparseTensorProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::SparseTensorProto::values’ with type ‘onnx::TensorProto*’ at offset 40 [-Warray-bounds]
  • [2021-08-10T11:25:00.308Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [73, 80] from the object at ‘onnx::TrainingInfoProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::TrainingInfoProto::initialization’ with type ‘onnx::GraphProto*’ at offset 64 [-Warray-bounds]
  • [2021-08-10T11:25:00.308Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 32] from the object at ‘onnx::TensorProto_Segment_default_instance’ is out of the bounds of referenced subobject ‘onnx::TensorProto_Segment::begin’ with type ‘long int’ at offset 16 [-Warray-bounds]
  • [2021-08-10T11:25:00.308Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Tensor_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Tensor::shape’ with type ‘onnx::TensorShapeProto*’ at offset 16 [-Warray-bounds]
  • [2021-08-10T11:25:00.308Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Map_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Map::value_type’ with type ‘onnx::TypeProto*’ at offset 16 [-Warray-bounds]
  • [2021-08-10T11:25:00.308Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [49, 56] from the object at ‘onnx::SparseTensorProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::SparseTensorProto::values’ with type ‘onnx::TensorProto*’ at offset 40 [-Warray-bounds]
  • [2021-08-10T11:25:00.308Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [73, 80] from the object at ‘onnx::TrainingInfoProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::TrainingInfoProto::initialization’ with type ‘onnx::GraphProto*’ at offset 64 [-Warray-bounds]

@phsft-bot
Copy link
Collaborator

Build failed on ROOT-ubuntu18.04/default.
Running on sft-ubuntu-1804-2.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Warnings:

  • [2021-08-10T11:16:04.803Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:1295:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]
  • [2021-08-10T11:16:04.803Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:2422:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]
  • [2021-08-10T11:16:04.803Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:3010:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]
  • [2021-08-10T11:16:04.803Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:3757:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]
  • [2021-08-10T11:16:05.060Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:4417:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]
  • [2021-08-10T11:16:05.060Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:5190:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]
  • [2021-08-10T11:16:05.060Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:5566:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]
  • [2021-08-10T11:16:05.060Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:6099:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]
  • [2021-08-10T11:16:05.060Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:6884:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]
  • [2021-08-10T11:16:05.060Z] /mnt/build/workspace/root-pullrequests-build/build/tmva/sofie_parsers/onnx_proto3.pb.cc:7483:10: warning: unused parameter ‘deterministic’ [-Wunused-parameter]

And 44 more

Copy link
Member

@lmoneta lmoneta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Warnings are caused by using an old protobuf version in the build nodes. When using protobuf version 3.17 they should be fixed.
The PR can then be merged.
Thank you @sanjibansg for this contribution!

@lmoneta lmoneta merged commit 0bd9252 into root-project:master Aug 10, 2021
pzhristov pushed a commit to alisw/root that referenced this pull request Aug 27, 2021
* Initial commit for serialisation of RModel

* Modify: Casting of shared_ptr to persistent pointer and vice-versa

* Add: Tests for serialisation of RModel

* Resolve: Merge Conflict

* Modify: Use CMAKE_BINARY_DIRECTORY/ginclude instead of /include

* Modify: Names of Data members & member functions in struct InitializedTensor
@sanjibansg sanjibansg deleted the serialisation branch September 5, 2021 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants