[GSoC][TMVA][SOFIE] PyTorch Parser TMVA #8684

sanjibansg · 2021-07-15T12:51:19Z

Description

Development of the functionality for parsing a PyTorch PT file to ROOT format using the Fast Inference System SOFIE for the ROOT Storage of Deep Learning Models Project of Google Summer of Code'21.

Project Link: https://summerofcode.withgoogle.com/projects/#5424575602491392

Progress

Parser function for extracting the model information and weights and instantiate a RModel Object
- Support for PyTorch nn.Module, nn.Sequential, nn.ModuleList containers.
- Supports Linear, ReLU and Transpose Layers/operations.
- Header file for the function
- Function implementation

Interface

The parser requires the shape and data type of the input tensors. Shape is a mandatory parameter, whereas the function defaults to Float data-type for the input tensors, if not explicitly mentioned.

Example usage

std::vector<size_t> s1{120,1};
std::vector<std::vector<size_t>> inputShape{s1};
auto model = TMVA::Experimental::SOFIE::PyTorch::Parse("trained_model_dense.pt",inputShape);

Checklist:

tested changes locally
- Parser

Tests

Emit Files for generating header files
Tests for Parser
CMake Configuration for tests

Documentation

Doxygen Docs

phsft-bot · 2021-07-15T12:51:21Z

Can one of the admins verify this patch?

lmoneta · 2021-08-09T15:26:43Z

@phsft-bot build just on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On

phsft-bot · 2021-08-09T15:26:50Z

Starting build on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On
How to customize builds

phsft-bot · 2021-08-09T15:28:28Z

Build failed on ROOT-ubuntu2004/default.
Running on root-ubuntu-2004-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Errors:

[2021-08-09T15:28:27.613Z] CMake Error at /home/sftnight/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1045 (message):

lmoneta · 2021-08-16T13:50:20Z

@phsft-bot build just on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On

phsft-bot · 2021-08-16T13:50:25Z

Starting build on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On
How to customize builds

phsft-bot · 2021-08-16T13:50:39Z

Build failed on ROOT-ubuntu2004/default.
Running on root-ubuntu-2004-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Errors:

[2021-08-16T13:50:38.925Z] CMake Error at /home/sftnight/build/workspace/root-pullrequests-build/rootspi/jenkins/root-build.cmake:1045 (message):

lmoneta · 2021-08-17T09:31:46Z

@phsft-bot build just on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On

phsft-bot · 2021-08-17T09:31:52Z

Starting build on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On
How to customize builds

phsft-bot · 2021-08-17T10:24:54Z

Build failed on ROOT-ubuntu2004/default.
Running on root-ubuntu-2004-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Warnings:

[2021-08-17T10:04:09.426Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 32] from the object at ‘onnx::TensorProto_Segment_default_instance’ is out of the bounds of referenced subobject ‘onnx::TensorProto_Segment::begin’ with type ‘long int’ at offset 16 [-Warray-bounds]
[2021-08-17T10:04:09.426Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Tensor_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Tensor::shape’ with type ‘onnx::TensorShapeProto*’ at offset 16 [-Warray-bounds]
[2021-08-17T10:04:09.426Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Map_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Map::value_type’ with type ‘onnx::TypeProto*’ at offset 16 [-Warray-bounds]
[2021-08-17T10:04:09.426Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [49, 56] from the object at ‘onnx::SparseTensorProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::SparseTensorProto::values’ with type ‘onnx::TensorProto*’ at offset 40 [-Warray-bounds]
[2021-08-17T10:04:09.426Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [73, 80] from the object at ‘onnx::TrainingInfoProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::TrainingInfoProto::initialization’ with type ‘onnx::GraphProto*’ at offset 64 [-Warray-bounds]
[2021-08-17T10:04:09.696Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 32] from the object at ‘onnx::TensorProto_Segment_default_instance’ is out of the bounds of referenced subobject ‘onnx::TensorProto_Segment::begin’ with type ‘long int’ at offset 16 [-Warray-bounds]
[2021-08-17T10:04:09.696Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Tensor_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Tensor::shape’ with type ‘onnx::TensorShapeProto*’ at offset 16 [-Warray-bounds]
[2021-08-17T10:04:09.696Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Map_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Map::value_type’ with type ‘onnx::TypeProto*’ at offset 16 [-Warray-bounds]
[2021-08-17T10:04:09.696Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [49, 56] from the object at ‘onnx::SparseTensorProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::SparseTensorProto::values’ with type ‘onnx::TensorProto*’ at offset 40 [-Warray-bounds]
[2021-08-17T10:04:09.696Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [73, 80] from the object at ‘onnx::TrainingInfoProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::TrainingInfoProto::initialization’ with type ‘onnx::GraphProto*’ at offset 64 [-Warray-bounds]

lmoneta

Thank you @sanjibansg for adding the documentation.
The PR is ready to be merged after the Keras converter

lmoneta · 2021-08-30T09:36:39Z

@phsft-bot build just on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On

phsft-bot · 2021-08-30T09:36:44Z

Starting build on ROOT-ubuntu2004/default with flags -Dtmva-sofie=On
How to customize builds

phsft-bot · 2021-08-30T10:02:14Z

Build failed on ROOT-ubuntu2004/default.
Running on root-ubuntu-2004-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Warnings:

[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 32] from the object at ‘onnx::TensorProto_Segment_default_instance’ is out of the bounds of referenced subobject ‘onnx::TensorProto_Segment::begin’ with type ‘long int’ at offset 16 [-Warray-bounds]
[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Tensor_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Tensor::shape’ with type ‘onnx::TensorShapeProto*’ at offset 16 [-Warray-bounds]
[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Map_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Map::value_type’ with type ‘onnx::TypeProto*’ at offset 16 [-Warray-bounds]
[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [49, 56] from the object at ‘onnx::SparseTensorProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::SparseTensorProto::values’ with type ‘onnx::TensorProto*’ at offset 40 [-Warray-bounds]
[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [73, 80] from the object at ‘onnx::TrainingInfoProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::TrainingInfoProto::initialization’ with type ‘onnx::GraphProto*’ at offset 64 [-Warray-bounds]
[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 32] from the object at ‘onnx::TensorProto_Segment_default_instance’ is out of the bounds of referenced subobject ‘onnx::TensorProto_Segment::begin’ with type ‘long int’ at offset 16 [-Warray-bounds]
[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Tensor_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Tensor::shape’ with type ‘onnx::TensorShapeProto*’ at offset 16 [-Warray-bounds]
[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [25, 28] from the object at ‘onnx::TypeProto_Map_default_instance’ is out of the bounds of referenced subobject ‘onnx::TypeProto_Map::value_type’ with type ‘onnx::TypeProto*’ at offset 16 [-Warray-bounds]
[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [49, 56] from the object at ‘onnx::SparseTensorProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::SparseTensorProto::values’ with type ‘onnx::TensorProto*’ at offset 40 [-Warray-bounds]
[2021-08-30T09:41:34.085Z] /usr/include/x86_64-linux-gnu/bits/string_fortified.h:71:33: warning: ‘void* _builtin_memset(void*, int, long unsigned int)’ offset [73, 80] from the object at ‘onnx::TrainingInfoProto_default_instance’ is out of the bounds of referenced subobject ‘onnx::TrainingInfoProto::initialization’ with type ‘onnx::GraphProto*’ at offset 64 [-Warray-bounds]

lmoneta

The PR looks good, we need just to apply the same fixes done in the Keras converters

tmva/pymva/inc/TMVA/RModelParser_PyTorch.h

tmva/pymva/test/EmitFromPyTorch.cxx

1) Added support for Tensors of Dynamic Axes 2) Modified Map for DType for PyTorch

1) Fix : segmentation violation by removing the Py_XDECREF() 2) Remove : RModelParser_Common.h and moved helper functions to SOFIE_common & under namespace SOFIE::PyTorch 3) Modify : Casting and usage of PyArrayObject* (now a float* is used instead of RTensor) 4) Add : Runtime error catch if Python String is not successfully run 5) Modify : Minor improvements and modifications in code structure and naming conventions

1) Added GTests for RModelParser_PyTorch 2) Added EmitFromPyTorch to generate header files 3) Configured CMake for tests

1. Refactor RModelParser_PyTorch 2. Call to AddBlasRoutines() while adding Gemm Node

…elative path

lmoneta · 2021-08-30T21:56:11Z

@phsft-bot build!

phsft-bot · 2021-08-30T21:56:17Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, mac1014/python3, mac11.0/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2021-08-30T23:16:04Z

Build failed on mac11.0/cxx17.
Running on macphsft20.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

lmoneta

Thank you for the changes. The PR is now ready to be merged

jalopezg-git assigned lmoneta and sitongan Jul 15, 2021

jalopezg-git added improvement in:TMVA labels Jul 15, 2021

sanjibansg force-pushed the pytorch branch from 350db07 to 27d6f20 Compare August 3, 2021 11:00

sanjibansg force-pushed the pytorch branch from 7f34357 to eaab7d4 Compare August 12, 2021 08:17

sanjibansg marked this pull request as ready for review August 12, 2021 08:28

sanjibansg requested review from ashlaban, bellenot, lmoneta, sitongan and stwunsch as code owners August 12, 2021 08:28

sanjibansg changed the title ~~[WIP][GSoC][TMVA][SOFIE] PyTorch Parser TMVA~~ [GSoC][TMVA][SOFIE] PyTorch Parser TMVA Aug 12, 2021

sanjibansg force-pushed the pytorch branch from 4511e11 to 7cb6d39 Compare August 16, 2021 14:53

sanjibansg mentioned this pull request Aug 20, 2021

[GSoC][TMVA][SOFIE] Tutorials for RModelParsers #8874

Merged

sanjibansg force-pushed the pytorch branch from 7cb6d39 to b597a10 Compare August 24, 2021 23:25

lmoneta approved these changes Aug 30, 2021

View reviewed changes

sanjibansg force-pushed the pytorch branch from 6d152d3 to 7be29cb Compare August 30, 2021 09:23

lmoneta requested changes Aug 30, 2021

View reviewed changes

sanjibansg added 15 commits August 30, 2021 22:42

Initial Commit for RModelParser_Pytorch

c432fe0

Modify PyTorch Parser

0d1c2f7

1) Added support for Tensors of Dynamic Axes 2) Modified Map for DType for PyTorch

Add: Tests for PyTorch Parser and Converter

fefd90f

Modify: convertStringToType()

0ed71b4

Modify: Tests for PyTorch Parser

e4c3c79

Modify: Configure CMakeLists.txt and Tests for RModelParser_PyTorch

112fcc3

1) Added GTests for RModelParser_PyTorch 2) Added EmitFromPyTorch to generate header files 3) Configured CMake for tests

Remove: unnecesarry files for tests

e683140

Modify: RModelParser_PyTorch

d86ec4a

1. Refactor RModelParser_PyTorch 2. Call to AddBlasRoutines() while adding Gemm Node

Fix: Failing test in TestRModelParserPyTorch for nn.Module

6a73356

Move Python Utility functions to PyMethodBase

75279fc

Add Documentation

eeab78d

Modify CMakeLists.txt for tests to have CMAKE_SOURCE_DIR instead of r…

ef035e5

…elative path

Resolve Merge conflicts

a79be51

Modify Public Header file to have minimal dependency

26fb78d

sanjibansg force-pushed the pytorch branch from 8ab3acc to 26fb78d Compare August 30, 2021 17:15

lmoneta approved these changes Aug 31, 2021

View reviewed changes

lmoneta merged commit a5541b7 into root-project:master Aug 31, 2021

sanjibansg deleted the pytorch branch September 5, 2021 13:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GSoC][TMVA][SOFIE] PyTorch Parser TMVA #8684

[GSoC][TMVA][SOFIE] PyTorch Parser TMVA #8684

sanjibansg commented Jul 15, 2021 •

edited

Loading

phsft-bot commented Jul 15, 2021

lmoneta commented Aug 9, 2021

phsft-bot commented Aug 9, 2021

phsft-bot commented Aug 9, 2021

lmoneta commented Aug 16, 2021

phsft-bot commented Aug 16, 2021

phsft-bot commented Aug 16, 2021

lmoneta commented Aug 17, 2021

phsft-bot commented Aug 17, 2021

phsft-bot commented Aug 17, 2021

lmoneta left a comment

lmoneta commented Aug 30, 2021

phsft-bot commented Aug 30, 2021

phsft-bot commented Aug 30, 2021

lmoneta left a comment

lmoneta commented Aug 30, 2021

phsft-bot commented Aug 30, 2021

phsft-bot commented Aug 30, 2021

lmoneta left a comment

[GSoC][TMVA][SOFIE] PyTorch Parser TMVA #8684

[GSoC][TMVA][SOFIE] PyTorch Parser TMVA #8684

Conversation

sanjibansg commented Jul 15, 2021 • edited Loading

Description

Progress

Interface

Checklist:

Tests

Documentation

phsft-bot commented Jul 15, 2021

lmoneta commented Aug 9, 2021

phsft-bot commented Aug 9, 2021

phsft-bot commented Aug 9, 2021

Errors:

lmoneta commented Aug 16, 2021

phsft-bot commented Aug 16, 2021

phsft-bot commented Aug 16, 2021

Errors:

lmoneta commented Aug 17, 2021

phsft-bot commented Aug 17, 2021

phsft-bot commented Aug 17, 2021

Warnings:

lmoneta left a comment

Choose a reason for hiding this comment

lmoneta commented Aug 30, 2021

phsft-bot commented Aug 30, 2021

phsft-bot commented Aug 30, 2021

Warnings:

lmoneta left a comment

Choose a reason for hiding this comment

lmoneta commented Aug 30, 2021

phsft-bot commented Aug 30, 2021

phsft-bot commented Aug 30, 2021

Failing tests:

lmoneta left a comment

Choose a reason for hiding this comment

sanjibansg commented Jul 15, 2021 •

edited

Loading