Add a demo backend with compiler #52603

iseeyuan · 2021-02-22T16:42:49Z

This PR introduced a backend with minimum compilation capability to the to_ flow. The targets are:

Demonstrate the end-to-end flow with adding a backend -> compilation -> runtime
How the backend compilation errors be surfaced to the user, with the original model's source code information. (C++ only in this PR. Python APIs will be demonstrated in a following PR.)

Changes:

Compilation

A backend with minimum compilation features, "backend_with_compiler_demo" is added.
The compilation happens AOT in the pre_process function registered to this backend.
Compiled results are stored in a string blob for each method. They are serialized to the lowered module with __get_state__ function.
Error message with model source code is thrown, for features not handled by the backend compiler.

Runtime

The compiled blob is loaded in __set_state__ method.
The compile function of the backend parses the blob to the format that the backend can understand.
The execute function of the backend executes the specified method (handle).

Tests:

BackendTest.TestCompiler: the C++ end-to-end demonstration on a supported model. After compilation and running, the lowered model produces the same result as the original torchscript model.
BackendTest.TestCompilerNotSupport: Demonstrate the error message from the AOT compilation for a feature not supported from the input module. The error message looks like:

"The node of aten::mul is not supported in this compiler. Source code:   File "<string>", line 3

    def forward(self, x, h):
        return x * h
               ~~~~~ <--- HERE

Stack from ghstack:

[Lite Interpreter] Support features from to_backend #52870 [Lite Interpreter] Support features from to_backend
Add a demo backend with compiler #52603 Add a demo backend with compiler

Differential Revision: D26593968

[ghstack-poisoned]

facebook-github-bot · 2021-02-22T16:43:00Z

💊 CI failures summary and remediations

As of commit d1fd302 (more details on the Dr. CI page):

1/1 failures possibly* introduced in this PR
- 1/1 non-scanned failure(s)

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

[ghstack-poisoned]

raziel

Thanks Martin, this is great.

I left some minor comments in the code.

The main suggestion, which is something you're probably already working on, is to make the backend really implement execution capabilities according to what was compiled rather than hardcoding some constants and ignoring those in the original Module (nothing fancy though).

test/cpp/jit/CMakeLists.txt

test/cpp/jit/backend_with_compiler.cpp

raziel · 2021-02-22T18:27:59Z

test/cpp/jit/backend_with_compiler.cpp

+
+namespace torch {
+namespace jit {
+


It'd be good to document up here the behaviour/contract of this backend: e.g. Implementation of a PyTorch Backend that can process, compile and execute TorchScript Modules composed of 'add' and 'sub' operators between constants and expect 2 inputs of type Tensor of blah... which are added or subtracted to blah blah

You can punt this until you add support for constants.

But basically it'd be good if this backend, however simple (it really doesn't need to be fancy), implemented some well defined algorithm that used all the bits the preprocessed/compiled Module rather than just hardcoding some values, and ignoring others.

Good suggestion! I'll add more comments, and discuss with you on defined algorithms offline.

Now I get your point of "well defined algorithm". I'll add the support for constant.

test/cpp/jit/test_backend.cpp

raziel · 2021-02-22T18:54:27Z

test/cpp/jit/test_backend.cpp

+  compile_spec.insert("forward", fake_dict);
+  auto any_dict_ty = DictType::create(StringType::get(), AnyType::get());
+  // lowered module
+  //  auto lm = torch::jit::detail::codegen_backend_module(


delete commented code?

This PR introduced a backend with minimum compilation capability to the to_<backend> flow. The targets are: - Demonstrate the end-to-end flow with adding a backend -> compilation -> runtime - How the backend compilation errors be surfaced to the user, with the original model's source code information. (C++ only in this PR. Python APIs will be demonstrated in a following PR.) Changes: - Compilation 1. A backend with minimum compilation features, "backend_with_compiler_demo" is added. 2. The compilation happens AOT in the ```pre_process``` function registered to this backend. 3. Compiled results are stored in a string blob for each method. They are serialized to the lowered module with ```__get_state__``` function. 4. Error message with model source code is thrown, for features not handled by the backend compiler. - Runtime 1. The compiled blob is loaded in ```__set_state__``` method. 2. The ```compile``` function of the backend pass through the AOT compiled blob. (TODO: parsing the blob to the format that the backend can understand can happen here.) 3. The ```execute``` function of the backend executes the specified method (handle). Tests: - ```BackendTest.TestCompiler```: the C++ end-to-end demonstration on a supported model. After compilation and running, the lowered model produces the same result as the original torchscript model. - ```BackendTest.TestCompilerNotSupport```: Demonstrate the error message from the AOT compilation for a feature not supported from the input module. The error message looks like: ``` "The node of aten::mul is not supported in this compiler. Source code: File "<string>", line 3 def forward(self, x, h): return x * h ~~~~~ <--- HERE ``` [ghstack-poisoned]

ghstack-source-id: f0cdacf3fc68b5720329d29ef17fb07bf405d95b Pull Request resolved: #52603

This PR introduced a backend with minimum compilation capability to the to_<backend> flow. The targets are: - Demonstrate the end-to-end flow with adding a backend -> compilation -> runtime - How the backend compilation errors be surfaced to the user, with the original model's source code information. (C++ only in this PR. Python APIs will be demonstrated in a following PR.) Changes: - Compilation 1. A backend with minimum compilation features, "backend_with_compiler_demo" is added. 2. The compilation happens AOT in the ```pre_process``` function registered to this backend. 3. Compiled results are stored in a string blob for each method. They are serialized to the lowered module with ```__get_state__``` function. 4. Error message with model source code is thrown, for features not handled by the backend compiler. - Runtime 1. The compiled blob is loaded in ```__set_state__``` method. 2. The ```compile``` function of the backend pass through the AOT compiled blob. (TODO: parsing the blob to the format that the backend can understand can happen here.) 3. The ```execute``` function of the backend executes the specified method (handle). Tests: - ```BackendTest.TestCompiler```: the C++ end-to-end demonstration on a supported model. After compilation and running, the lowered model produces the same result as the original torchscript model. - ```BackendTest.TestCompilerNotSupport```: Demonstrate the error message from the AOT compilation for a feature not supported from the input module. The error message looks like: ``` "The node of aten::mul is not supported in this compiler. Source code: File "<string>", line 3 def forward(self, x, h): return x * h ~~~~~ <--- HERE ``` Differential Revision: [D26593968](https://our.internmc.facebook.com/intern/diff/D26593968) [ghstack-poisoned]

ghstack-source-id: 6f821f36c591b92fa76020e91298ac5d60ed4d17 Pull Request resolved: #52603

test/cpp/jit/backend_with_compiler.cpp

SplitInfinity

I left a few comments inline (mostly spelling errors).

Maybe it's okay for a demo backend, but it seems weird to preprocess a module by creating a list of std::string instructions. Why not encode the instructions in some way?

test/cpp/jit/test_utils.h

SplitInfinity · 2021-02-23T20:39:07Z

test/cpp/jit/test_utils.h

+  try {                                                                  \
+    (void)statement;                                                     \
+    ASSERT_TRUE(false);                                                  \
+  } catch (const std::exception& e) {                                    \


Consider being more specific about the exception type (and also checking that only that type is thrown).

The purpose of this macro is to compare the exception message itself to differentiate the exact throw. It's used to check expected messages from all exception types.

SplitInfinity · 2021-02-23T20:47:59Z

test/cpp/jit/test_backend.cpp

@@ -32,9 +33,95 @@ TEST(BackendTest, ToBackend) {
  // lowered module
  auto lm = torch::jit::detail::codegen_backend_module(
      "test_backend", m, compile_spec, any_dict_ty);
+  // lowered module code:
+  /*
+class test_backendLoweredModule(Module):


Nit: Indentation looks a bit weird for this comment but it's probably just my browser.

Thanks. Fixed.

test/cpp/jit/backend_with_compiler.cpp

SplitInfinity · 2021-02-23T20:50:35Z

test/cpp/jit/backend_with_compiler.cpp

+// by the backend compiler.
+//
+// Runtime
+// 1. The compiled blob is loaded in __set_state__ method.


Suggested change

// 1. The compiled blob is loaded in __set_state__ method.

// 1. The compiled blob is loaded in __setstate__ method.

test/cpp/jit/backend_with_compiler.cpp

SplitInfinity · 2021-02-23T20:55:50Z

test/cpp/jit/backend_with_compiler.cpp

+  c10::impl::GenericList execute(
+      c10::IValue handle,
+      c10::impl::GenericList inputs) override {
+    //    TORCH_INTERNAL_ASSERT(handle.isString());


Suggested change

// TORCH_INTERNAL_ASSERT(handle.isString());

test/cpp/jit/backend_with_compiler.cpp

raziel

Thanks!

test/cpp/jit/backend_with_compiler.cpp

raziel · 2021-02-23T20:20:14Z

test/cpp/jit/backend_with_compiler.cpp

+// by the backend compiler.
+//
+// Runtime
+// 1. The compiled blob is loaded in __set_state__ method.


set_state -> setstate

test/cpp/jit/backend_with_compiler.cpp

This PR introduced a backend with minimum compilation capability to the to_<backend> flow. The targets are: - Demonstrate the end-to-end flow with adding a backend -> compilation -> runtime - How the backend compilation errors be surfaced to the user, with the original model's source code information. (C++ only in this PR. Python APIs will be demonstrated in a following PR.) Changes: - Compilation 1. A backend with minimum compilation features, "backend_with_compiler_demo" is added. 2. The compilation happens AOT in the ```pre_process``` function registered to this backend. 3. Compiled results are stored in a string blob for each method. They are serialized to the lowered module with ```__get_state__``` function. 4. Error message with model source code is thrown, for features not handled by the backend compiler. - Runtime 1. The compiled blob is loaded in ```__set_state__``` method. 2. The ```compile``` function of the backend parses the blob to the format that the backend can understand. 3. The ```execute``` function of the backend executes the specified method (handle). Tests: - ```BackendTest.TestCompiler```: the C++ end-to-end demonstration on a supported model. After compilation and running, the lowered model produces the same result as the original torchscript model. - ```BackendTest.TestCompilerNotSupport```: Demonstrate the error message from the AOT compilation for a feature not supported from the input module. The error message looks like: ``` "The node of aten::mul is not supported in this compiler. Source code: File "<string>", line 3 def forward(self, x, h): return x * h ~~~~~ <--- HERE ``` Differential Revision: [D26593968](https://our.internmc.facebook.com/intern/diff/D26593968) [ghstack-poisoned]

ghstack-source-id: 6f5673b016f037cb306330d3856651816c8bfe5f Pull Request resolved: #52603

This PR introduced a backend with minimum compilation capability to the to_<backend> flow. The targets are: - Demonstrate the end-to-end flow with adding a backend -> compilation -> runtime - How the backend compilation errors be surfaced to the user, with the original model's source code information. (C++ only in this PR. Python APIs will be demonstrated in a following PR.) Changes: - Compilation 1. A backend with minimum compilation features, "backend_with_compiler_demo" is added. 2. The compilation happens AOT in the ```pre_process``` function registered to this backend. 3. Compiled results are stored in a string blob for each method. They are serialized to the lowered module with ```__get_state__``` function. 4. Error message with model source code is thrown, for features not handled by the backend compiler. - Runtime 1. The compiled blob is loaded in ```__set_state__``` method. 2. The ```compile``` function of the backend parses the blob to the format that the backend can understand. 3. The ```execute``` function of the backend executes the specified method (handle). Tests: - ```BackendTest.TestCompiler```: the C++ end-to-end demonstration on a supported model. After compilation and running, the lowered model produces the same result as the original torchscript model. - ```BackendTest.TestCompilerNotSupport```: Demonstrate the error message from the AOT compilation for a feature not supported from the input module. The error message looks like: ``` "The node of aten::mul is not supported in this compiler. Source code: File "<string>", line 3 def forward(self, x, h): return x * h ~~~~~ <--- HERE ``` Differential Revision: [D26593968](https://our.internmc.facebook.com/intern/diff/D26593968) [ghstack-poisoned]

ghstack-source-id: 43c3838bc5723c676bb4a9bd2fc95823cb13fad5 Pull Request resolved: #52603

This PR introduced a backend with minimum compilation capability to the to_<backend> flow. The targets are: - Demonstrate the end-to-end flow with adding a backend -> compilation -> runtime - How the backend compilation errors be surfaced to the user, with the original model's source code information. (C++ only in this PR. Python APIs will be demonstrated in a following PR.) Changes: - Compilation 1. A backend with minimum compilation features, "backend_with_compiler_demo" is added. 2. The compilation happens AOT in the ```pre_process``` function registered to this backend. 3. Compiled results are stored in a string blob for each method. They are serialized to the lowered module with ```__get_state__``` function. 4. Error message with model source code is thrown, for features not handled by the backend compiler. - Runtime 1. The compiled blob is loaded in ```__set_state__``` method. 2. The ```compile``` function of the backend parses the blob to the format that the backend can understand. 3. The ```execute``` function of the backend executes the specified method (handle). Tests: - ```BackendTest.TestCompiler```: the C++ end-to-end demonstration on a supported model. After compilation and running, the lowered model produces the same result as the original torchscript model. - ```BackendTest.TestCompilerNotSupport```: Demonstrate the error message from the AOT compilation for a feature not supported from the input module. The error message looks like: ``` "The node of aten::mul is not supported in this compiler. Source code: File "<string>", line 3 def forward(self, x, h): return x * h ~~~~~ <--- HERE ``` Differential Revision: [D26593968](https://our.internmc.facebook.com/intern/diff/D26593968) [ghstack-poisoned]

ghstack-source-id: 0bbf80bd64f6076b83022ad64c872ed188af9370 Pull Request resolved: #52603

This PR introduced a backend with minimum compilation capability to the to_<backend> flow. The targets are: - Demonstrate the end-to-end flow with adding a backend -> compilation -> runtime - How the backend compilation errors be surfaced to the user, with the original model's source code information. (C++ only in this PR. Python APIs will be demonstrated in a following PR.) Changes: - Compilation 1. A backend with minimum compilation features, "backend_with_compiler_demo" is added. 2. The compilation happens AOT in the ```pre_process``` function registered to this backend. 3. Compiled results are stored in a string blob for each method. They are serialized to the lowered module with ```__get_state__``` function. 4. Error message with model source code is thrown, for features not handled by the backend compiler. - Runtime 1. The compiled blob is loaded in ```__set_state__``` method. 2. The ```compile``` function of the backend parses the blob to the format that the backend can understand. 3. The ```execute``` function of the backend executes the specified method (handle). Tests: - ```BackendTest.TestCompiler```: the C++ end-to-end demonstration on a supported model. After compilation and running, the lowered model produces the same result as the original torchscript model. - ```BackendTest.TestCompilerNotSupport```: Demonstrate the error message from the AOT compilation for a feature not supported from the input module. The error message looks like: ``` "The node of aten::mul is not supported in this compiler. Source code: File "<string>", line 3 def forward(self, x, h): return x * h ~~~~~ <--- HERE ``` Differential Revision: [D26593968](https://our.internmc.facebook.com/intern/diff/D26593968) [ghstack-poisoned]

ghstack-source-id: 24d4c6f485de2e7e4ceb2548a3c31a1989335ee1 Pull Request resolved: #52603

This PR introduced a backend with minimum compilation capability to the to_<backend> flow. The targets are: - Demonstrate the end-to-end flow with adding a backend -> compilation -> runtime - How the backend compilation errors be surfaced to the user, with the original model's source code information. (C++ only in this PR. Python APIs will be demonstrated in a following PR.) Changes: - Compilation 1. A backend with minimum compilation features, "backend_with_compiler_demo" is added. 2. The compilation happens AOT in the ```pre_process``` function registered to this backend. 3. Compiled results are stored in a string blob for each method. They are serialized to the lowered module with ```__get_state__``` function. 4. Error message with model source code is thrown, for features not handled by the backend compiler. - Runtime 1. The compiled blob is loaded in ```__set_state__``` method. 2. The ```compile``` function of the backend parses the blob to the format that the backend can understand. 3. The ```execute``` function of the backend executes the specified method (handle). Tests: - ```BackendTest.TestCompiler```: the C++ end-to-end demonstration on a supported model. After compilation and running, the lowered model produces the same result as the original torchscript model. - ```BackendTest.TestCompilerNotSupport```: Demonstrate the error message from the AOT compilation for a feature not supported from the input module. The error message looks like: ``` "The node of aten::mul is not supported in this compiler. Source code: File "<string>", line 3 def forward(self, x, h): return x * h ~~~~~ <--- HERE ``` Differential Revision: [D26593968](https://our.internmc.facebook.com/intern/diff/D26593968) [ghstack-poisoned]

facebook-github-bot · 2021-02-26T19:56:21Z

@iseeyuan merged this pull request in b2520ab.

Summary: Pull Request resolved: pytorch#52603 This PR introduced a backend with minimum compilation capability to the to_<backend> flow. The targets are: - Demonstrate the end-to-end flow with adding a backend -> compilation -> runtime - How the backend compilation errors be surfaced to the user, with the original model's source code information. (C++ only in this PR. Python APIs will be demonstrated in a following PR.) Changes: - Compilation 1. A backend with minimum compilation features, "backend_with_compiler_demo" is added. 2. The compilation happens AOT in the ```pre_process``` function registered to this backend. 3. Compiled results are stored in a string blob for each method. They are serialized to the lowered module with ```__get_state__``` function. 4. Error message with model source code is thrown, for features not handled by the backend compiler. - Runtime 1. The compiled blob is loaded in ```__set_state__``` method. 2. The ```compile``` function of the backend pass through the AOT compiled blob. (TODO: parsing the blob to the format that the backend can understand can happen here.) 3. The ```execute``` function of the backend executes the specified method (handle). Test Plan: - ```BackendTest.TestCompiler```: the C++ end-to-end demonstration on a supported model. After compilation and running, the lowered model produces the same result as the original torchscript model. - ```BackendTest.TestCompilerNotSupport```: Demonstrate the error message from the AOT compilation for a feature not supported from the input module. The error message looks like: ``` "The node of aten::mul is not supported in this compiler. Source code: File "<string>", line 3 def forward(self, x, h): return x * h ~~~~~ <--- HERE ``` Reviewed By: raziel Differential Revision: D26593968 Pulled By: iseeyuan fbshipit-source-id: 8f264f60a0470e9f07e36fdeccbf17da6c1d7cd7

Add a demo backend with compiler

f242f07

[ghstack-poisoned]

facebook-github-bot added the cla signed label Feb 22, 2021

facebook-github-bot added the oncall: jit Add this issue/PR to JIT oncall triage queue label Feb 22, 2021

Update on "Add a demo backend with compiler"

d9efb98

[ghstack-poisoned]

iseeyuan requested review from suo, raziel, ljk53, SplitInfinity and kimishpatel February 22, 2021 17:58

raziel reviewed Feb 22, 2021

View reviewed changes

iseeyuan added a commit that referenced this pull request Feb 22, 2021

Add a demo backend with compiler

e95abc3

ghstack-source-id: f0cdacf3fc68b5720329d29ef17fb07bf405d95b Pull Request resolved: #52603

iseeyuan added a commit that referenced this pull request Feb 23, 2021

Add a demo backend with compiler

d563765

ghstack-source-id: 6f821f36c591b92fa76020e91298ac5d60ed4d17 Pull Request resolved: #52603

iseeyuan requested a review from raziel February 23, 2021 19:08

kimishpatel reviewed Feb 23, 2021

View reviewed changes

test/cpp/jit/backend_with_compiler.cpp Outdated Show resolved Hide resolved

kimishpatel reviewed Feb 23, 2021

View reviewed changes

test/cpp/jit/backend_with_compiler.cpp Outdated Show resolved Hide resolved

kimishpatel reviewed Feb 23, 2021

View reviewed changes

test/cpp/jit/backend_with_compiler.cpp Outdated Show resolved Hide resolved

SplitInfinity approved these changes Feb 23, 2021

View reviewed changes

raziel approved these changes Feb 23, 2021

View reviewed changes

suo removed their request for review February 24, 2021 00:57

iseeyuan added a commit that referenced this pull request Feb 24, 2021

Add a demo backend with compiler

3a5c11c

ghstack-source-id: 6f5673b016f037cb306330d3856651816c8bfe5f Pull Request resolved: #52603

iseeyuan added a commit that referenced this pull request Feb 24, 2021

Add a demo backend with compiler

9ac88a5

ghstack-source-id: 43c3838bc5723c676bb4a9bd2fc95823cb13fad5 Pull Request resolved: #52603

iseeyuan added the ci/all label Feb 24, 2021

iseeyuan added a commit that referenced this pull request Feb 24, 2021

Add a demo backend with compiler

93ebda1

ghstack-source-id: 0bbf80bd64f6076b83022ad64c872ed188af9370 Pull Request resolved: #52603

iseeyuan added a commit that referenced this pull request Feb 25, 2021

Add a demo backend with compiler

369c260

ghstack-source-id: 24d4c6f485de2e7e4ceb2548a3c31a1989335ee1 Pull Request resolved: #52603

iseeyuan mentioned this pull request Feb 25, 2021

[Lite Interpreter] Support features from to_backend #52870

Closed

iseeyuan removed the ci/all label Feb 25, 2021

iseeyuan added 5 commits February 25, 2021 14:12

facebook-github-bot closed this in b2520ab Feb 26, 2021

facebook-github-bot added the Merged label Feb 26, 2021

facebook-github-bot deleted the gh/iseeyuan/111/head branch March 2, 2021 15:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a demo backend with compiler #52603

Add a demo backend with compiler #52603

iseeyuan commented Feb 22, 2021 •

edited

facebook-github-bot commented Feb 22, 2021 •

edited

raziel left a comment

raziel Feb 22, 2021

iseeyuan Feb 22, 2021

iseeyuan Feb 23, 2021

raziel Feb 22, 2021

SplitInfinity left a comment

SplitInfinity Feb 23, 2021

iseeyuan Feb 24, 2021

SplitInfinity Feb 23, 2021

iseeyuan Feb 24, 2021

SplitInfinity Feb 23, 2021

SplitInfinity Feb 23, 2021

raziel left a comment

raziel Feb 23, 2021

facebook-github-bot commented Feb 26, 2021

	// 1. The compiled blob is loaded in __set_state__ method.
	// 1. The compiled blob is loaded in __setstate__ method.


		namespace torch {
		namespace jit {

Add a demo backend with compiler #52603

Add a demo backend with compiler #52603

Conversation

iseeyuan commented Feb 22, 2021 • edited

facebook-github-bot commented Feb 22, 2021 • edited

💊 CI failures summary and remediations

raziel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SplitInfinity left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raziel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Feb 26, 2021

iseeyuan commented Feb 22, 2021 •

edited

facebook-github-bot commented Feb 22, 2021 •

edited