[RUNTIME] Support standardize runtime module #4532

FrozenGene · 2019-12-17T10:56:06Z

As RFC https://discuss.tvm.ai/t/standardize-graphruntime-exports-into-a-single-dll/4667 proposed, we want to standardize runtime export.

python/tvm/module.py

src/codegen/codegen.cc

src/runtime/library_module.cc

FrozenGene · 2019-12-18T12:49:53Z

@tqchen I have addressed the comments you mentioned. And I also supply one more test of dso module import another dso module.

FrozenGene · 2019-12-18T12:53:31Z

@zhiics Would like you could help to review too. Your recent PR #4482 also change the export_library and serialization / deserialization logic. Currently, I test the unit testing of #4482 and could pass.

src/codegen/codegen.cc

src/runtime/library_module.cc

python/tvm/module.py

src/codegen/codegen.cc

tests/python/unittest/test_standardize_runtime.py

FrozenGene · 2019-12-19T03:45:15Z

@tqchen @zhiics Please help to review it again.

python/tvm/module.py

src/codegen/codegen.cc

src/runtime/library_module.cc

tests/python/unittest/test_standardize_runtime.py

src/runtime/library_module.cc

src/codegen/codegen.cc

FrozenGene · 2019-12-20T04:05:14Z

@tqchen @zhiics code is updated and help to review it again.

tqchen

some final nits, perhaps we need a testcase with a few C modules

python/tvm/module.py

tests/python/unittest/test_standardize_runtime.py

FrozenGene · 2019-12-21T13:02:26Z

@tqchen @zhiics please help to review it again.

tqchen

some final nits :)

python/tvm/module.py

tests/python/unittest/test_runtime_module_export.py

tqchen · 2019-12-21T17:06:36Z

cc @zhiics please also take a look again

FrozenGene · 2019-12-21T17:38:51Z

@tqchen @zhiics Has updated the code according to the comments. Please help to review it again.

zhiics

LGTM

tqchen · 2019-12-22T04:14:57Z

Thanks @FrozenGene @zhiics this PR is now merged

tqchen · 2019-12-22T04:33:47Z

@FrozenGene great job, can you followup with a PR that add developer docs describes the module serialization format standard?

FrozenGene · 2019-12-22T05:34:57Z

@tqchen, like the table of https://docs.tvm.ai/dev/nnvm_json_spec.html?

tqchen · 2019-12-22T05:42:58Z

yap, just to introduce how do we serialize and package all these modules and how does the Module's API play together.

FrozenGene · 2019-12-22T05:52:51Z

So, likely to say, the doc should combine your rfc (could cover how do we serialize and package) and the implementation of def export_library / ModuleSerializer::SerializeModule / ProcessModuleBlob (could cover how they work together). Not just the table I mentioned just now.

tqchen · 2019-12-22T06:13:49Z

Hopefully it will also help more people recognize your work :)

apivovarov · 2019-12-31T06:25:52Z

@FrozenGene Is this change backward compatible? Can new TVM runtime load compiled Cuda models compiled before this change?

FrozenGene · 2019-12-31T06:56:54Z

@FrozenGene Is this change backward compatible? Can new TVM runtime load compiled Cuda models compiled before this change?

Yes. Backward compatible. For example, you compile one compiled cuda model before this change and export to deploy_old.so, you could load deploy_old.so in the new runtime (merged this change). However, if you compile one model using new runtime (merged this change) and export to deploy_new.so, which can not be loaded into old runtime (before this change). Because we will write _lib for LLVM mod / write _import_tree for constructing model import relationship. Your refer issue is to use new runtime to export library, but loaded into old runtime (DLR), so you can not find _lib loader and _import_tree.

@cchung100m

* Change upstream url * Fix bias_add gradient (apache#4516) * Fix bias_add gradient A change caused collapse_sum_like to reject implicit dimension broadcasting for bias_add gradient, so switch to explicit sum reduction on the non-bias axis dimensions. * Lint fix * [Bugfix][Frontend][TFlite] Fix wrong function call in TANH tests (apache#4517) * Replace sigmoid() with tanh() in tests for TANH * Fixed extra reshape parameter bug. (apache#4524) * Use the best tuner possible (apache#4397) * Use the best tuner possible * Add comment denoting availability of better tuners * Fix typos and wording * [ir] use DataType instead of Type for readability because Type has been deprecated (apache#4513) * add bfloat16 typeflag support (apache#4525) * fix empty config caused KeyError (apache#4520) * fix onnx shape dtype (apache#4528) * fix crash issue in tsim backend (apache#4527) * PIL is depreciated and should be replaced with pillow (a fork of PIL) (apache#4533) Change-Id: If2075df5475505f2da87dae7145af5a7ab83d8a4 * [Relay] External codegen (apache#4482) * Update legacy places from nnvm to relay. (apache#4535) * Update legacy places from nnvm to relay. This PR prepares the current mainline to remove nnvm compiler dep. * remove legacy stage * Implement 1d deconvolution (apache#4476) * [relay][op] add expand op (from ONNX) to relay frontend (apache#4483) * Add Expand to onnx.py * add test function for expand * Fix a onnx frontend test * Add tests for the value itself instead of shape only on test_expand * Cleaned up some unnecessary modifications. * [TOPI] Allow batch matmul to be fused into injective ops (apache#4537) * [TOPI] Fixed nms max_output_size loop (apache#4541) One of the loops in hybrid_nms used for performing the max_output_size reordering was incorrectly designated as parallel resulting in incorrect behaviour. This patch changes that loop to a serial loop. Change-Id: I97184f5887f5f028d8ab339fa2808eb7630a4017 * [DOCS] Mention Ninja build system in install/from_source.rst (apache#4554) * [DOCS] Mention Ninja build system in install/from_source.rst * Address comments * [PYTHON][FFI] Cythonize NDArray.copyto (apache#4549) * [PYTHON][FFI] Cythonize NDArray.copyto * Cythonize the shape property * vm external codegen (apache#4544) * [COMMUNITY] @cchung100m -> reviewer (apache#4557) * [VTA] improved virtual memory mapping (apache#4545) * [VTA] improved virtual memory mapping * Update virtual_memory.cc * [IR] fix style in ir_mutator and ir_visitor (apache#4561) * [RUNTIME][VULKAN] Fix compiler warning (apache#4559) * [REFACTOR][DTYPE] Isolate dtype to runtime (apache#4560) dtype.h -> runtime/data_type.h Changes: - Rename all old reference of tvm::Type to DataType - ExprNode.type -> ExprNode.dtype - Expr.type() -> Expr.dtype() - Change Expr related functions to expr_operator. - DataType::min() -> min_value(DataType) - DataType::max() -> max_value(DataType) - Move type constructor Int, UInt, Float, Handle, Bool into DataType. - Int(bits) -> DataType::Int(bits) - UInt(bits) -> DataType::UInt(bits) * Support standardize runtime module (apache#4532) * [Relay][Frontend][ONNX] Support auto_pad in Conv and ConvTranspose (apache#4563) * [TEST] Remove nnvm related code in topi and test script (apache#4562) * [TEST] Remove nnvm related code in topi and test script * Remove docs dep * [Relay] add max_pool3d in relay and TF converter (apache#4551) * [Relay] add max_pool3d in relay and TF converter * fix comments * Remove nnvm (apache#4565) * [VTA][Chisel] End-to-end Inference with Chisel VTA (apache#4574) * [VTA][Chisel] End-to-end Inference with Chisel VTA * Update TensorAlu.scala * remove unnecessary cast to int32 (apache#4573) * Fix llvm-enabled build by adding missing intrinsics headers (apache#4575) * [DEPRECATION] Remove NNVM compiler (apache#4571) * Remove NNVM compiler * [Relay/Topi][Op] Added native DepthToSpace and SpaceToDepth Operators (apache#4566) * Added tvm function stencil for subpixel operations to topi. * Topi subpixel operators added and tested. * Added subpixel attrs. * Added depth_to_space relay attributes. * depth_to_space fully working. * Fixed NHWC shape bug. * SpaceToDepth in and all tests passing. * lint fixes. * Added string include * Fixed topi formatting. * Added DCR/CDR mode to depthtospace operator. * [DOC] fix doc in api.py (apache#4580) * [DEPRECATION] Cleanup legacy verilog support (apache#4576) This PR cleans up the left over code for legacy verilog support which was experimental. The new hardware backend path is now support by VTA via TSIM. * [RUNTIME] Remove Extension VTable in favor of Unified Object system. (apache#4578) Before the unified object protocol, we support pass additional extension objects around by declaring a type as an extension type. The old extension mechanism requires the types to register their constructor and deleter to a VTable and does not enjoy the benefit of the self-contained deletion property of the new Object system. This PR upgrades the extension example to make use of the new object system and removed the old Extension VTable. Note that the register_extension funtion in the python side continues to work when the passed argument does not require explicit container copy/deletion, which covers the current usecases of the extension mechanism. * Some Windows and MSVC fixes (apache#4569) * fix python exception creation in Windows * better string conversion for msvc * fix cpp style issue * [NEWS] add v0.6 release (apache#4558) * [NEWS] add v0.6 release * remove link prefix * fix issue number * [DOCS]fix typos in autotvm tutorial (apache#4585) * [Quantization, Calibrate] Fix context creation when current_target is explicity set (apache#4582) * [Container] Fix NDArray SaveDLTensor declaration and implementation signature different (apache#4586) * [TOPI][AutoTVM] NHWC conv2d templates for ARM (apache#3859) * [AutoTVM][TOPI] NHWC conv2d templates (spatial pack) for ARM As some frontends (tflite for example) are using NHWC as the default layout, we are enabling NHWC schedule templates in TOPI and AutoTVM. * some comments fix * [FIX][TOPI][X86] schedule dense pack (apache#4539) * [Relay] Convert Layout Pass. (apache#4335) * [Relay][AlterLayout] Broadcast with scalar shape (apache#4577) * [TOPI] add 3D upsampling Op. (apache#4584) * [TOPI] add 3D upsampling Op. * fix lint issues * change align_corners to coordinate_transformation_mode * fix resize3d half_pixel * make a simple function and clean up trilinear_resize3d_python * fix doc * [Runtime] add necessary const qualifier for NDArray container of parameters (apache#4590) * [autotvm] fix typos in comment (apache#4591) * fix tf.compat.v1 issue for tf verison <=1.12 (apache#4593) * [FRONTEND][TF] conv2d_transpose 'SAME' support kernel more than 1x1 (apache#4484) * [FRONTEND][TF] conv3d_transpose 'SAME' support kernel more than 1x1 * revised per as review comments * add more fallback wolkaround to make all tests pass * [GraphRuntime] Support parameter out in the graph runtime debug (apache#4598) * [GraphRuntime] Support parameter out in the graph runtime debug * Dummy commit to trigger build * [Perf] Add CublasLt extern support for better Igemm performance (apache#4550) * cublaslt added * fix lint * address comments * address more comments * Trigger CI * Trigger CI * fix codegenc (apache#4597) * [REFACTOR][RUNTIME] Update NDArray use the Unified Object System (apache#4581) * [REFACTOR][RUNTIME] Move NDArray to Object System. Previously NDArray has its own object reference counting mechanism. This PR migrates NDArray to the unified object protocol. The calling convention of NDArray remained intact. That means NDArray still has its own type_code and its handle is still DLTensor compatible. In order to do so, this PR added a few minimum runtime type detection in TVMArgValue and RetValue only when the corresponding type is a base type(ObjectRef) that could also refer to NDArray. This means that even if we return a base reference object ObjectRef which refers to the NDArray. The type_code will still be translated correctly as kNDArrayContainer. If we assign a non-base type(say Expr) that we know is not compatible with NDArray during compile time, no runtime type detection will be performed. This PR also adopts the object protocol for NDArray sub-classing and removed the legacy NDArray subclass protocol. Examples in apps/extension are now updated to reflect that. Making NDArray as an Object brings all the benefits of the object system. For example, we can now use the Array container to store NDArrays. * Address review comments * [Relay][Convert Layout] Handling batch norm layout change. (apache#4600) * [relay][refactor] Cache Op::Get in passes to reduce lookup overhead (apache#4594) * Refactor to use IsOp utility * retrigger CI * Update dmlc_tvm_commit_id.txt * disable one test_batch_norm unit test for now to check CI * enable test_batch_norm Co-authored-by: SWu <SWu@users.noreply.github.com> Co-authored-by: Ina Dobreva <55383260+inadob@users.noreply.github.com> Co-authored-by: Josh Fromm <jwfromm@uw.edu> Co-authored-by: miheer vaidya <v.miheer@gmail.com> Co-authored-by: Liang ZOU <liang.d.zou@gmail.com> Co-authored-by: YixinBao <yixin.bao@intel.com> Co-authored-by: Cody Yu <comaniac0422@gmail.com> Co-authored-by: masahi <masahi129@gmail.com> Co-authored-by: Liangfu Chen <liangfu.chen@icloud.com> Co-authored-by: lhutton1 <35535092+lhutton1@users.noreply.github.com> Co-authored-by: Tianqi Chen <tqchen@users.noreply.github.com> Co-authored-by: Alex Gladkov <gladkov_alex@yahoo.com> Co-authored-by: Takato Yamada <tkclimb0911@gmail.com> Co-authored-by: Haichen Shen <shenhaichen@gmail.com> Co-authored-by: mbarrett97 <55580676+mbarrett97@users.noreply.github.com> Co-authored-by: Hideto Ueno <uenoku.tokotoko@gmail.com> Co-authored-by: Siyuan Feng <Hzfengsy@sjtu.edu.cn> Co-authored-by: Zhao Wu <wuzhaozju@gmail.com> Co-authored-by: Neo Chien <cchung100m@cs.ccu.edu.tw> Co-authored-by: Yong Wu <55wuyong@163.com> Co-authored-by: Dmitri Makarov <dmakarov@users.noreply.github.com> Co-authored-by: Bohan Hou <32121147+spectrometerHBH@users.noreply.github.com> Co-authored-by: kice <wslikerqs@gmail.com> Co-authored-by: Yizhi Liu <liuyizhi@apache.org> Co-authored-by: Wang Yucheng <wyc91543@163.com> Co-authored-by: 王振华(Zhenhua WANG) <i@jackwish.net> Co-authored-by: deepIgnorance <zhengsizemax@outlook.com> Co-authored-by: Animesh Jain <anijain@umich.edu> Co-authored-by: optima2005 <56945758+optima2005@users.noreply.github.com> Co-authored-by: zhuochen <zhuochen@outlook.com> Co-authored-by: Leyuan Wang <laurawly@gmail.com>

FrozenGene force-pushed the standarize_runtime branch 3 times, most recently from 782a066 to f0eac3a Compare December 17, 2019 15:20

tqchen requested changes Dec 17, 2019

View reviewed changes

tqchen added the status: need update need update based on feedbacks label Dec 17, 2019

FrozenGene force-pushed the standarize_runtime branch from f0eac3a to d35d4c5 Compare December 18, 2019 12:46

FrozenGene marked this pull request as ready for review December 18, 2019 12:49

FrozenGene changed the title ~~[WIP] Support standardize runtime module~~ Support standardize runtime module Dec 18, 2019

FrozenGene commented Dec 18, 2019

View reviewed changes

src/codegen/codegen.cc Outdated Show resolved Hide resolved

tqchen requested changes Dec 18, 2019

View reviewed changes

tqchen changed the title ~~Support standardize runtime module~~ [RUNTIME] Support standardize runtime module Dec 18, 2019

tqchen added the status: need review label Dec 18, 2019

zhiics reviewed Dec 18, 2019

View reviewed changes

python/tvm/module.py Outdated Show resolved Hide resolved

python/tvm/module.py Outdated Show resolved Hide resolved

src/codegen/codegen.cc Outdated Show resolved Hide resolved

zhiics reviewed Dec 18, 2019

View reviewed changes

tests/python/unittest/test_standardize_runtime.py Outdated Show resolved Hide resolved

FrozenGene force-pushed the standarize_runtime branch from d35d4c5 to d47164f Compare December 19, 2019 03:43

FrozenGene force-pushed the standarize_runtime branch from d47164f to f9ad013 Compare December 19, 2019 06:34

tqchen requested changes Dec 19, 2019

View reviewed changes

tests/python/unittest/test_standardize_runtime.py Outdated Show resolved Hide resolved

tqchen requested changes Dec 19, 2019

View reviewed changes

src/runtime/library_module.cc Outdated Show resolved Hide resolved

zhiics reviewed Dec 19, 2019

View reviewed changes

src/codegen/codegen.cc Outdated Show resolved Hide resolved

src/codegen/codegen.cc Outdated Show resolved Hide resolved

FrozenGene force-pushed the standarize_runtime branch from f9ad013 to 40f9d0e Compare December 20, 2019 04:04

tqchen requested changes Dec 20, 2019

View reviewed changes

python/tvm/module.py Outdated Show resolved Hide resolved

tqchen requested changes Dec 20, 2019

View reviewed changes

tests/python/unittest/test_standardize_runtime.py Outdated Show resolved Hide resolved

FrozenGene force-pushed the standarize_runtime branch 3 times, most recently from 84dace4 to 2308047 Compare December 21, 2019 13:00

FrozenGene force-pushed the standarize_runtime branch 2 times, most recently from 6b7d542 to 7472af7 Compare December 21, 2019 13:31

tqchen requested changes Dec 21, 2019

View reviewed changes

python/tvm/module.py Outdated Show resolved Hide resolved

python/tvm/module.py Outdated Show resolved Hide resolved

tests/python/unittest/test_runtime_module_export.py Show resolved Hide resolved

Support standardize runtime module

350d8b9

FrozenGene force-pushed the standarize_runtime branch from 7472af7 to 350d8b9 Compare December 21, 2019 17:36

zhiics approved these changes Dec 22, 2019

View reviewed changes

tqchen approved these changes Dec 22, 2019

View reviewed changes

tqchen merged commit f076c83 into apache:master Dec 22, 2019

tqchen added status: accepted and removed status: need review status: need update need update based on feedbacks labels Dec 22, 2019

FrozenGene mentioned this pull request Dec 22, 2019

[Doc] Introduction to module serialization #4564

Merged

zhiics pushed a commit to zhiics/tvm that referenced this pull request Dec 31, 2019

Support standardize runtime module (apache#4532)

e43e575

apivovarov mentioned this pull request Dec 31, 2019

CreateDLRModel failed for cuda model neo-ai/neo-ai-dlr#120

Open

zhiics pushed a commit to neo-ai/tvm that referenced this pull request Jan 11, 2020

Support standardize runtime module (apache#4532)

1efe1f0

zhiics mentioned this pull request Sep 15, 2020

TVM v0.7 Release Note Candidate #6486

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RUNTIME] Support standardize runtime module #4532

[RUNTIME] Support standardize runtime module #4532

FrozenGene commented Dec 17, 2019 •

edited

FrozenGene commented Dec 18, 2019

FrozenGene commented Dec 18, 2019 •

edited

FrozenGene commented Dec 19, 2019

FrozenGene commented Dec 20, 2019

tqchen left a comment

FrozenGene commented Dec 21, 2019

tqchen left a comment

tqchen commented Dec 21, 2019

FrozenGene commented Dec 21, 2019

zhiics left a comment

tqchen commented Dec 22, 2019

tqchen commented Dec 22, 2019

FrozenGene commented Dec 22, 2019

tqchen commented Dec 22, 2019

FrozenGene commented Dec 22, 2019

tqchen commented Dec 22, 2019

apivovarov commented Dec 31, 2019 •

edited

FrozenGene commented Dec 31, 2019 •

edited

[RUNTIME] Support standardize runtime module #4532

[RUNTIME] Support standardize runtime module #4532

Conversation

FrozenGene commented Dec 17, 2019 • edited

FrozenGene commented Dec 18, 2019

FrozenGene commented Dec 18, 2019 • edited

FrozenGene commented Dec 19, 2019

FrozenGene commented Dec 20, 2019

tqchen left a comment

Choose a reason for hiding this comment

FrozenGene commented Dec 21, 2019

tqchen left a comment

Choose a reason for hiding this comment

tqchen commented Dec 21, 2019

FrozenGene commented Dec 21, 2019

zhiics left a comment

Choose a reason for hiding this comment

tqchen commented Dec 22, 2019

tqchen commented Dec 22, 2019

FrozenGene commented Dec 22, 2019

tqchen commented Dec 22, 2019

FrozenGene commented Dec 22, 2019

tqchen commented Dec 22, 2019

apivovarov commented Dec 31, 2019 • edited

FrozenGene commented Dec 31, 2019 • edited

FrozenGene commented Dec 17, 2019 •

edited

FrozenGene commented Dec 18, 2019 •

edited

apivovarov commented Dec 31, 2019 •

edited

FrozenGene commented Dec 31, 2019 •

edited