You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
R1: The creation of ML model can be context dependent. For example, the user needs to be able to specify which GPU to run the graph runtime on.
R2: We start to have multiple variations of model runtime, such as RelayVM. While it does not makes sense to force all model runtime to have the same set of APIs, it would be helpful to have a same mechanism for packaging and loading.
R3: In advanced use cases, we want to be able to bundle multiple models into a single shared library.
After discussion, we have sorted out the API and reach an agreement. Here, I want to summary the API and give it an example.
# lib is a GraphRuntimeFactoryModule# that contains json and parameterslib=relay.build(...)
# We provide way to let user control whether we# want to package_params or not. # You could save it like beforelib_no_params=lib["remove_params"]()
withopen(temp.relpath("deploy_param.params"), "wb") asfo:
fo.write(relay.save_param_dict(lib.get_params()))
# we could export it to shared library and load it backlib.export_library("resnet18.so")
# lib_no_params.export_library("resnet18.so")# load it backlib=tvm.module.load("resnet18.so")
# Call into the factory module to create a graph runtime# Having this additional factory create step solves R1# Note that parameters are already set# The first argument is a key that helps to solve R3, allow list of context in the future# gmod = lib["resnet18"]([tvm.cpu(0), tvm.gpu(0)])gmod=lib["resnet18"](tvm.cpu(0))
set_input=gmod["set_input"]
run=gmod["run"]
get_output=gmod["get_output"]
# We do not need to set the parameters here# as the modelsset_input(data=my_data)
run()
get_output()
# we could use wrappergmod=graph_runtime.GraphModule(lib["resnet18"](tvm.cpu(0)))
gmod.set_input(data=my_data)
gmod.run()
gmod.get_output()
As https://discuss.tvm.ai/t/discuss-module-based-model-runtime-interface/5025 discussed, we want to support Module based Model Runtime Interface and solve the following challenges:
After discussion, we have sorted out the API and reach an agreement. Here, I want to summary the API and give it an example.
More details and the decision procedure could be seen: https://discuss.tvm.ai/t/discuss-module-based-model-runtime-interface/5025
The text was updated successfully, but these errors were encountered: