Module Based Model Runtime for AOT #46
Conversation
|
Thanks! @areusch I ll try to do a pass soon. Not sure its in scope for this work, but definitely something to consider: w.r.t. MetaDataModule, if we are re-purposing it to be just ConstLoader, @ashutosh-arm has contributed pass that could kick constants back to the main function. https://github.com/apache/tvm/blob/main/src/relay/backend/contrib/cmsisnn/extract_constants.cc That could be optionally enabled for BYOC backends that does not like to handle constants. I think using that in the BYOC pipeline with -link-params should just work if I am not mistaken. Therefore, we would not need a Metadata module of type "metadata" that is used for constant storage. |
|
@manupa-arm thanks! I think ultimately the @ashutosh-arm pass could be a good way to implement ConstLoader, but for now I'm hoping we can simply rename. In the long run, I'd actually like to be able to do a similar thing with MetadataModule--define a special data type in Relay which holds this Metadata, and "link" tvm_struct_get calls against it just before codegen with an IRModule -> IRModule pass to replace such tvm_struct_get with the value from Metadata. I'm not sure what the best approach is here, probably need to collect some feedback from the community/users. However, at some level it all seems like metadata to me (constants could be params tho, and it may be desirable to be able to swap those params after compilation). |
|
@manupa-arm @comaniac @zhiics could you take a look here? |
|
|
||
| 3. AotExecutorCodegen and GraphExecutorCodegen have adopted the practice of producing the | ||
| graph-level | ||
| [`runtime::MetadataNode`](https://github.com/apache/tvm/blob/main/src/runtime/meta_data.h#L55): |
There was a problem hiding this comment.
FYI, I did move this to become ExecutorCodegenMetadata now as it did not feel like a runtime concept.
There was a problem hiding this comment.
ah updated the RFC text to point to that. thanks!
|
|
||
|
|
||
| Some duplication of information is already present. Likely this is due in part to the existing | ||
| middle-end compiler design, in which a separate `IRModule` is produced for each backend. Another |
There was a problem hiding this comment.
runtime.Module is produced per backend. As long as that exists, somewhere in the compiler per-backend IRModule have to exist -- that does the translation from IRModule --> runtime.Module.
However, we have hierarchical runtime.Module trees being built -- so what we can eventually do maybe create hierarchical IRModule structure. In that structure, it might make sense to keep this metadata.
There was a problem hiding this comment.
a couple points:
- more than one
runtime.Modulecan be produced by a given backend - depending on the load path in C++ runtime, the tree can become rearranged. see https://discuss.tvm.apache.org/t/introduce-artifact-a-container-for-generated-code/11099
There was a problem hiding this comment.
Agreed.
The requirement to mention this here seems like stemming from the fact it is aligned with other RFC/pre-RFCs that single (tree of) IRModule --> (list of) runtime.Modules -- is a proposed change -- therefore, this motivates the change proposed here.
Thus, would it be possible to add a reference to this proposal (if any)?
There was a problem hiding this comment.
hmm, i'm not quite sure i follow you here. I'm happy to add a reference to the Artifact proposal, but I'm not sure it's quite exactly what I'm stating here. Here, what I mean is that the TIR-to-Runtime interface is IRModule -> (tree of runtime.Module). The existing MetadataModule (which is proposed to rename to ConstLoaderModule here) seems to have arisen out of a desire to build common infrastructure to handle loading DLTensor from .text in the C++ runtime. Here what I'm trying to point out is that since the TIR-to-Runtime interface provides no facility for the TIR-to-runtime processes to return metadata outside of the runtime::Module, this leads to duplication of information should it be required by the compiler in any way. For example, constant_sizes could be deduced from the DLTensor passed to ConstLoaderModule, but ConstLoaderModule is not supported by all runtimes and not the de-facto way to load constant data or metadata at runtime because it doesn't support encoding structs and scalar values. You can also see some duplication in CudaModule.
I think this proposal is attempting to start down the path of unifying these different methods of providing data generated during lowering to the runtime as Metadata. I think that's mainly covered here, but happy to add a reference to the Artifact thing if it helps, it just seems a bit orthogonal to me.
There was a problem hiding this comment.
I think I dont dispute the claim that IRModule --> (tree of runtime.Module).
The current lowering flow creates IRModule per backend that get translated to runtime.Module(s).
I see that as (correct me if I am wrong) : Unified IRModule --> [IRModule per backend] --> tree of runtime.Module s
which is basically a host runtime.Module includes a flat array device runtime.Modules.
Now the proposal here want to attach model-level Metadata from the Unified IRModule to (root of) tree of runtime.Module s.
So the gap in the text here is that the text assumes it is common knowledge that how [IRModule per backend] disappears. Hypothetically, lets say it is not there -- then text make sense because the model-level metadata could be attached to Unified IRModule and then passed onto tree of runtime.Module s.
Now my questions is, in the absense of a proposal how to remove the IRModule per backend (or if its already there -- please link the RFC/pre-RFC), this RFC needs to outline a way how this will be communicated from Unified IRModule to tree of runtime.Module s in the current lowering flow and/or the changes brought by this RFC.
There was a problem hiding this comment.
I see--so this is more about the wording a couple paragraphs down, no?
Work is currently ongoing to unify the pre-codegen IRModule into a single instance. After this work is completed, it will be much easier to produce a centralized module-level Metadata.
cc @jroesch i am actually not sure if there is an RFC describing this.
i'm hoping to describe my ambitions to "link" metadata into TIR via tir.load_metadata node in a following RFC, and I definitely would need consolidated metadata for this at the IRModule level. I'm not sure if there is anything in code-generation that strictly requires this--it's just cleaner in my book. Let me know if I'm missing anything here.
There was a problem hiding this comment.
per our offline discussion, clarified how the metadata is carried through the current compiler design, removed references to un-RFC'd design efforts and replaced with text to motivate them. also clarified some wording--ptal
| Some duplication of information is already present. Likely this is due in part to the existing | ||
| middle-end compiler design, in which a separate `IRModule` is produced for each backend. Another | ||
| factor may be: since `runtime::Module` are responsible for their own serialization, and passing | ||
| `Node` across `PackedFunc` requires a cast, the lack of a centralized facility for |
There was a problem hiding this comment.
What is meant by Node here ?
There was a problem hiding this comment.
tvm::Node; clarified
|
|
||
| Work is currently ongoing to unify the pre-codegen `IRModule` into a single instance. After this | ||
| work is completed, it will be much easier to produce a centralized module-level Metadata. This RFC | ||
| argues for the expansion of `runtime::MetadataNode` in the following ways: |
There was a problem hiding this comment.
I think the following arguments argues for re-structure (not just runtime::MetadataNode).
There was a problem hiding this comment.
I dont see how 1.) is relevant to the expansion of runtime::MetadataNode.
My comment was to adjust the text here to say something like "modify the lowering flow" rather than "the expansion of runtime::MetadataNode" -- because latter is a sub-part of the former and I believe former is what is proposed here and covered by the following points.
There was a problem hiding this comment.
ah i see, changed the wording. i guess i was trying to say that the name MetadataNode should cover a more complex thing, but in practice you're right that this proposes a restructuring.
|
|
||
| 1. Rename `runtime::MetadataModule` to `runtime::ConstLoaderModule` to disambiguate the two and make | ||
| its purpose in life clearer. | ||
| 2. Expand `input_args` in the existing `runtime::Metadata` to parity with `runtime::FunctionInfo`, |
There was a problem hiding this comment.
I believe you are referring to runtime::MetadataNode here.
This is now : https://github.com/apache/tvm/blob/e6af87491eb250a3266b1f09b36055c6ee79146b/src/relay/backend/utils.h#L60-L84
There was a problem hiding this comment.
ah yeah. i think perhaps this is done now, then, since ExecutorCodegenMetadata exports tir::Var. However, to then fit this proposal we need to reduce it to something exportable. updated the text to clarify, see what you think.
There was a problem hiding this comment.
thanks @manupa-arm ! please take another look when you have a minute.
|
|
||
| 3. AotExecutorCodegen and GraphExecutorCodegen have adopted the practice of producing the | ||
| graph-level | ||
| [`runtime::MetadataNode`](https://github.com/apache/tvm/blob/main/src/runtime/meta_data.h#L55): |
There was a problem hiding this comment.
ah updated the RFC text to point to that. thanks!
|
|
||
|
|
||
| Some duplication of information is already present. Likely this is due in part to the existing | ||
| middle-end compiler design, in which a separate `IRModule` is produced for each backend. Another |
There was a problem hiding this comment.
a couple points:
- more than one
runtime.Modulecan be produced by a given backend - depending on the load path in C++ runtime, the tree can become rearranged. see https://discuss.tvm.apache.org/t/introduce-artifact-a-container-for-generated-code/11099
| Some duplication of information is already present. Likely this is due in part to the existing | ||
| middle-end compiler design, in which a separate `IRModule` is produced for each backend. Another | ||
| factor may be: since `runtime::Module` are responsible for their own serialization, and passing | ||
| `Node` across `PackedFunc` requires a cast, the lack of a centralized facility for |
There was a problem hiding this comment.
tvm::Node; clarified
|
|
||
| Work is currently ongoing to unify the pre-codegen `IRModule` into a single instance. After this | ||
| work is completed, it will be much easier to produce a centralized module-level Metadata. This RFC | ||
| argues for the expansion of `runtime::MetadataNode` in the following ways: |
|
|
||
| 1. Rename `runtime::MetadataModule` to `runtime::ConstLoaderModule` to disambiguate the two and make | ||
| its purpose in life clearer. | ||
| 2. Expand `input_args` in the existing `runtime::Metadata` to parity with `runtime::FunctionInfo`, |
There was a problem hiding this comment.
ah yeah. i think perhaps this is done now, then, since ExecutorCodegenMetadata exports tir::Var. However, to then fit this proposal we need to reduce it to something exportable. updated the text to clarify, see what you think.
|
|
||
|
|
||
| Some duplication of information is already present. Likely this is due in part to the existing | ||
| middle-end compiler design, in which a separate `IRModule` is produced for each backend. Another |
There was a problem hiding this comment.
Agreed.
The requirement to mention this here seems like stemming from the fact it is aligned with other RFC/pre-RFCs that single (tree of) IRModule --> (list of) runtime.Modules -- is a proposed change -- therefore, this motivates the change proposed here.
Thus, would it be possible to add a reference to this proposal (if any)?
|
|
||
| Work is currently ongoing to unify the pre-codegen `IRModule` into a single instance. After this | ||
| work is completed, it will be much easier to produce a centralized module-level Metadata. This RFC | ||
| argues for the expansion of `runtime::MetadataNode` in the following ways: |
There was a problem hiding this comment.
I dont see how 1.) is relevant to the expansion of runtime::MetadataNode.
My comment was to adjust the text here to say something like "modify the lowering flow" rather than "the expansion of runtime::MetadataNode" -- because latter is a sub-part of the former and I believe former is what is proposed here and covered by the following points.
There was a problem hiding this comment.
thanks @manupa-arm, clarified according to your comments here. PTAL!
|
|
||
|
|
||
| Some duplication of information is already present. Likely this is due in part to the existing | ||
| middle-end compiler design, in which a separate `IRModule` is produced for each backend. Another |
There was a problem hiding this comment.
hmm, i'm not quite sure i follow you here. I'm happy to add a reference to the Artifact proposal, but I'm not sure it's quite exactly what I'm stating here. Here, what I mean is that the TIR-to-Runtime interface is IRModule -> (tree of runtime.Module). The existing MetadataModule (which is proposed to rename to ConstLoaderModule here) seems to have arisen out of a desire to build common infrastructure to handle loading DLTensor from .text in the C++ runtime. Here what I'm trying to point out is that since the TIR-to-Runtime interface provides no facility for the TIR-to-runtime processes to return metadata outside of the runtime::Module, this leads to duplication of information should it be required by the compiler in any way. For example, constant_sizes could be deduced from the DLTensor passed to ConstLoaderModule, but ConstLoaderModule is not supported by all runtimes and not the de-facto way to load constant data or metadata at runtime because it doesn't support encoding structs and scalar values. You can also see some duplication in CudaModule.
I think this proposal is attempting to start down the path of unifying these different methods of providing data generated during lowering to the runtime as Metadata. I think that's mainly covered here, but happy to add a reference to the Artifact thing if it helps, it just seems a bit orthogonal to me.
|
|
||
| Work is currently ongoing to unify the pre-codegen `IRModule` into a single instance. After this | ||
| work is completed, it will be much easier to produce a centralized module-level Metadata. This RFC | ||
| argues for the expansion of `runtime::MetadataNode` in the following ways: |
There was a problem hiding this comment.
ah i see, changed the wording. i guess i was trying to say that the name MetadataNode should cover a more complex thing, but in practice you're right that this proposes a restructuring.
There was a problem hiding this comment.
Hi Andrew,
Thanks for the discussion, it was thought provoking :).
I have only a single comment asking for clarification about the lowering flow change that is brought by this RFC to attach model-level metadata to the tree of runtime.Modules.
Other than that this makes sense!
|
|
||
|
|
||
| Some duplication of information is already present. Likely this is due in part to the existing | ||
| middle-end compiler design, in which a separate `IRModule` is produced for each backend. Another |
There was a problem hiding this comment.
I think I dont dispute the claim that IRModule --> (tree of runtime.Module).
The current lowering flow creates IRModule per backend that get translated to runtime.Module(s).
I see that as (correct me if I am wrong) : Unified IRModule --> [IRModule per backend] --> tree of runtime.Module s
which is basically a host runtime.Module includes a flat array device runtime.Modules.
Now the proposal here want to attach model-level Metadata from the Unified IRModule to (root of) tree of runtime.Module s.
So the gap in the text here is that the text assumes it is common knowledge that how [IRModule per backend] disappears. Hypothetically, lets say it is not there -- then text make sense because the model-level metadata could be attached to Unified IRModule and then passed onto tree of runtime.Module s.
Now my questions is, in the absense of a proposal how to remove the IRModule per backend (or if its already there -- please link the RFC/pre-RFC), this RFC needs to outline a way how this will be communicated from Unified IRModule to tree of runtime.Module s in the current lowering flow and/or the changes brought by this RFC.
|
Thanks @areusch . Looking forward for AoT being enabled in more places :). |
|
Thanks @manupa-arm for reviewing this RFC and providing crucial feedback in time-bound fashion. |
This adds the RFC for MBMR for AOT. See also discussion on the Discuss forum: https://discuss.tvm.apache.org/t/aot-module-based-model-runtime-interface-for-aot/11068
@tqchen @manupa-arm @jroesch @mehrdadh @csullivan @adstraw @Mousius @kparzysz-quic