[aot_inductor] Lightweight model runner #110158
Conversation
It's useful to have a simple, lightweight way to run a model that adds essentially no overhead to calling the model's generated `run_impl` method. This C API is a super thin wrapper around AOTInductorModel: Create, Run, and Delete are provided, and do very little work beyond dispatch to the appropriate helpers. Note the Create function also provides additional functionality beyond the Container API; it allows the user to pass in a weight map defined in userland, which is a requirement for several serving use cases. Differential Revision: [D49670711](https://our.internmc.facebook.com/intern/diff/D49670711/) [ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/110158
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 54 PendingAs of commit 7554786 with merge base 92d86cd ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
It's useful to have a simple, lightweight way to run a model that adds essentially no overhead to calling the model's generated `run_impl` method. This C API is a super thin wrapper around AOTInductorModel: Create, Run, and Delete are provided, and do very little work beyond dispatch to the appropriate helpers. Note the Create function also provides additional functionality beyond the Container API; it allows the user to pass in a weight map defined in userland, which is a requirement for several serving use cases. Differential Revision: [D49670711](https://our.internmc.facebook.com/intern/diff/D49670711/) ghstack-source-id: 202095319 Pull Request resolved: #110158
| @@ -112,4 +118,15 @@ AOTIRuntimeError AOTInductorModelContainerGetMaxOutputShape( | |||
| const int64_t** ret_output_sizes, | |||
| int64_t* ret_output_ndim); | |||
|
|
|||
| AOTIRuntimeError AOTInductorModelCreate( | |||
There was a problem hiding this comment.
I would prefer to have the naming more explicitly state how it is different from AOTInductorModelContainerCreate, or at least add comments here to explain that.
There was a problem hiding this comment.
Can you suggest a name? IMO this is pretty self explanatory so I'm struggling to come up with something satisfactory.
There was a problem hiding this comment.
Perhaps we could just add comments to explain when the user would prefer to use this function, e.g. like those sentences from the summary?
…"[aot_inductor] Lightweight model runner" It's useful to have a simple, lightweight way to run a model that adds essentially no overhead to calling the model's generated `run_impl` method. This C API is a super thin wrapper around AOTInductorModel: Create, Run, and Delete are provided, and do very little work beyond dispatch to the appropriate helpers. Note the Create function also provides additional functionality beyond the Container API; it allows the user to pass in a weight map defined in userland, which is a requirement for several serving use cases. Differential Revision: [D49670711](https://our.internmc.facebook.com/intern/diff/D49670711/) cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 aakhundov ColinPeppler [ghstack-poisoned]
Pull Request resolved: #110158 It's useful to have a simple, lightweight way to run a model that adds essentially no overhead to calling the model's generated `run_impl` method. This C API is a super thin wrapper around AOTInductorModel: Create, Run, and Delete are provided, and do very little work beyond dispatch to the appropriate helpers. Note the Create function also provides additional functionality beyond the Container API; it allows the user to pass in a weight map defined in userland, which is a requirement for several serving use cases. ghstack-source-id: 202242949 @exported-using-ghexport Differential Revision: [D49670711](https://our.internmc.facebook.com/intern/diff/D49670711/)
|
@pytorchbot merge -f 'Landed internally' (Initiating merge automatically since Phabricator Diff has merged, using force because this PR might not pass merge_rules.json but landed internally) |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Stack from ghstack (oldest at bottom):
It's useful to have a simple, lightweight way to run a model that adds
essentially no overhead to calling the model's generated
run_implmethod.This C API is a super thin wrapper around AOTInductorModel: Create, Run, and
Delete are provided, and do very little work beyond dispatch to the appropriate
helpers.
Note the Create function also provides additional functionality beyond the
Container API; it allows the user to pass in a weight map defined in userland,
which is a requirement for several serving use cases.
Differential Revision: D49670711
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler