New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add sample code for GenAI model inferencing #1131

Merged

jambayk merged 1 commit into main from shaahji/Task29521

May 9, 2024

Contributor

shaahji commented May 1, 2024

Add sample code for GenAI model inferencing

Also, update the packaging logic to include sample code and required runtime binaries.

NOTE: Native (both CPP & CS) binaries are not yet being published by the GenAI team and needs revisit once those binaries are available for download.

Checklist before requesting a review

Add unit tests for this change.
Make sure all tests can pass.
Update documents if necessary.
Lint and apply fixes to your code by running lintrunner -a
Is this a user-facing change? If yes, give a description of this change to be included in the release notes.
Is this PR including examples changes? If yes, please remember to update example documentation in a follow-up PR.

(Optional) Issue link

github-advanced-security bot found potential problems

View reviewed changes

olive/engine/packaging/sample_code/GenAIOnnxModel/python/code_sample.py Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems

View reviewed changes

olive/engine/packaging/sample_code/GenAIOnnxModel/cs/olive-genai-cs-sample.sln Fixed Show fixed Hide fixed

olive/engine/packaging/sample_code/GenAIOnnxModel/cs/olive-genai-cs-sample.sln Fixed Show fixed Hide fixed

olive/engine/packaging/sample_code/GenAIOnnxModel/cs/olive-genai-cs-sample.sln Fixed Show fixed Hide fixed

olive/engine/packaging/sample_code/GenAIOnnxModel/cs/olive-genai-cs-sample.sln Fixed Show fixed Hide fixed

olive/engine/packaging/sample_code/GenAIOnnxModel/cs/olive-genai-cs-sample.sln Fixed Show fixed Hide fixed

olive/engine/packaging/sample_code/GenAIOnnxModel/cs/olive-genai-cs-sample.sln Fixed Show fixed Hide fixed

olive/engine/packaging/sample_code/GenAIOnnxModel/cs/olive-genai-cs-sample.sln Fixed Show fixed Hide fixed

olive/engine/packaging/sample_code/GenAIOnnxModel/cs/olive-genai-cs-sample.sln Fixed Show fixed Hide fixed

olive/engine/packaging/sample_code/GenAIOnnxModel/cs/olive-genai-cs-sample.sln Fixed Show fixed Hide fixed

olive/engine/packaging/sample_code/GenAIOnnxModel/python/code_sample.py Fixed Show fixed Hide fixed

devang-ml reviewed

View reviewed changes

olive/engine/packaging/packaging_generator.py Outdated Show resolved Hide resolved

devang-ml requested a review from xiaoyu-work

May 2, 2024 18:26

shaahji force-pushed the shaahji/Task29521 branch 2 times, most recently from f799ca4 to 84d842a Compare

May 2, 2024 20:09

github-advanced-security bot found potential problems

View reviewed changes

olive/engine/packaging/sample_code/GenAIOnnxModel/python/code_sample.py Fixed Show fixed Hide fixed

shaahji force-pushed the shaahji/Task29521 branch from 84d842a to 7ab37b3 Compare

May 3, 2024 00:24

jambayk reviewed

View reviewed changes

.lintrunner.toml Outdated Show resolved Hide resolved

shaahji requested review from jambayk and xiaoyu-work and removed request for xiaoyu-work

May 3, 2024 21:36

Contributor

xiaoyu-work commented May 8, 2024

Can we also update unit tests for this?

xiaoyu-work reviewed

View reviewed changes

olive/engine/packaging/packaging_generator.py Outdated Show resolved Hide resolved

olive/engine/packaging/packaging_generator.py

+                  installed_packages = [
+                      pkg
+                      for pkg in pkg_resources.working_set
+                      if pkg.key.startswith("onnxruntime-genai") or pkg.project_name.startswith("onnxruntime-genai")

Contributor

xiaoyu-work May 8, 2024

Any difference between key and project_name?

Contributor Author

shaahji May 8, 2024 •

edited

I found some context here https://github.com/recipy/recipy/blob/d8f8fe8ace3659f1d700bb454e68a8db453e84f4/integration_test/environment.py#L125 which gives the impression that they could be different.

Contributor

xiaoyu-work May 9, 2024

Not a big deal for now, we could be more consistent in our code.

olive/engine/packaging/packaging_generator.py Outdated Show resolved Hide resolved


          Add sample code for GenAI model inferencing

73bef78

Also, update the packaging logic to include sample code and
required runtime binaries.

NOTE: Native (both CPP & CS) binaries are not yet being published
by the GenAI team and needs revisit once those binaries are
available for download.

shaahji force-pushed the shaahji/Task29521 branch from 7ab37b3 to 73bef78 Compare

May 8, 2024 21:46

Contributor Author

shaahji commented May 8, 2024

Can we also update unit tests for this?

I will revisit the unit test once the runtime headers/binaries are available from genai repo. There is also some cleanup work in the packaging unit test to make packaging and sample code testing possible in CI. There is an open task on Dashboard from grab.

xiaoyu-work approved these changes

View reviewed changes

olive/engine/packaging/packaging_generator.py

+                  installed_packages = [
+                      pkg
+                      for pkg in pkg_resources.working_set
+                      if pkg.key.startswith("onnxruntime-genai") or pkg.project_name.startswith("onnxruntime-genai")

Contributor

xiaoyu-work May 9, 2024

Not a big deal for now, we could be more consistent in our code.

jambayk merged commit 3e5ce3f into main

35 checks passed

jambayk deleted the shaahji/Task29521 branch

May 9, 2024 04:08

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment