-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Apply mlagility to create ONNX model in CI #605
Merged
Merged
Changes from 45 commits
Commits
Show all changes
57 commits
Select commit
Hold shift + click to select a range
a5a82ef
apply mlagility to create ONNX model in CI
jcwchen 9db5fb2
correct path
jcwchen 65c5d7e
which benchit
jcwchen d913609
cd mlagility
jcwchen ccbaf72
pip install -e .
jcwchen b24081d
add models
jcwchen acf2326
../mlagility_models/alexnet_torch_hub_2891f54c/onnx/
jcwchen 2731ca4
git diff --exit-code -- .
jcwchen 398b63d
add opset 17 models
jcwchen f6c54a1
add faster-rcnn in CIs
jcwchen 0c06ef3
cd
jcwchen ac5f5ec
use 18 instead of 17
jcwchen 05c495d
Merge branch 'jcw/add-mlagility' of https://github.com/jcwchen/models…
jcwchen 1a583ec
test opset_version 18
jcwchen 1f1db90
git diff subdirectory
jcwchen 500b542
torch==2.0.0 torchvision==0.15.1
jcwchen ab889ae
run_mlagility.py
jcwchen 0380b2f
use better path to store .onnx
jcwchen 6ff1014
print exception
jcwchen db69afb
copy
jcwchen a0f0339
+base_name
jcwchen dd41d77
correct path from mlagility
jcwchen 2795d20
add another way to upload new model
jcwchen 37cf55e
validate_model
jcwchen cea6c52
import os.path as osp
jcwchen 19f6bc3
correct path
jcwchen 934765a
from pathlib import Path
jcwchen a6f9618
pip install -r
jcwchen fa5d290
[]
jcwchen 00e4e01
fix
jcwchen 5edb28f
dir_path
jcwchen 388806f
correct path
jcwchen 943fc34
add bart.py
jcwchen d789edd
change name
jcwchen 3b34a19
update path
jcwchen ff87370
bart
jcwchen 28c7da1
not os.path.exists(model_name)
jcwchen d755004
"18"
jcwchen 9eff439
correct
jcwchen e0f0ff0
change path
jcwchen 358887a
add new-models to CI trigger
jcwchen bec94e5
add test_data_set
jcwchen 2e99292
add .yml
jcwchen 9a1c97a
ls
jcwchen 6192ee7
stdout=sys.stdout
jcwchen 53b3c98
change commit
jcwchen 0158b32
ls .cache
jcwchen 6196319
ls .cache/alexnet_torch_hub_7df2a577/onnx
jcwchen 2f6d3ee
ls
jcwchen cfeacf1
stdout=sys.stdout
jcwchen 38d44af
torch==2.0.0
jcwchen d7d3d27
mlagility_models_dir
jcwchen 7af0369
new config
jcwchen 355c6c3
add 3 more models
jcwchen 4dae47a
add config
jcwchen 0e1a570
remove bert and gpt2 for now
jcwchen 0ef4ef4
clean path
jcwchen File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
name: Validate created ONNX model from mlagility | ||
|
||
on: | ||
push: | ||
branches: [ main, new-models] | ||
pull_request: | ||
branches: [ main, new-models] | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
python-version: ['3.8'] | ||
|
||
steps: | ||
- uses: actions/checkout@v3 | ||
name: Checkout repo | ||
- uses: conda-incubator/setup-miniconda@v2 | ||
with: | ||
miniconda-version: "latest" | ||
activate-environment: mla | ||
python-version: ${{ matrix.python-version }} | ||
|
||
- name: Install dependencies and mlagility | ||
run: | | ||
python -m pip install --upgrade pip | ||
python -m pip install onnx onnxruntime requests py-cpuinfo | ||
# Print CPU info for debugging ONNX Runtime inference difference | ||
python -m cpuinfo | ||
# or python -m pip install mlagility | ||
python -m pip install transformers | ||
git clone https://github.com/groq/mlagility.git | ||
cd mlagility | ||
pip install -r models/requirements.txt | ||
pip install -e . | ||
|
||
- name: Validate created ONNX model from mlagility | ||
run: | | ||
pip install -r models/mlagility/requirements.txt | ||
python workflow_scripts/run_mlagility.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Git LFS file not shown
Git LFS file not shown
Git LFS file not shown
3 changes: 3 additions & 0 deletions
3
models/mlagility/fasterrcnn_resnet50_fpn_v2-18/fasterrcnn_resnet50_fpn_v2-18.onnx
Git LFS file not shown
3 changes: 3 additions & 0 deletions
3
models/mlagility/fasterrcnn_resnet50_fpn_v2-18/test_data_set_0/input_0.pb
Git LFS file not shown
3 changes: 3 additions & 0 deletions
3
models/mlagility/fasterrcnn_resnet50_fpn_v2-18/test_data_set_0/output_0.pb
Git LFS file not shown
3 changes: 3 additions & 0 deletions
3
models/mlagility/fasterrcnn_resnet50_fpn_v2-18/test_data_set_0/output_1.pb
Git LFS file not shown
3 changes: 3 additions & 0 deletions
3
models/mlagility/fasterrcnn_resnet50_fpn_v2-18/test_data_set_0/output_2.pb
Git LFS file not shown
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
models_info = [ | ||
# (script_path, model_name, model_zoo_path) | ||
("torch_hub/alexnet.py", "alexnet_torch_hub_2891f54c", "alexnet-18/alexnet-18.onnx"), | ||
("torchvision/fasterrcnn_resnet50_fpn_v2.py", "fasterrcnn_resnet50_fpn_v2_torchvision_ae446d48", "faster-rcnn/fasterrcnn_resnet50_fpn_v2.onnx"), | ||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
import config | ||
import os.path as osp | ||
from pathlib import Path | ||
import shutil | ||
import subprocess | ||
import sys | ||
|
||
|
||
base_name = "-op18-base.onnx" | ||
cwd_path = Path.cwd() | ||
mlagility_root = "mlagility/models" | ||
mlagility_models_dir = ".cache" | ||
ZOO_OPSET_VERSION = "18" | ||
|
||
errors = 0 | ||
|
||
for script_path, model_name, model_zoo_path in config.models_info: | ||
try: | ||
final_model_path = osp.join(mlagility_models_dir, model_zoo_path.replace(".onnx", "-" + ZOO_OPSET_VERSION + ".onnx")) | ||
subprocess.run(["benchit", osp.join(mlagility_root, script_path), "--cache-dir", mlagility_models_dir, "--onnx-opset", ZOO_OPSET_VERSION], | ||
cwd=cwd_path, stdout=sys.stdout, | ||
stderr=stdout=sys.stdout) | ||
|
||
subprocess.run(["ls", ".cache"], cwd=cwd_path, stdout=stdout=sys.stdout, stderr=stdout=sys.stdout) | ||
shutil.copy(osp.join(mlagility_models_dir, model_name, "onnx", model_name + base_name), final_model_path) | ||
subprocess.run(["git", "diff", "--exit-code", "--", final_model_path], | ||
cwd=cwd_path, stdout=subprocess.PIPE, | ||
stderr=subprocess.PIPE) | ||
print(f"Successfully checked {model_zoo_path}.") | ||
except Exception as e: | ||
errors += 1 | ||
print(f"Failed to check {model_zoo_path} because of {e}.") | ||
|
||
if errors > 0: | ||
print(f"All {len(config.models_info)} model(s) have been checked, but {errors} model(s) failed.") | ||
|
||
sys.exit(1) | ||
else: | ||
print(f"All {len(config.models_info)} model(s) have been checked.") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Line 36 installs both onnx and onnxruntime. Do we need a separate pip install here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess you are talking about this requirements: https://github.com/groq/mlagility/blob/main/models/requirements.txt, but actually onnx and onnxruntime are not there?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Installation of MLAgility here installs onnx and onnxruntime