Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Include SAM Fast in torch.compile nightly benchmark workflow #2856

Merged
merged 11 commits into from
Dec 19, 2023
Merged

Include SAM Fast in torch.compile nightly benchmark workflow #2856

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
085eb8c
Include SAM Fast in torch.compile nightly benchmark workflow
Dec 18, 2023
bcda479
Update path of SAM dependency file
Dec 18, 2023
156352c
Update Python version
Dec 18, 2023
b6eb538
Install chardet
Dec 18, 2023
684b99e
Disable BERT and ResNet benchmarks for test
Dec 18, 2023
c695afd
Retrigger tests
Dec 18, 2023
48e872f
Retrigger tests
Dec 18, 2023
301ac9c
Revert changes made for SAM fast test
Dec 19, 2023
d12c116
Rerun SAM Fast benchmark
Dec 19, 2023
e89c1ba
Retrigger tests
Dec 19, 2023
e65e083
Revert changes made for SAM Fast benchmark test
Dec 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .github/workflows/benchmark_torch_compile_nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ jobs:
cd ..
pwd
rm -rf _tool
- name: Setup Python 3.8
- name: Setup Python 3.10
uses: actions/setup-python@v4
with:
python-version: 3.8
python-version: 3.10.9
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just have 3.10?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the previous runs had failed when I specified just 3.10: https://github.com/pytorch/serve/actions/runs/7255048620/job/19764929373

Commit ID: bcda479

We can choose the versions according to https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to use quotes, check this workflow https://github.com/pytorch/serve/blob/master/.github/workflows/regression_tests_cpu_binaries.yml#L20

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I understand. I will update the workflow accordingly.

architecture: x64
- name: Setup Java 17
uses: actions/setup-java@v3
Expand All @@ -39,5 +39,8 @@ jobs:
sudo apt-get update -y
sudo apt-get install -y apache2-utils
pip install -r benchmarks/requirements-ab.txt
chmod +x examples/large_models/segment_anything_fast/install_segment_anything_fast.sh
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make this a separate step

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will update the PR.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking this should be a script under benchmarks. You can call it benchmark_model_dependencies.sh . So, if there are more things in the future, we can add it there

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I will create this script.

source examples/large_models/segment_anything_fast/install_segment_anything_fast.sh
export SEGMENT_ANYTHING_FAST_USE_FLASH_4=0
- name: Benchmark gpu nightly
run: python benchmarks/auto_benchmark.py --input benchmarks/benchmark_config_torch_compile_gpu.yaml --skip false --nightly True
1 change: 1 addition & 0 deletions benchmarks/benchmark_config_torch_compile_gpu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
models:
- "bert_torch_compile_gpu.yaml"
- "resnet50_torch_compile_gpu.yaml"
- "sam_fast_torch_compile_gpu.yaml"

# benchmark on "cpu" or "gpu".
# "cpu" is set if "hardware" is not specified
Expand Down
18 changes: 18 additions & 0 deletions benchmarks/models_config/sam_fast_torch_compile_gpu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
sam_fast_vit_b:
process_batch_size_4:
benchmark_engine: "ab"
url: https://torchserve.pytorch.org/mar_files/sam-fast-vit-b-process-batch-size-4.tar.gz
agunapal marked this conversation as resolved.
Show resolved Hide resolved
workers:
- 4
batch_delay: 100
batch_size:
- 1
input: "./examples/large_models/segment_anything_fast/kitten.jpg"
requests: 1000
concurrency: 4
backend_profiling: False
exec_env: "local"
processors:
- "cpu"
- "gpus": "all"
1 change: 1 addition & 0 deletions examples/large_models/segment_anything_fast/custom_handler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ def initialize(self, ctx):
self.device = torch.device(
self.map_location + ":" + str(properties.get("gpu_id"))
)
torch.cuda.set_device(self.device)

model_type = ctx.model_yaml_config["handler"]["model_type"]
sam_checkpoint = ctx.model_yaml_config["handler"]["sam_checkpoint"]
Expand Down
1 change: 1 addition & 0 deletions examples/large_models/segment_anything_fast/install_segment_anything_fast.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
pip uninstall torchtext torchdata torch torchvision torchaudio -y

# Install nightly PyTorch and torchvision from the specified index URL
pip install chardet
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this for ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Included this dependency because one of the previous benchmark runs failed because of the absence of this module: https://github.com/pytorch/serve/actions/runs/7255062486/job/19764967013

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this happening locally? I didnt notice this. If not, can you please move this to the dependencies section of the benchmark run?

Copy link
Collaborator Author

@sachanub sachanub Dec 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I noticed it locally as well. I will include it in the benchmark_model_dependencies.sh script which you suggested.

pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121 --ignore-installed

# Optional: Display the installed PyTorch and torchvision versions
Expand Down
Loading