Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support workflow in benchmark-ab.py #1445

Merged
merged 6 commits into from
Feb 23, 2022
Merged

support workflow in benchmark-ab.py #1445

merged 6 commits into from
Feb 23, 2022

Conversation

lxning
Copy link
Collaborator

@lxning lxning commented Feb 18, 2022 •
edited

Description

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #(issue)
#1443

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Feature/Issue validation/testing

Please describe the tests [UT/IT] that you ran to verify your changes and relevent result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

  • Test A

  • Test B

  • UT/IT execution results

  • Logs
    python ./benchmarks/benchmark-ab.py --config_properties ./benchmarks/config.p

Configured execution parameters are:
{'url': 'https://torchserve.pytorch.org/war_files/dog_breed_wf.war', 'gpus': '', 'exec_env': 'local', 'batch_size': 1, 'batch_delay': 100, 'workers': 2, 'concurrency': 10, 'requests': 1000, 'input': './docs/images/kitten_small.jpg', 'content_type': 'application/jpg', 'image': '', 'docker_runtime': '', 'backend_profiling': False, 'config_properties': './benchmarks/config.properties', 'inference_model_url': 'predictions/benchmark', 'report_location': '/tmp', 'inference_url': 'http://0.0.0.0:8080', 'management_url': 'http://0.0.0.0:8081', 'config_properties_name': 'config.properties'}

Preparing local execution...
*Terminating any existing Torchserve instance ...
torchserve --stop
TorchServe has stopped.
*Setting up model store...
*Starting local Torchserve instance...
torchserve --start --model-store /tmp/model_store --workflow-store /tmp/wf_store --ts-config /tmp/benchmark/conf/config.properties > /tmp/benchmark/logs/model_metrics.log
*Testing system health...
{
"status": "Healthy"
}

*Registering model...
{
"status": "Workflow benchmark has been registered and scaled successfully."
}

Executing warm-up ...
ab -c 10 -n 100.0 -k -p /tmp/benchmark/input -T application/jpg http://0.0.0.0:8080/wfpredict/benchmark > /tmp/benchmark/result.txt

Executing inference performance tests ...
ab -c 10 -n 1000 -k -p /tmp/benchmark/input -T application/jpg http://0.0.0.0:8080/wfpredict/benchmark > /tmp/benchmark/result.txt
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests

Checklist:

  • Have you added tests that prove your fix is effective or that this feature works?
  • New and existing unit tests pass locally with these changes?
  • Has code been commented, particularly in hard-to-understand areas?
  • Have you made corresponding changes to the documentation?

@lxning lxning self-assigned this Feb 18, 2022
@lxning lxning added benchmark enhancement New feature or request labels Feb 18, 2022
@lxning lxning added this to the v0.5.3 milestone Feb 18, 2022
@lxning lxning added this to In Review in v0.5.3 lifecycle Feb 18, 2022
@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-win
  • Commit ID: 5be8791
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: 5be8791
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: 5be8791
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-win
  • Commit ID: 83af112
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: 83af112
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: 83af112
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@lxning lxning mentioned this pull request Feb 21, 2022
Merged
8 tasks
maaquib
maaquib reviewed Feb 23, 2022
View reviewed changes
benchmarks/benchmark-ab.py Outdated
ab_cmd = f"ab -c {execution_params['concurrency']} -n {execution_params['requests']/10} -k -p {TMP_DIR}/benchmark/input -T " \
f"{execution_params['content_type']} {execution_params['inference_url']}/{execution_params['inference_model_url']} > {result_file}"

execute(ab_cmd, wait=True)


def run_benchmark():
if execution_params['url'].endswith('.war'):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extract this into a method is_workflow_run or something similar, so that in the future if we decide to change the extension for workflow artifacts we won't have to change all if-else statements

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-win
  • Commit ID: 7339ce6
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-win
  • Commit ID: 31fbce6
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: 7339ce6
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: 31fbce6
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: 7339ce6
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: 31fbce6
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@lxning lxning merged commit 1ea3eb3 into master Feb 23, 2022
@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-win
  • Commit ID: 31fbce6
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: 31fbce6
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maaquib maaquib maaquib approved these changes

@msaroufim msaroufim msaroufim approved these changes

Assignees

@lxning lxning

Labels
benchmark enhancement New feature or request
Projects
v0.5.3 lifecycle
In Review
Milestone
v0.5.3
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@lxning @sagemaker-neo-ci-bot @maaquib @msaroufim