Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improvements to pytest --benchmark option. #209

Open
11 of 13 tasks
monorimet opened this issue Jul 23, 2022 · 0 comments
Open
11 of 13 tasks

Improvements to pytest --benchmark option. #209

monorimet opened this issue Jul 23, 2022 · 0 comments
Assignees
@monorimet
Labels
documentation Improvements or additions to documentation enhancement New feature or request

Comments

@monorimet
Copy link
Collaborator

monorimet commented Jul 23, 2022 •
edited

Several features/improvements to SHARK's pytest --benchmark option are tracked in this issue:

  • Improve "frontend" / MLIR dialect argument transmission through SharkBenchmarkRunner
  • Verify benchmark results for PyTorch+CUDA on Vision Models.
  • Benchmarks in CI should upload bench_results.csv to gs://iree-shared-files/nod-perf/bench_results/{Y-M-D}/bench_results_{cpu/gpu}_{github-SHA}.csv (Upload benchmark results for every test-models workflow (excl. Vulkan) #241 )
  • Update README with benchmarking instructions. (Update README.md #239 )
  • Enable pytest --benchmark for TensorFlow shark tank module tests.
  • Add options to setup_venv.sh for ONNX benchmarking requirements
  • Benchmarks should be able to produce ONNX results and provide better data in generated results. (see: nod-ai/transformer-benchmarks)
  • Make benchmark results more accessible -- upload to gs://shark-public/builder/...
  • Thread counts
  • save compile-time flags
  • useful logs, traces, etc.
  • metadata
  • comparison %'s
@monorimet monorimet added documentation Improvements or additions to documentation enhancement New feature or request labels Jul 23, 2022
@monorimet monorimet self-assigned this Jul 23, 2022
@monorimet monorimet changed the title Add more data to test benchmark results, and enable --benchmark for TF test runs. Improvements to pytest --benchmark option. Aug 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@monorimet monorimet

Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@monorimet