Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add performance benchmark tests #98

Open
hatemhelal opened this issue Sep 20, 2023 · 2 comments
Open

Add performance benchmark tests #98

hatemhelal opened this issue Sep 20, 2023 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@hatemhelal
Copy link
Contributor

As the project evolves and we add more features it will be important to continually check that we aren't regressing on performance metrics. This issue can be used to discuss what we consider important performance metrics, here are a few that have been discussed previously:

  • C20, this seems to be used externally as a benchmark molecule
  • DFT throughput, e.g. number of single point DFT evaluations we can perform per second per chip

Beyond this we may want some microbenchmarks for performance critical steps in the SCF algorithm (e.g. eigensolve, evaluation of ERI)

Ideally we would track this continuously for every PR so we are getting feedback closer to the time when possible regressions are introduced. This may require we run these checks on IPU hardware. Further requirements to be scoped as part of addressing this issue.

@hatemhelal hatemhelal added the enhancement New feature or request label Sep 20, 2023
@AlexanderMath
Copy link
Contributor

AlexanderMath commented Oct 4, 2023

Write a "--benchmark_tracking True" argparse argument.

Goals:

  • Report cycle count of different molecules (@balancap ).
  • Keep compile time < 30min.

Molecules/hw configuration

  • Do C20 (single IPU) -eri_threshold 1e-9 -dense_ERI false
  • Do C100 (pod16) -eri_threshold 1e-9 -dense_ERI -false
  • SPICE dipeptide w/ N=1024 on 16 IPUs.

@hatemhelal
Copy link
Contributor Author

We should look at this action:

https://github.com/benchmark-action/github-action-benchmark

@awf awf mentioned this issue Oct 18, 2023
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants