-
Notifications
You must be signed in to change notification settings - Fork 180
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Return non-zero status code if there are any failures #449
Labels
enhancement
Triaged as an enhancement request
Comments
For sympy_benchmarks, we would also be interested in a
|
This would indeed be nice to ensure that benchmarks don't get stale because of changes in program code. |
crepererum
added a commit
to JDASoftwareGroup/kartothek
that referenced
this issue
May 8, 2019
Sadly, we cannot use ASV in CI because it would not report such failures. See airspeed-velocity/asv#449
mtreinish
added a commit
to mtreinish/asv
that referenced
this issue
Aug 15, 2019
This commit adds a new flag, --strict, to the run (and by extension dev) and continous commands. When set any benchmark failures encountered will be treated as an error and the command will exit non-zero when the run is finished. This is especially useful when running benchmarks in a CI system or similar automated environment and wish to know if a change was made to break a benchmark. Fixes airspeed-velocity#449
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For the astropy-benchmarks, I would like the Travis status to be a failure if there are any failures during the benchmarks (currently there are three benchmark failures but Travis passes). Maybe a flag could be added to this effect?
The text was updated successfully, but these errors were encountered: