Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return non-zero status code if there are any failures #449

Closed
astrofrog opened this issue Jun 26, 2016 · 2 comments · Fixed by #865
Closed

Return non-zero status code if there are any failures #449

astrofrog opened this issue Jun 26, 2016 · 2 comments · Fixed by #865
Labels
enhancement Triaged as an enhancement request

Comments

@astrofrog
Copy link
Contributor

For the astropy-benchmarks, I would like the Travis status to be a failure if there are any failures during the benchmarks (currently there are three benchmark failures but Travis passes). Maybe a flag could be added to this effect?

@pv pv added the enhancement Triaged as an enhancement request label Jun 29, 2016
@bjodah
Copy link

bjodah commented Jul 28, 2016

For sympy_benchmarks, we would also be interested in a asv --strict flag or something along those lines. But in that case we would also like to be able to mark some legacy benchmarks as allowed failures, e.g. along the lines of:

@asv.mark.may_fail
def time_some_old_method():
    ...

@pitrou
Copy link
Contributor

pitrou commented Apr 10, 2018

This would indeed be nice to ensure that benchmarks don't get stale because of changes in program code.

crepererum added a commit to JDASoftwareGroup/kartothek that referenced this issue May 8, 2019
Sadly, we cannot use ASV in CI because it would not report such
failures. See airspeed-velocity/asv#449
mtreinish added a commit to mtreinish/asv that referenced this issue Aug 15, 2019
This commit adds a new flag, --strict, to the run (and by extension dev)
and continous commands. When set any benchmark failures encountered
will be treated as an error and the command will exit non-zero when the
run is finished. This is especially useful when running benchmarks in
a CI system or similar automated environment and wish to know if a
change was made to break a benchmark.

Fixes airspeed-velocity#449
@pv pv closed this as completed in #865 Aug 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Triaged as an enhancement request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants