Add fix_setup method to clean up if setup fails

If the `setup` method fails, the tests session waits for timeout (1800s per
benchmark!) provided a sub process had been created beforehand. This lead to
excruciatingly long test sessions, appearing fully stuck.

asv/benchmark.py

@@ -442,6 +442,7 @@ def __init__(self, name, func, attr_sources):
         self.pretty_name = getattr(func, "pretty_name", None)
         self._attr_sources = attr_sources
         self._setups = list(_get_all_attrs(attr_sources, 'setup', True))[::-1]
+        self._fix_setups = list(_get_all_attrs(attr_sources, 'fix_setup', True))
         self._teardowns = list(_get_all_attrs(attr_sources, 'teardown', True))
         self._setup_cache = _get_first_attr(attr_sources, 'setup_cache', None)
         self.setup_cache_key = get_setup_cache_key(self._setup_cache)
@@ -551,6 +552,10 @@ def do_setup(self):
         except NotImplementedError:
             # allow skipping test
             return True
+        except BaseException as e:
+            for fix_setup in self._fix_setups:
+                fix_setup(e, self._current_params)
+            raise
         return False
 
     def redo_setup(self):

docs/source/writing_benchmarks.rst

@@ -117,7 +117,25 @@ profiling occur within the same process.  For these cases, the setup
 and teardown routines are run multiple times in the same process.
 
 If ``setup`` raises a ``NotImplementedError``, the benchmark is marked
-as skipped.
+as skipped. If ``setup`` raises any other exception, the ``fix_setup``
+method (if it exist) will be called`` with the exception as first
+argument, and the current benchmark parameters as second.
+This method gives you a chance to clean up any side-effects from the
+setup if it fails::
+
+  class Suite:
+      def setup(self, *args, **kwargs):
+          self._server = MySuperServer()
+          self._server.launch()
+          # other code
+
+      def fix_setup(self, exception, bench_params):
+          if hasattr(self, '_server'):
+              self._server.stop()
+
+Had the ``fix_setup`` not stopped the server, ASV would have waited
+until the ``setup`` timeout to go to the next benchmark, since a
+running process was still attached. The exception is then re-raised.
 
 The ``setup`` method is run multiple times, for each benchmark and for
 each repeat.  If the ``setup`` is especially expensive, the