Skip to content
This repository was archived by the owner on Nov 16, 2018. It is now read-only.

Autotest Integration

Jump to bottom
Peeja edited this page Aug 13, 2010 · 31 revisions

Cucumber comes with an autotest plugin that will hook into the regular autotest cycle to run the features after the tests/specs. The plugin is disabled by default but can be turned on by setting the AUTOFEATURE environment variable to ‘true’. For example:

$ AUTOFEATURE=true autospec

If you always want to have the plugin run then you can export the variable in your .bash_profile or other shell file like so:

POSIX shell:

$ export AUTOFEATURE=true

Windows command shell:

> set AUTOFEATURE=true

By default the plugin will run all the features in the features dir. To change the way the features are ran with autotest create an ‘autotest’ profile in your cucumber.yml. Please see Running Features for more information about profiles.

What does it do?

In autotest’s normal flow, it runs all of your tests and remembers the
ones which fail. Each time you save a file, it reruns the failing
tests along with tests related to the code you just changed. Once
everything is green again, it reruns all of the tests over again to
make sure everything is green. Then it waits for more changes.

It looks like this:

  1. Run red tests until they pass.
  2. Run all tests.

Once you get all the tests to pass, you’re ready to write a new test (or example) to implement.

With Cucumber, autotest does this:

  1. Run red tests until they pass.
  2. Run red scenarios until they pass.
  3. Run all tests until they pass.
  4. Run all features.

Here’s the intended (but flexible) BDD workflow:

  1. Write a new feature or scenario.
    • Autotest runs your features and finds pending steps. It remembers those scenarios.
  2. Define pending steps.
    • Each time you save, autotest reruns the pending scenarios. Watch them turn red and blue.
  3. Pick a red step. Write the spec which exercises the (unwritten) implementation behind that step.
    • Autotest sees your new red spec and remembers it.
  4. Work on implementing the spec.
    • Each time you save, autotest reruns the red spec. No need to run any features until you’re done with this. Autotest keeps you drilled down.
  5. Finish implementing the spec.
    • When the spec turns green, you’re done! Autotest pops the stack and reruns the scenarios you’re working on.
  6. Repeat from Step 3 until your scenarios are all green.
    • Now that you’ve done a chunk of work, Autotest pops the stack again and runs all your specs to make sure nothing’s broken.
  7. Fix any unit-level regressions.
    • Once your specs are green and the scenarios you were working on are running, Autotest runs your entire feature set.
  8. Fix any integration-level regressions.
    • Now your features should all be green. Congratulations! Your iteration is done.!
  9. Return to step 1.

If you like to write out lots of features at the beginning, you may want to keep them somewhere where they won’t run and slowly introduce them, implementing them along the way. Future versions may provide a way to mark a scenario as “pending” to accomplish something similar.

Customizing Autotest Behavior

When autotest runs your features, it looks for an autotest profile in your cucumber.yml. If it finds one, it uses that profile when running your features. It’s useful to use the Pretty formatter in autotest to see exactly what’s going on as your scenarios fail. However, the Pretty formatter may be too much information when autotest does its final run-through of all of your features. If you define an autotest-all profile, autotest will use this profile instead when it runs all of your features. You may want to use the Progress formatter for autotest-all.

For more information on defining profiles, see Running Features.

Why is it disabled by default?

Autotest is geared toward the Red→Green→Refactor cycle on an object level (unit tests). These object level specs/tests are generally highly focused and isolated per object so breaks can be detected on a very detailed level. The suites are meant to run extremely fast to give the developer quick feedback. Cucumber on the other hand provides end-to-end application level testing. By executing the entire stack features can help find integration failures between objects and provide large coverage to prevent regressions. A side effect of this is that features are generally much slower than object level specs. Due to their relative slowness and non-focused nature they may not be realistic to run along side the object-level suite. Of course every project is different and every developer has different workflow preferences.

Clone this wiki locally