Gem to profile cucumber steps and features.
- (J)Ruby - 1.9.3 -> 2.3.0
- Cucumber - 1.3.5, 2.4.x+
High level features
Step analysis including
- Location of step in steps file & regex
- Step usage location and number of times executed (background/outline, etc.)
- Counts for success/failure/pending/etc.
- Total time taken in test run
- Average, fastest, slowest times per step
- Variation, variance & standard deviation calculations
Feature analysis including
- Feature location
- Time taken to run feature
- Result of feature test (pass, fail, etc.)
- Number of steps run
- Breakdown of feature by individual example run if a scenario outline
- Reporting of ambiguous step calls
- Reporting of unused step definitions
Add this line to your application's Gemfile:
And then execute:
$ bundle install
Or install it yourself as:
$ gem install cucumber_characteristics
Add the following line to your cucumber environment file typically found at
For always-on automatic loading (recommended), add
features/support/yourapp_env.rb. It is recommended by cucumber that you do not enhance features/support/env.rb so that upgrades are painless (relatively)
Add it to your
std_opts = "-r features/. -r --quiet --format CucumberCharacteristics::Formatter --format progress"
Use it via command line with
You can configure the export of step characteristics via the following (defaults are same as example)
CucumberCharacteristics.configure do |config| config.export_json = true config.export_html = true config.precision = 4 config.target_filename = 'cucumber_step_characteristics' config.relative_path = 'features/characteristics' end
This again can be added to your cucumber environment file typically found at
Exported characteristic information is listed out at the end of the cucumber run in a message similar to
Step characteristic report written to /home/singram/projects/gems/cucumber_characteristics/features/characteristics/cucumber_step_characteristics.html Step characteristic report written to /home/singram/projects/gems/cucumber_characteristics/features/characteristics/cucumber_step_characteristics.json
depending on the options specified.
The JSON option is provided for convenience in case there is a further use case/analysis required that is not provided by the gem.
An example can be found here
The formatting hooks on the face of it provide the necessary event points to profile any given feature file. This is true for a Scenario, but consider the following ScenaioOutline
Feature: As a user I want to understand where my tests are spending their time Scenario Outline: Timings for scenario outline Given I wait <given_wait> seconds When I wait <when_wait> seconds Then I wait <then_wait> seconds And I wait 0.2 seconds Examples: | given_wait | when_wait | then_wait | | 1 | 2 | 3 | | 5 | 6 | 7 |
cucumber --format debug features/outline.feature
A couple of problems become evident
- There are step definitions walked prior to the examples_array. These steps are not actually invoked rendering these hooks points misleading for profiling purposes
- There are only 3 table_cell element blocks. These can be profiled, but what about the last step that does not have an input from the examples? There are no hook points to profile this step.
This is why when you use the 'progress' formatter you would get 4 'skipped' for the initial step hooks triggered and then only 6 green dots representing steps when there should be 8 as it key's off table cells not steps.
- Introduce new hook point for all true step invocations regardless of context.
- Adjust table_cell hooks to include 'null' cells when considering steps without definitions.
- Include profile information in runtime master object to parse out at end.
As it turns out it was pretty simple to enhance the runtime object to reliably return profile information.
Install development environment
Run formatter over default cucumber version
bundle exec cucumber
Run tests across all supported cucumber versions
bundle exec rake versions:bundle:install
bundle exec rake versions:test
- NOTE. When running the cucumber tests failures, pending etc are expected. All specs should pass *
- Fork it
- Create your feature branch (
git checkout -b my-new-feature)
- Commit your changes (
git commit -am 'Add some feature')
- Push to the branch (
git push origin my-new-feature)
- Create new Pull Request (after running tests!)