Issue #207 UsageFormatter #214

Merged
merged 10 commits into from Mar 26, 2012

Projects

None yet

2 participants

@klausbayrhammer

A usage formatter displaying duration of steps and calculating average and median of step durations

Sample Output:

0.000135 (median) : Given  precondition
0.025980 (average) : Given  precondition
    0.000025 : Given  precondition
    0.000108 : Given  precondition
    0.000163 : Given  precondition
    0.103624 : Given  precondition
0.069714 (median) : Then  I expect following
0.069737 (average) : Then  I expect following
    0.069334 : Then  I expect following
    0.069668 : Then  I expect following
    0.069674 : Then  I expect following
    0.069754 : Then  I expect following
    0.069842 : Then  I expect following
    0.070154 : Then  I expect following
0.000025 (median) : When  I do the action with param
0.000050 (average) : When  I do the action with param
    0.000011 : When  I do the action with param
    0.000012 : When  I do the action with param
    0.000023 : When  I do the action with param
    0.000025 : When  I do the action with param
    0.000025 : When  I do the action with param
    0.000026 : When  I do the action with param
    0.000140 : When  I do the action with param
    0.000141 : When  I do the action with param
@aslakhellesoy
Cucumber member

Nice work!

The original usage formatter in Cucumber displays the location of each step definition, and all matching steps with their location. Your example above doesn't show any locations, so I'm not sure what kind of usage your formatter produces.

There is already a --dotcucumber flag that produces what I would call an improved version of Cucumber's existing usage formatter. It has more information and a more versatile format (JSON). Here is an example: https://github.com/cucumber/cucumber-jvm/blob/master/picocontainer/src/test/resources/.cucumber/stepdefs.json

Were you aware of this? It looks like your motivation for writing this formatter is to get some more time stats. Do you think it would make sense to expand the .cucumber/stepdefs.json format include times instead?

@klausbayrhammer

Your right, the intention was to get some more time stats.

The usage-formatter in cuke4duke generates something like:

12.4327500 ^precondition$                                                                                       #PerformanceSteps.preconditionsellVehicle()
  12.4650000 Given precondition                                                                                   # features\performancetest\performance.feature:18
  12.6340000 Given precondition                                                                                   # features\performancetest\performance.feature:33
  12.7520000 Given precondition                                                                                   # features\performancetest\performance.feature:48
  11.8800000 Given precondition                                                                                   # features\performancetest\performance.feature:63

What is the target of the --docucumber flag? I did like the approach of different formatters, so we could activate a specific usage/performance-formatter

@aslakhellesoy
Cucumber member

The argument to the --dotcucumber option is a file path. While it generates "output" it isn't implemented via the Formatter and Reporter interfaces, because the original intent wasn't to report at all.

What it really is is metadata about the structure of gherkin and stepdefs, and how they relate. This is used for code completion in IDEs: https://github.com/cucumber/gherkin/wiki/Code-Completion. The generated file is meant to be committed to source control to make it easily available for IDEs. I think it would be a mistake to add runtime info to it (run results, time stats etc). It would cause a lot of merge problems for developers.

So I change my opinion about merging your work into it. I think your formatter should be a proper formatter/reporter. However, it would be nice if you made it output a similar format as --dotcucumber - perhaps just with extra fields. Having this information available in JSON would make it a lot more useful than plain text.

I suggest you take a look at the classes in the cucumber.runtime.autocomplete package. You could pregenerate a structure with List<MetaStepdef> StepdefGenerator.generate(List<StepDefinition> stepDefinitions, List<CucumberFeature> features). Then, as results come in you can add data about that in the same structure. You'd have to be able to lool up steps from a map, and add some attributes to the support classes (MetaStep perhaps).

@klausbayrhammer

Sounds like a good idea. I'll code it today or tomorrow.

@klausbayrhammer

Now the UsageFormatter generates a json report. I didn't use the StepdefGenerator because the output generated by the usageFormatter is much simpler. Furthermore I didn't have to re-create StepDefinitions and CucumberFeatures.

Sample output:

[
  {
    "stepName": "Given  precondition",
    "aggregatedResults": [
      {
        "strategy": "median",
        "value": "0.000119"
      },
      {
        "strategy": "average",
        "value": "0.030210"
      }
    ],
    "durations": [
      "0.120591",
      "0.000182",
      "0.000056",
      "0.000013"
    ]
  },
  {
    "stepName": "Then  I expect following",
    "aggregatedResults": [
      {
        "strategy": "median",
        "value": "0.069292"
      },
      {
        "strategy": "average",
        "value": "0.069335"
      }
    ],
    "durations": [
      "0.069264",
      "0.069281",
      "0.069388",
      "0.069511",
      "0.069304",
      "0.069267"
    ]
  },
  {
    "stepName": "When  I do the action with param",
    "aggregatedResults": [
      {
        "strategy": "median",
        "value": "0.000013"
      },
      {
        "strategy": "average",
        "value": "0.000049"
      }
    ],
    "durations": [
      "0.000128",
      "0.000193",
      "0.000015",
      "0.000013",
      "0.000013",
      "0.000013",
      "0.000012",
      "0.000012"
    ]
  }
]
@aslakhellesoy aslakhellesoy and 1 other commented on an outdated diff Feb 20, 2012
.../src/main/java/cucumber/formatter/UsageFormatter.java
+ public void result(Result result)
+ {
+ if (!steps.isEmpty())
+ {
+ Step step = steps.remove(0);
+ String stepNameWithArgs = formatStepNameWithArgs(result, step);
+ addUsageEntry(result, stepNameWithArgs);
+ }
+ }
+
+ private String formatStepNameWithArgs(Result result, Step step)
+ {
+ StringBuffer buffer = new StringBuffer();
+ buffer.append(step.getKeyword()).append(" ");
+ Format format = getFormat(result.getStatus());
+ Format argFormat = getArgFormat(result.getStatus());
@aslakhellesoy
aslakhellesoy Feb 20, 2012

Format is for ANSI-coloured output. Did you really intend to put colours in the JSON?

@klausbayrhammer
klausbayrhammer Feb 20, 2012

I did use the MonochromeFormat in order to reuse the StepPrinter for printing Steps and their corresponding arguments.

@aslakhellesoy aslakhellesoy commented on an outdated diff Feb 20, 2012
.../src/main/java/cucumber/formatter/UsageFormatter.java
+
+ @Override
+ public void result(Result result)
+ {
+ if (!steps.isEmpty())
+ {
+ Step step = steps.remove(0);
+ String stepNameWithArgs = formatStepNameWithArgs(result, step);
+ addUsageEntry(result, stepNameWithArgs);
+ }
+ }
+
+ private String formatStepNameWithArgs(Result result, Step step)
+ {
+ StringBuffer buffer = new StringBuffer();
+ buffer.append(step.getKeyword()).append(" ");
@aslakhellesoy
aslakhellesoy Feb 20, 2012

The keyword already contains a space, so son't add an extra one.

@aslakhellesoy
Cucumber member

I'm still puzzled by a few things:

The step definition is not in the report. Instead of "stepName": "When I do the action with param" - wouldn't it make more sense to have stepdef: "^I do the action with param$".

The durations don't seem to have much value if I don't know where they come from.

@klausbayrhammer

Good point on the stepdefs. I'll put them in the report.

What I want to achieve by using the step-names with their corresponding arguments can be explained by the following example:

Given a stepdef like "^I create (\d+) entries$"
It wouldn't make much sense to compute the average runtime if one step is "I create 2 entries" and the second step is "I create 5 entries". But it would make sense if I would compare all "I create 2 entries" and "I create 5 entries" separately.

Maybe we should include the stepLocations of the executed steps as well?

@aslakhellesoy
Cucumber member

Yeah, I think the structure that is used for --docucumber is pretty close to what we want - except it has more info about execution times.

I also think the averages can be simplified:

"aggregatedResults": {
    "median": 0.000013
    "average": 0.000049
}
@klausbayrhammer

Yes your right.

So the output could be like the following:

[
  {
    "source": "^I have (\\d+) (.*) in my belly$",
    "flags": "",
    "steps": [
      {
        "name": "I have 12 cukes in my belly",
        "aggregatedResults": {
          "median": 0.000013
          "average": 0.000049
        },
        "duration" : [
          0.000013, 
          0.000063
        ]
      },
      {
        "name": "I have 3 cukes in my belly",
        "aggregatedResults": {
          "median": 0.000008
          "average": 0.000009
        },
        "duration" : [
          0.000006, 
          0.000012
        ]
      },
    ]
  },
]

we could also add the stepLocations e.g.

[
  {
    "source": "^I have (\\d+) (.*) in my belly$",
    "flags": "",
    "steps": [
      {
        "name": "I have 12 cukes in my belly",
        "aggregatedResults": {
          "median": 0.000013
          "average": 0.000013
        },
        "durations" : [
          {
            "duration" : 0.000013, 
            "location" : "/features/usage.feature:3"
          },
          {
            "duration" : 0.000013, 
            "location" : "/features/usage.feature:7"
          },
        ]
      }
    ]
  }
]

Would you suggest to use the StepdefGenerator.generate? In this case we would have to do some modifications so we can create the MetaStepDef-Structur with the information available in the reporter/formatter.

@klausbayrhammer klausbayrhammer * added steplocation to report
 * added stepDefinition (pattern)
 * grouped steps under stepdefinitions
2e56457
@klausbayrhammer

I modified the code to according to your feedback:

This is the current output:

[
  {
    "source": "^the result should be (.+)$",
    "steps": [
      {
        "name": "the result should be 1",
        "aggregatedResults": {
          "median": 0.05107028,
          "average": 0.05107028
        },
        "durations": [
          {
            "duration": 0.102112843,
            "location": "C:\\fake\\feature.feature:3"
          },
          {
            "duration": 0.000027717,
            "location": "C:\\fake\\feature.feature:4"
          }
        ]
      },
      {
        "name": "the result should be 4",
        "aggregatedResults": {
          "median": 0.000017708,
          "average": 0.000020017
        },
        "durations": [
          {
            "duration": 0.000025022,
            "location": "C:\\fake\\feature.feature:5"
          },
          {
            "duration": 0.000017708,
            "location": "C:\\fake\\feature.feature:6"
          },
          {
            "duration": 0.000017323,
            "location": "C:\\fake\\feature.feature:7"
          }
        ]
      }
    ]
  }
]
@aslakhellesoy
Cucumber member

I need some more time to go over this - haven't forgotten about it!

@aslakhellesoy
Cucumber member

Hi there. I'm afraid this no longer merges cleanly due to recent refactorings. Any chance you can update this pull request?

@klausbayrhammer

Of course. I'll go for it tomorrow

klausbayrhammer added some commits Mar 22, 2012
@klausbayrhammer klausbayrhammer ignore failed and skipped steps a865428
@klausbayrhammer klausbayrhammer Merge remote-tracking branch 'upstream/master' into usageFormatter
Conflicts:
	core/src/main/java/cucumber/formatter/FormatterFactory.java
	core/src/test/java/cucumber/formatter/FormatterFactoryTest.java
3302b9c
@aslakhellesoy aslakhellesoy merged commit 3302b9c into cucumber:master Mar 26, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment