Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

improve test reports for cucumber-android #598

Merged
merged 5 commits into from

5 participants

@SierraGolf
Collaborator

This PR is a follow up for "extension of the Formatter interface for more precise lifecycle handling" and will not compile as long as the other PR is not merged.

This PR is a refactoring of the previously quite buggy AndroidReporter. The following bugs have been fixed:

  • double entries for end tags
  • one to many entry start/end tag for outlines
  • green tests for exceptions coming from hooks
  • android maven plugin exceptions for undefined steps
  • wrong count of scenario examples
  • wrong step associated with missing step definition

In order to test the behaviour properly three new test dependencies have been added. Additionally the project structure was changed to the usual maven structure.

Questions/Problems:
1. Which tests should I extend in order consider the newly added Formatter methods?
2. In general the android module builds and also the whole project builds, when skipping tests. However there are weird tests errors which differ depending on which java version is used, when trying to build the whole project. With 1.6.0_51 I get the following errors:

Tests in error:
  URLOutputStreamTest.throws_ioe_if_http_response_is_500:109 � Connect Connectio...
  URLOutputStreamTest.throws_fnfe_if_http_response_is_404:85 � Connect Connectio...
  URLOutputStreamTest.can_http_put:76 � Connect Connection refused

and JSONPrettyFormatterTest.featureWithOutlineTest fails: because the expected out contains the before hook information in the wrong scenario. When I run with 1.7.0_21 the errors disappear, but the failure remains.

@friederbluemle

Looks like a great improvement! Any chance this could get merged soon?
Just a minor suggestion for the future: Please use separate commits for things like restructuring the project and making actual code changes. Otherwise it is hard to see what has changed.
Thanks.

@SierraGolf
Collaborator

Just a minor suggestion for the future: Please use separate commits for things like restructuring the project and making actual code changes. Otherwise it is hard to see what has changed.

Yeah, I am sorry for that. The restructuring happened somewhere half way through.

@SierraGolf SierraGolf * fixed several bugs in the AndroidTestReporter formerly known as And…
…roidReporter

* restructured project to be more maven compliant
* added dependencies for testing purposes
7bf5074
@brasmusson
Collaborator

@friederbluemle Unfortunately this PR build on a PR for Gherkin, and (as far as I know) snapshot releases are not deployed for Gherkin, so this PR cannot be merges until after the next Gherkin release.

@mfellner
Collaborator

Note: depends on cucumber/gherkin#275.

@SierraGolf
Collaborator

Answer to my second question: the commit cucumber/gherkin@b425544 changed the output of the JsonFormattter (to the better) which requires changing the JSONPrettyFormatterTest.json in the cucumber-core module.

@aslakhellesoy Do you want me to include the fix for the failing test in this PR?

@brasmusson
Collaborator

@SierraGolf The needed change in the JsonFormatterTest due to the change in cucumber/gherkin@b425544 is taken care of in #570, which supplies the failing test to drive cucumber/gherkin#270 (which the commit belongs to), so it is not necessary for you to fix it in this PR. But if you what a passing build together with Gherkin built from the latest source, you would also need to include #570.

@mfellner
Collaborator

@SierraGolf, I think there is another general problem with the expected output in JSONPrettyFormatterTest.json. As of now, a "before" element is included in these JSON objects:

  • "id": "feature-3;scenario-1"
  • "id": "feature-3;scenariooutline-1"
  • "id": "feature-3;scenariooutline-1;;2"
  • "id": "feature-3;scenariooutline-1;;3"

However I believe the element should be placed in these objects instead:

  • "id": "feature-3;scenario-1"
  • "id": "feature-3;scenariooutline-1;;1"
  • "id": "feature-3;scenariooutline-1;;2"
  • "id": "feature-3;scenariooutline-1;;3"
  • "id": "feature-3;scenario-2"

Note that "id": "feature-3;scenariooutline-1;;1" is not even included in the JSON at all, even though it is referenced in "id": "feature-3;scenariooutline-1;".

In b425544abc04821be2319d692aebb7d67c3d5daf @brasmusson noted where the hooks should be placed.

Update: Ok, I just realized this is at least partially fixed in #570. But "id": "feature-3;scenariooutline-1;;1" is still missing, why?

@SierraGolf
Collaborator

@brasmusson thanks for the info. in case #570 is merged before this one, I will rebase. I probably need to update this PR anyways as soon as the gherkin project is released.

@aslakhellesoy

I tried building this branch (after first updating the gherkin dependency to 2.12.2.

The cucumber-core module has one failing test, JSONPrettyFormatterTest.featureWithOutlineTest:35.

Can you take a look at that, @SierraGolf ?

@brasmusson
Collaborator

@aslakhellesoy You need both #570 and this PR to get a passing build with Gherkin 2.12.2. #570 because of cucumber/gherkin#270 and this PR because of cucumber/gherkin#275

@aslakhellesoy

@brasmusson thanks! that seems to work.

@aslakhellesoy aslakhellesoy referenced this pull request from a commit
@aslakhellesoy aslakhellesoy Attributions for #598 and #570 25f198b
@aslakhellesoy aslakhellesoy merged commit 7bf5074 into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 21, 2013
  1. @SierraGolf
  2. @SierraGolf
  3. @SierraGolf
Commits on Sep 22, 2013
  1. @SierraGolf
Commits on Oct 2, 2013
  1. @SierraGolf

    * fixed several bugs in the AndroidTestReporter formerly known as And…

    SierraGolf authored
    …roidReporter
    
    * restructured project to be more maven compliant
    * added dependencies for testing purposes
Something went wrong with that request. Please try again.