New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show code for blocks without description #96

Closed
hexmind opened this Issue Feb 23, 2017 · 18 comments

Comments

Projects
None yet
3 participants
@hexmind

hexmind commented Feb 23, 2017

Currently, empty blocks are shown like this:
Where: ----

Groovy tests should be written meaningful and often it doesn't make sense to describe each block (duplication or comment out of date):

def "should add numbers"() {
        expect: "sum of 2 and 2 is 4"
            2 + 2 == 4
}

What do you think about to generate block description from code (with Groovy ASTTransformation):

when: "send \"hello\""
    publisher.send("hello")
then:
    2 * subscriber.receive("hello")

can be in report:

When:  send "hello"
Then: 2 times subscriber receive "hello"

It could be experimental feature enabled by annotation.
In MVP only code when no block description provided (without human-readable transformation like in example).

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Feb 23, 2017

Owner

That could be nice. Are you able to make a minimal implementation of that? I can help if you get it started.

If you make a PR, please target the next branch.

Owner

renatoathaydes commented Feb 23, 2017

That could be nice. Are you able to make a minimal implementation of that? I can help if you get it started.

If you make a PR, please target the next branch.

@hexmind

This comment has been minimized.

Show comment
Hide comment
@hexmind

hexmind Feb 24, 2017

I can't promise when but yes, I will try to do it.

hexmind commented Feb 24, 2017

I can't promise when but yes, I will try to do it.

@hexmind

This comment has been minimized.

Show comment
Hide comment
@hexmind

hexmind Mar 13, 2017

I have a code draft that works for this example test like below.

package com.athaydes.spockframework.report

import spock.lang.Issue
import spock.lang.Narrative
import spock.lang.See
import spock.lang.Specification
import spock.lang.Unroll

@Narrative("""
As a user
I want foo
So that bar""")
class VividFakeTest extends Specification {

    def "A first test with Then code block"() {
        given:
            "we have x and y"

        and:
            "some more things"

        when:
            "I do crazy things"

        then:
            x == y

        where:
            "The examples below are used"
            x | y
            'a' | 'a'
            'b' | 'c'
    }

    def "Another feature without code"() {
        setup:
        "Setup block here"
        expect:
        "Expecting something ??"
    }

    def "Another feature with method call"() {
        expect:
        add(1, 2) == 3
    }

    private static int add(int a, int b) {
        return a + b
    }

    @Issue( [ "http://myhost.com/issues/995", "http://myhost.com/issues/973" ] )
    def "A test with an error"() {
        when:
        "An Exception is thrown"
        throw new RuntimeException( 'As expected' )

        then:
        "Will never succeed"
    }

    @See( "http://myhost.com/features/feature-234" )
    def "A test with a failure"() {
        when:
        "Do nothing"
        then:
        "Test fails"
        assert 3 == 2
    }

    def "A Spec without block Strings"() {
        given:
        int a = 0

        and:
        int b = 1

        when:
        int c = a + b

        then:
        c == 1

        and:
        c > 0
    }

    @Unroll
    def "An @Unrolled spec with x=#x and y=#y"() {
        setup:
        "nothing"
        expect:
        x == 0
        and:
        "An error if y is 5"
        if ( y == 5 ) throw new RuntimeException( 'y is 5' )
        where:
        x | y
        0 | 1
        2 | 3
        0 | 5
    }

}

Report for com.athaydes.spockframework.report.VividFakeTest

##Summary

  • Total Runs: 9
  • Success Rate: 44.44%
  • Failures: 3
  • Errors: 2
  • Skipped: 0
  • Total time: Unknown

###As a user
###I want foo
###So that bar

Features

A first test with Then code block

Result: FAIL

  • Given: we have x and y

  • And: some more things

  • When: I do crazy things

  • Then: x == y

  • Where: The examples below are used

x y
a a
b c

The following problems occurred:

  • [b, c]
Condition not satisfied:

x == y
| |  |
b |  c
  false
  1 difference (0% similarity)
  (b)
  (c)

Another feature without code

Result: PASS

  • Given: Setup block here

  • Expect: Expecting something ??

Another feature with method call

Result: PASS

  • Expect: add(1, 2) == 3

A test with an error

Issues:

Result: ERROR

  • When: An Exception is thrown

  • Then: Will never succeed

The following problems occurred:

java.lang.RuntimeException: As expected

A test with a failure

See:

Result: FAIL

  • When: Do nothing

  • Then: Test fails

The following problems occurred:

Condition not satisfied:

3 == 2
  |
  false

A Spec without block Strings

Result: PASS

  • Given: int a = 0

  • And: int b = 1

  • When: int c = a + b

  • Then: c == 1

  • And: c > 0

An @unrolled spec with x=0 and y=1

Result: PASS

  • Given: nothing

  • Expect: x == 0

  • And: An error if y is 5

  • Where: ----

x y
0 1

An @unrolled spec with x=2 and y=3

Result: FAILURE

  • Given: nothing

  • Expect: x == 0

  • And: An error if y is 5

  • Where: ----

x y
2 3

The following problems occurred:

  • [2, 3]
Condition not satisfied:

x == 0
| |
2 false

An @unrolled spec with x=0 and y=5

Result: ERROR

  • Given: nothing

  • Expect: x == 0

  • And: An error if y is 5

  • Where: ----

x y
0 5

The following problems occurred:

  • [0, 5]
java.lang.RuntimeException: y is 5

Generated by Athaydes Spock Reports

hexmind commented Mar 13, 2017

I have a code draft that works for this example test like below.

package com.athaydes.spockframework.report

import spock.lang.Issue
import spock.lang.Narrative
import spock.lang.See
import spock.lang.Specification
import spock.lang.Unroll

@Narrative("""
As a user
I want foo
So that bar""")
class VividFakeTest extends Specification {

    def "A first test with Then code block"() {
        given:
            "we have x and y"

        and:
            "some more things"

        when:
            "I do crazy things"

        then:
            x == y

        where:
            "The examples below are used"
            x | y
            'a' | 'a'
            'b' | 'c'
    }

    def "Another feature without code"() {
        setup:
        "Setup block here"
        expect:
        "Expecting something ??"
    }

    def "Another feature with method call"() {
        expect:
        add(1, 2) == 3
    }

    private static int add(int a, int b) {
        return a + b
    }

    @Issue( [ "http://myhost.com/issues/995", "http://myhost.com/issues/973" ] )
    def "A test with an error"() {
        when:
        "An Exception is thrown"
        throw new RuntimeException( 'As expected' )

        then:
        "Will never succeed"
    }

    @See( "http://myhost.com/features/feature-234" )
    def "A test with a failure"() {
        when:
        "Do nothing"
        then:
        "Test fails"
        assert 3 == 2
    }

    def "A Spec without block Strings"() {
        given:
        int a = 0

        and:
        int b = 1

        when:
        int c = a + b

        then:
        c == 1

        and:
        c > 0
    }

    @Unroll
    def "An @Unrolled spec with x=#x and y=#y"() {
        setup:
        "nothing"
        expect:
        x == 0
        and:
        "An error if y is 5"
        if ( y == 5 ) throw new RuntimeException( 'y is 5' )
        where:
        x | y
        0 | 1
        2 | 3
        0 | 5
    }

}

Report for com.athaydes.spockframework.report.VividFakeTest

##Summary

  • Total Runs: 9
  • Success Rate: 44.44%
  • Failures: 3
  • Errors: 2
  • Skipped: 0
  • Total time: Unknown

###As a user
###I want foo
###So that bar

Features

A first test with Then code block

Result: FAIL

  • Given: we have x and y

  • And: some more things

  • When: I do crazy things

  • Then: x == y

  • Where: The examples below are used

x y
a a
b c

The following problems occurred:

  • [b, c]
Condition not satisfied:

x == y
| |  |
b |  c
  false
  1 difference (0% similarity)
  (b)
  (c)

Another feature without code

Result: PASS

  • Given: Setup block here

  • Expect: Expecting something ??

Another feature with method call

Result: PASS

  • Expect: add(1, 2) == 3

A test with an error

Issues:

Result: ERROR

  • When: An Exception is thrown

  • Then: Will never succeed

The following problems occurred:

java.lang.RuntimeException: As expected

A test with a failure

See:

Result: FAIL

  • When: Do nothing

  • Then: Test fails

The following problems occurred:

Condition not satisfied:

3 == 2
  |
  false

A Spec without block Strings

Result: PASS

  • Given: int a = 0

  • And: int b = 1

  • When: int c = a + b

  • Then: c == 1

  • And: c > 0

An @unrolled spec with x=0 and y=1

Result: PASS

  • Given: nothing

  • Expect: x == 0

  • And: An error if y is 5

  • Where: ----

x y
0 1

An @unrolled spec with x=2 and y=3

Result: FAILURE

  • Given: nothing

  • Expect: x == 0

  • And: An error if y is 5

  • Where: ----

x y
2 3

The following problems occurred:

  • [2, 3]
Condition not satisfied:

x == 0
| |
2 false

An @unrolled spec with x=0 and y=5

Result: ERROR

  • Given: nothing

  • Expect: x == 0

  • And: An error if y is 5

  • Where: ----

x y
0 5

The following problems occurred:

  • [0, 5]
java.lang.RuntimeException: y is 5

Generated by Athaydes Spock Reports

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Mar 13, 2017

Owner

It looks good. Do you think this is reliable (will work with any code in the blocks)? Are you planning to make a pull request soon?

Owner

renatoathaydes commented Mar 13, 2017

It looks good. Do you think this is reliable (will work with any code in the blocks)? Are you planning to make a pull request soon?

@hexmind

This comment has been minimized.

Show comment
Hide comment
@hexmind

hexmind Mar 13, 2017

I am working on PR and it should be reliable.
It will be triggered by com.athaydes.spockframework.report.showCodeBlocks=true property.

hexmind commented Mar 13, 2017

I am working on PR and it should be reliable.
It will be triggered by com.athaydes.spockframework.report.showCodeBlocks=true property.

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Mar 13, 2017

Owner

I think you can make this property default to true. I can't think of a reason why people would prefer to see --- rather than a little code (hopefully people don't write specs that have hundreds of lines of code per block :D ).

Owner

renatoathaydes commented Mar 13, 2017

I think you can make this property default to true. I can't think of a reason why people would prefer to see --- rather than a little code (hopefully people don't write specs that have hundreds of lines of code per block :D ).

@hexmind

This comment has been minimized.

Show comment
Hide comment
@hexmind

hexmind Mar 13, 2017

Eventually yes. But remember, currently we should think about this feature like about experimental one. It wasn't tested on too many specifications classes.

hexmind commented Mar 13, 2017

Eventually yes. But remember, currently we should think about this feature like about experimental one. It wasn't tested on too many specifications classes.

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Mar 13, 2017

Owner

why not? Just add tests... I can add more later.

Owner

renatoathaydes commented Mar 13, 2017

why not? Just add tests... I can add more later.

hexmind pushed a commit to hexmind/spock-reports that referenced this issue Mar 14, 2017

hexmind pushed a commit to hexmind/spock-reports that referenced this issue Mar 14, 2017

hexmind pushed a commit to hexmind/spock-reports that referenced this issue Mar 14, 2017

renatoathaydes added a commit that referenced this issue Mar 19, 2017

Merge pull request #97 from hexmind/issue-96-show-code-blocks
Show code for blocks without description #96 MVP
@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Mar 30, 2017

Owner

@hexmind I've been trying to fix the code you submitted for a week now, and I am nowhere near finishing. I had thought you had implemented something that at least would work for simple cases, but it doesn't even work for the simplest of cases at all.

Only one line of code is captured, and only if the line happens to be a simple case where the first statement is not a constant or any expression other than ExpressionStatement.

FYI you did not handle the following other cases:

screen shot 2017-03-30 at 21 23 56

The mapping between code lines and blocks would only work if the spec had a single block definition for each keyword (ie. only one given, when and then). Nevermind most specifications repeat blocks, sometimes several times...

Unfortunately, the code has been merged to next, which already contained quite a few other changes for the next release...

My question is: do you have time to help finish implementing this feature correctly?

I know how to do it, but I don't have all the time in the world to spend on this! So any help would be appreciated, I can tell you what needs to be done. Let me know.

Owner

renatoathaydes commented Mar 30, 2017

@hexmind I've been trying to fix the code you submitted for a week now, and I am nowhere near finishing. I had thought you had implemented something that at least would work for simple cases, but it doesn't even work for the simplest of cases at all.

Only one line of code is captured, and only if the line happens to be a simple case where the first statement is not a constant or any expression other than ExpressionStatement.

FYI you did not handle the following other cases:

screen shot 2017-03-30 at 21 23 56

The mapping between code lines and blocks would only work if the spec had a single block definition for each keyword (ie. only one given, when and then). Nevermind most specifications repeat blocks, sometimes several times...

Unfortunately, the code has been merged to next, which already contained quite a few other changes for the next release...

My question is: do you have time to help finish implementing this feature correctly?

I know how to do it, but I don't have all the time in the world to spend on this! So any help would be appreciated, I can tell you what needs to be done. Let me know.

@rdmueller

This comment has been minimized.

Show comment
Hide comment
@rdmueller

rdmueller Mar 30, 2017

Contributor

This feature is great and I would like to see it implemented!
So do I undestand the current problem the right way that we have a code block as AST and have to render it as code?

Contributor

rdmueller commented Mar 30, 2017

This feature is great and I would like to see it implemented!
So do I undestand the current problem the right way that we have a code block as AST and have to render it as code?

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Mar 30, 2017

Owner

@rdmueller thanks for letting me know :) I need quite a lot of motivation to keep working on it!

Owner

renatoathaydes commented Mar 30, 2017

@rdmueller thanks for letting me know :) I need quite a lot of motivation to keep working on it!

@rdmueller

This comment has been minimized.

Show comment
Hide comment
@rdmueller

rdmueller Mar 30, 2017

Contributor

that's why I keep telling you :-)

I have the feeling that the groovyConsole with the included AST browser (Strg+T) might contain the solution. Isn't it open source?

Contributor

rdmueller commented Mar 30, 2017

that's why I keep telling you :-)

I have the feeling that the groovyConsole with the included AST browser (Strg+T) might contain the solution. Isn't it open source?

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Mar 30, 2017

Owner

I know how to do it the hard way, which is by handling every single statement type (shown in the image I posted). Would be awesome to let some library do it for us, though... Spock itself captures only single expressions, it seems, not the full source for a block... that's why it's hard, because we need to iterate through each statement, figure out how to handle each type, then ask the Spock's org.spockframework.compiler.SourceLookup#lookup method for the source code of the statement. See my latest commit where I started doing it.

Owner

renatoathaydes commented Mar 30, 2017

I know how to do it the hard way, which is by handling every single statement type (shown in the image I posted). Would be awesome to let some library do it for us, though... Spock itself captures only single expressions, it seems, not the full source for a block... that's why it's hard, because we need to iterate through each statement, figure out how to handle each type, then ask the Spock's org.spockframework.compiler.SourceLookup#lookup method for the source code of the statement. See my latest commit where I started doing it.

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Apr 1, 2017

Owner

This feature is now implemented correctly! I found out that Spock's source code lookup works for any AST node, no need to recurse into each expression, so it's much easier and much more reliable.

I had to change completely the implementation, but that was mostly refactoring and cleaning up, the hard part is done by Spock (to actually read the source code from each class).

I will still add more tests for the lots of edge cases (nested classes, several classes in one file, specs with methods that look like tests but are not, etc).

Soon we can have this released.

Owner

renatoathaydes commented Apr 1, 2017

This feature is now implemented correctly! I found out that Spock's source code lookup works for any AST node, no need to recurse into each expression, so it's much easier and much more reliable.

I had to change completely the implementation, but that was mostly refactoring and cleaning up, the hard part is done by Spock (to actually read the source code from each class).

I will still add more tests for the lots of edge cases (nested classes, several classes in one file, specs with methods that look like tests but are not, etc).

Soon we can have this released.

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Apr 1, 2017

Owner

BTW. Please checkout the "next" branch and take this code for a spin!

Add this to your META-INF/services/com.athaydes.spockframework.report.IReportCreator.properties file and run your tests:

com.athaydes.spockframework.report.IReportCreator=com.athaydes.spockframework.report.template.TemplateReportCreator

# Set properties of the report creator
com.athaydes.spockframework.report.template.TemplateReportCreator.showCodeBlocks=true

Let me know if you have issues.

Owner

renatoathaydes commented Apr 1, 2017

BTW. Please checkout the "next" branch and take this code for a spin!

Add this to your META-INF/services/com.athaydes.spockframework.report.IReportCreator.properties file and run your tests:

com.athaydes.spockframework.report.IReportCreator=com.athaydes.spockframework.report.template.TemplateReportCreator

# Set properties of the report creator
com.athaydes.spockframework.report.template.TemplateReportCreator.showCodeBlocks=true

Let me know if you have issues.

@hexmind

This comment has been minimized.

Show comment
Hide comment
@hexmind

hexmind Apr 2, 2017

Hi @renatoathaydes and sorry for delayed response. I see that you did it.
Can I finish with HtmlReportCreator?

hexmind commented Apr 2, 2017

Hi @renatoathaydes and sorry for delayed response. I see that you did it.
Can I finish with HtmlReportCreator?

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Apr 3, 2017

Owner

@hexmind you can submit a PR, of course, if you want to help.

Owner

renatoathaydes commented Apr 3, 2017

@hexmind you can submit a PR, of course, if you want to help.

hexmind pushed a commit to hexmind/spock-reports that referenced this issue Apr 8, 2017

hexmind Tomasz Skowroński
#96 Show code for blocks without description - HtmlReportCreator
fix decimalSeparator
add VividFakeTestReport.html

hexmind pushed a commit to hexmind/spock-reports that referenced this issue Apr 8, 2017

hexmind pushed a commit to hexmind/spock-reports that referenced this issue Apr 11, 2017

hexmind pushed a commit to hexmind/spock-reports that referenced this issue Apr 11, 2017

@renatoathaydes

This comment has been minimized.

Show comment
Hide comment
@renatoathaydes

renatoathaydes Apr 21, 2017

Owner

Done in version 1.3.0.

Owner

renatoathaydes commented Apr 21, 2017

Done in version 1.3.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment