Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unrolling a test only executes the first test. #1070

Open
Xaseron opened this issue Jan 9, 2020 · 10 comments
Open

Unrolling a test only executes the first test. #1070

Xaseron opened this issue Jan 9, 2020 · 10 comments
Milestone

Comments

@Xaseron
Copy link

@Xaseron Xaseron commented Jan 9, 2020

Issue description

Unrolling a test only executes the first test.
If i remove the @Unroll everything works.
If i remove the #queryValue from the method name everythings works.

How to reproduce

import spock.lang.Specification
import spock.lang.Unroll

class UnrollBugSpec extends Specification {

    @Unroll 
    void "Valid Mandatory QueryValue #queryValues"(String queryValues, Boolean returnStatus) {
        expect:
            assert returnStatus

        where:
            queryValues                                                        || returnStatus
            '?from=2018-12&to=2019-12'                                         || true
            '?from=2018-12&to=2019-12'                                         || true
            '?to=2019-12&from=2018-12&divisions=test&teams=test&products=test' || false
    }
}

Additional Environment information

Java/JDK

java -version
openjdk version "1.8.0_202"
OpenJDK Runtime Environment Corretto-8.202.08.2 (build 1.8.0_202-b08)
OpenJDK 64-Bit Server VM Corretto-8.202.08.2 (build 25.202-b08, mixed mode)

Groovy version

2.5

Build tool version

Gradle

5.3

Operating System

Windows

IDE

IntelliJ

Build-tool dependencies used

Gradle/Grails

testImplementation("org.spockframework:spock-core:1.3-groovy-2.5")
@szpak

This comment has been minimized.

Copy link
Member

@szpak szpak commented Jan 9, 2020

I'm not able to reproduce the problem with your tests with 1.3-groovy-2.5 in Idea 2019.3.2 Preview:

image

How to your run your tests?

Btw, you can remove String queryValues, Boolean returnStatus from the method arguments definition - they are not needed. The variables from where should be created by Spock AST and visible by IDE.

@Xaseron

This comment has been minimized.

Copy link
Author

@Xaseron Xaseron commented Jan 9, 2020

If i create a new gradle project it works. But if copy the build.gradle from my project i can recreate the bug. i've added everything needed to this sample project https://github.com/Xaseron/UnrollBug

@Xaseron

This comment has been minimized.

Copy link
Author

@Xaseron Xaseron commented Jan 9, 2020

You have to enable annotation processing to run this test in Intellij

@szpak

This comment has been minimized.

Copy link
Member

@szpak szpak commented Jan 9, 2020

Thanks for the sample project. It definitely should not be required to enable annotation processing to be able to run a Spock test in Idea. As you have a lot of dependencies there, I propose to start removing them to determine which causes the aforementioned behavior (what breaks parameterized tests, in some scenarios).

@Xaseron

This comment has been minimized.

Copy link
Author

@Xaseron Xaseron commented Jan 10, 2020

I have found the issue if i delete:

test {
    useJUnitPlatform()
}

It executes all tests correctly.
Is this related to Spock?
We are not mixing JUnit and Spock tests so it would be safe to elave this dependecy out?

@Xaseron

This comment has been minimized.

Copy link
Author

@Xaseron Xaseron commented Jan 10, 2020

I've updated the repo https://github.com/Xaseron/UnrollBug with a minimal build.gradle.

plugins {
    id 'groovy'
}

group 'org.example'
version '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.5.6'
    testImplementation("org.spockframework:spock-core:1.3-groovy-2.5")
    testRuntimeOnly "org.junit.vintage:junit-vintage-engine:5.5.0"
}

test {
    useJUnitPlatform()
}
@Vampire

This comment has been minimized.

Copy link
Member

@Vampire Vampire commented Jan 10, 2020

useJUnitPlatform means "Use JUnit 5", but Spock < v2 is based on JUnit 4.
I wonder there was anything executed at all, especially without using Vintage platform.
Spock v2 will be based on JUnit 5 instead.

@Xaseron

This comment has been minimized.

Copy link
Author

@Xaseron Xaseron commented Jan 10, 2020

Is Vintage platform something different than junit-vintage-engine?

@Vampire

This comment has been minimized.

Copy link
Member

@Vampire Vampire commented Jan 10, 2020

Oh, yeah, sorry, I was blind, forgot it is just a dependency. I thought you have to enable it too.

@szpak

This comment has been minimized.

Copy link
Member

@szpak szpak commented Jan 12, 2020

@Xaseron Thanks for narrowing down the issue. Based on it I was able to determine what causes the problem - identical name of 2+ unrolled tests with @Unroll and junit-vintage-engine.

The following tests (in the aforementioned sample project) while running with JUnit Platform:

class UnrollBugSpec extends Specification {
    @Unroll
    void "Valid Mandatory QueryValue #queryValues"() {
        expect:
            assert returnStatus

        where:
            queryValues || returnStatus
            1           || true
            1           || true
            3           || false
    }
}

executes only one test - the first one.

What's "funny" with the following:

...
        where:
            queryValues || returnStatus
            1           || true
            3           || true
            3           || false

Gradle fails with:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> Received a failure event for test with unknown id '30.4'. Registered test ids: '[:test, 30.2, 30.1]'

The problem occurs only with junit-vintage-engine. With JUnit 4.12 it works fine (3 tests are executed).

@marcphilipp Is there any extra filtering in junit-vintage-engine which may cause that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.