New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

first failing feature marks entire spec as 'skipped'? #844

Closed
rosemead opened this Issue May 14, 2018 · 5 comments

Comments

Projects
None yet
3 participants
@rosemead

rosemead commented May 14, 2018

Hi.
Question. Given this spec:

@Stepwise
class StepwiseSpec extends Specification {
    def stepA() {
        expect: true
    }
    def stepB() {
        expect: false
    }
}

I would expect this, when reported, to present a success rate of 50%: stepA passes, stepB fails.
Yet spock-reports reports such a run as having 100% success rate.
If I am not mistaken StepwiseExtension::skipFeaturesAfterFirstFailingFeature marks all features as skipped, when a feature in a stepwise spec fails. That would eventually lead spock reports to spit-out a skewed result. Shouldn't we count an executed feature, even if it failed/resulted in an error?
Thanks,

@leonard84

This comment has been minimized.

Member

leonard84 commented Jul 7, 2018

Please provide some more information about your environment. Running this in Intellij correctly reports 1 failed/1 skipped of 2 tests

@tilmanginzel

This comment has been minimized.

Contributor

tilmanginzel commented Aug 5, 2018

Hi @leonard84, while IntelliJ is correctly reporting 1 failed/1 skipped of 2 tests, libraries like spock-reports are accessing the SpecElementInfo.isSkipped() method, which is falsely reporting each feature as skipped.

It indeed seems like the StepwiseExtension is marking each feature as skipped, as soon as one fails. Is there a specific reason for it?

Do you know if it is possible to only mark all following features as skipped?

@tilmanginzel

This comment has been minimized.

Contributor

tilmanginzel commented Aug 5, 2018

Something like this would do the trick:

private void skipFeaturesAfterFirstFailingFeature(final SpecInfo spec) {
    spec.getBottomSpec().addListener(new AbstractRunListener() {
        @Override
        public void error(ErrorInfo error) {
            // @Stepwise only affects class that carries the annotation,
            // but not sub- and super classes
            if (!error.getMethod().getParent().equals(spec)) return;

            // get index of failed feature and mark all subsequent features as skipped
            int indexOfFailedFeature = spec.getFeatures().indexOf(error.getMethod().getFeature());
            for (int i = indexOfFailedFeature + 1; i < spec.getFeatures().size(); i++) {
                spec.getFeatures().get(i).setSkipped(true);
            }
        }
    });
}

Would you accept a pull request for this? :)

@leonard84

This comment has been minimized.

Member

leonard84 commented Aug 31, 2018

Yes, but when you do please use local variables instead of always retrieving the value anew.

@tilmanginzel

This comment has been minimized.

Contributor

tilmanginzel commented Sep 1, 2018

Hi @leonard84, thanks for the reply! I will do so in the coming week. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment