Capability of repeat (number of times) for a test with where: block

[Guter1988]

Is your feature request related to a problem?

Description

I have tried to implement a repeat extension as fedor.belov wrote here.
With small change of public class RepeatExtension implements IAnnotationDrivenExtension<Repeat> instant of public class RepeatExtension extends AbstractAnnotationDrivenExtension<Repeat>
This is working great in tests without where: block.
With where block (in debugging) can see that children created under this and as object himself.
I'm not sure but it look why it delicate the repeat in

To Reproduce

Add extension as fedor.belov wrote and try to run @repeat in test with where: block more that 2 times.

Describe the solution you'd like

Be able to have this @repeat extension and use it with where: block

Describe alternatives you've considered

No response

Additional context

No response

[kriegaex]

For me, it does not need a where: block in order to fail with 2+ iterations. I quickly tried in Spock 2.1-M2 on Groovy 3. Which versions did you use? FYI, @Guter1988, writing such an extension has been discussed briefly on an internal Spock developer channel about 6 weeks ago already. For now, this is not easily possible, because simply calling IMethodInvocation::proceed multiple times on the same invocation instance does not initialise everything you would need from scratch.

By the way, this question seems to be kind of a duplicate of #1285. You will also find the simple workaround where: run << (1..10) there. I am not really convinced why repeating tests would be a good idea, other than maybe in a load-testing scenario. I know that some people want to repeat tests until they pass because they are flaky, but I do hope that this is not your use case.

[Guter1988]

Hi @kriegaex
I needed a where: block (use CSV where each row is a test and provide it with where block), and I need to repeat (number of times same test over and over again).
Thank you for the link. the workaround succeeded. It worked with adding multivariable data pipes.
Repeating tests are used in my performance testing, as performance methodology advises to use 3 or 5 times.

[kriegaex]

Yes, of course you should use a where: block if you need one. What I said was that even without a where: block, @Repeat(2) or higher as described in Fedor Belov's blog post is not working. Your subject line and description are imprecise there, possibly making readers think the problem only occurs in parametrised tests.

[leonard84]

You can achieve this with the new IDataDriver added in #1479

[kriegaex]

You can achieve this with the new IDataDriver added in #1479

@leonard84, is there any example for how to do this? The existing data driver tests have not been very instructive after an - admittedly, cursory - glance, and there is nothing about it in the Spock manual either. Maybe you can give a few hints or sketch a solution. I do not need it at the moment, I was just curious and had your comment on my follow-up list for a while. So before I forget about it, ... 😉

[leonard84]

@kriegaex here is an example from an yet unfinished experiment that can easily be adapted. Note, this only works, if the feature is data-driven otherwise the DataDriver will not be used. For non data-driven features you can use some hackery to get it working as gradle did.

import org.spockframework.runtime.extension.ExtensionAnnotation

import java.lang.annotation.ElementType
import java.lang.annotation.Retention
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Target

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@ExtensionAnnotation(RepeatUntilFailureExtension)
@interface RepeatUntilFailure {
    long max() default Long.MAX_VALUE
}

import org.spockframework.runtime.IDataIterator
import org.spockframework.runtime.extension.IAnnotationDrivenExtension
import org.spockframework.runtime.extension.IDataDriver
import org.spockframework.runtime.extension.IIterationRunner
import org.spockframework.runtime.model.ExecutionResult
import org.spockframework.runtime.model.FeatureInfo
import org.spockframework.runtime.model.ParameterInfo

class RepeatUntilFailureExtension implements IAnnotationDrivenExtension<RepeatUntilFailure> {
    @Override
    void visitFeatureAnnotation(RepeatUntilFailure annotation, FeatureInfo feature) {
        feature.dataDriver = new RepeatUntilFailureDataDriver(annotation.max());
        super.visitFeatureAnnotation(annotation, feature)
    }

    static class RepeatUntilFailureDataDriver implements IDataDriver {
        private final long max;

        RepeatUntilFailureDataDriver(long max) {
            this.max = max;
        }

        @Override
        void runIterations(IDataIterator dataIterator, IIterationRunner iterationRunner, List<ParameterInfo> parameters) {
            List<Object[]> params = dataIterator as List<Object[]>
            (1..max).forEach {
                params.each {
                    def executionResult = iterationRunner.runIteration(it).get()
                    if (executionResult != ExecutionResult.SUCCESSFUL) {
                        throw new RuntimeException("Failed after $it iterations for params: $params")
                    }
                }
            }
        }
    }
}