Request: Add "Otherwise" to Theories

[EarthCitizen]

My appologies in advance if there is a way to do this, but in my search I could not find one.

It would be immensly useful if there were an @otherwise for theories. What this would do is execute for a datapoint or set of datapoints if, and only if, no theory method had a passing assume. This would allow the avoidance of a theory or theories that are basically just negations of all the other assumes.

In the example below, without the @otherwise, it would be necessary to write multiple @theory methods to have all of the paths leading to false. The other alternative would be have another set of datapoints that should all return false, but that does not allow the potential to catch possible holes in logic that are not obvious at first glance. Additionally, if the datapoints are large-volume and generated with a supplier, it could be a daunting task to make sure you catch all the "other" scenarios. Simply catching "everything else" is much easier than making sure you capture all the reverse scenarios.

Obviously this specific scenario below is not very complex, but is is just to demonstrate the purpose of @otherwise.

@RunWith(Theories.class)
public class BooleanTest {

    @DataPoints
    public static Object[] values = {
        true, false, "true", "false", "1", "0", 1, 0, new Object(), null 
    };

    @Theory
    public void booleanTrue(Object value) {
        assumeThat(value, instanceOf(Boolean.class));

        assertThat(convertToBoolean(value), equalTo(value));
    }

    @Theory
    public void stringTrue(Object value) {
        assumeThat(value, instanceOf(String.class));
        assumeThat((String) value, equalTo("true"));

        assertThat(convertToBoolean(value), equalTo(true));
    }

    @Theory
    public void stringOne(Object value) {
        assumeThat(value, instanceOf(String.class));
        assumeThat((String) value, equalTo("1"));

        assertThat(convertToBoolean(value), equalTo(true));
    }

    @Theory
    public void integerOne(Object value) {
        assumeThat(value, instanceOf(Integer.class));
        assumeThat((Integer) value, not(equalTo(0)));

        assertThat(convertToBoolean(value), equalTo(true));
    }

    @Otherwise
    public void allOtherCases(Object value) {
        assertThat(convertToBoolean(value), equalTo(false));
    }

    public static boolean convertToBoolean(Object value) {
        if (value instanceof Boolean) {
            return ((Boolean) value).booleanValue();
        } else if (value instanceof String) {
            if ("true".equalsIgnoreCase((String) value)) {
                return true;
            } else if ("1".equalsIgnoreCase((String) value)) {
                return true;
            }
        } else if (value instanceof Integer) {
            if (((Integer)value) != 0) {
                return true;
            }
        }

        return false;
    }
}

[ashleyfrieze]

This would be a big change, since the current way that the Theories runner operates is by running all permutations through each method. What you're asking is for it to run all methods through each permutation and then drop to the methods marked with Otherwise when no method is interested in this permutation.

It might be interesting to see some real-world tests you're trying to write, taken away from the syntax of any specific test runner, just to see whether any other solutions pop up.

[EarthCitizen]

The main intentions are serving as a safety net for cases not thought of and to avoid writing assumes that negate (try to catch the opposite) of what all the other assumes are excluding. The latter can get quite complex when writing tests for legacy code.

[ashleyfrieze]

Have you a simple real world example?

[EarthCitizen]

Mainly where I run into problems is with code at work, which I (obviously) cannot post here. But the example I gave is similar to a situation I had to deal with for work.

[ashleyfrieze]

I've been thinking about this. The problem is the way the Theories runner operates. It runs in a Theory-centric way. This means each Theory method gets called for all data points. For there to be a drop through to an Otherwise method, you'd need one of two things. Either the runner would have to become data-point centric, running all Theories against each datapoint and then dropping through to the Otherwise if they all had assumption failures, or it would have to keep track of which datapoints have ONLY had assumption failures and then run the Otherwise method if the count of those was >0.

This leads to the fundamental problem with the otherwise method, which is that there could be >1 of them and it's not easy to see whether they would all have to run, and what you would do if there were assumption failures in those too. I think the result of trying to make the change as written might be quite counter intuitive.

I have an alternative suggestion, though. What if the assumptions could be made reusable and what if you could associate assumptions with Theories via an annotation, marking whether you're looking for those assumptions to be true or false. Allow me to rewrite your proposed code as though such a mechanism existed:

@RunWith(Theories.class)
public class BooleanTest {

    @DataPoints
    public static Object[] values = {
        true, false, "true", "false", "1", "0", 1, 0, new Object()
    };

    @Assumption
    public void booleanType(Object value) {
        assumeThat(value, instanceOf(Boolean.class));
    }

    @Theory
    @Assumes("booleanType")
    public void booleanTrue(Object value) {
        assertThat(convertToBoolean(value), equalTo(value));
    }

    @Assumption
    public void stringType(Object value) {
        assumeThat(value, instanceOf(String.class));
    }

    @Assumption
    public void stringValueTrue(Object value) {
        assumeThat((String) value, equalTo("true"));
    }

    @Theory
    @Assumes({"stringType", "stringValueTrue"})
    public void stringTrue(Object value) {
        assertThat(convertToBoolean(value), equalTo(true));
    }

    @Assumption
    public void stringValueOne() {
        assumeThat((String) value, equalTo("1"));
    }

    @Theory
    @Assumes({"stringType", "stringValueOne")})
    public void stringOne(Object value) {
        assertThat(convertToBoolean(value), equalTo(true));
    }

    // this method is marked with the assumption that all assumptions
    // in the class will pass
    @Theory
    @Assumes(Assumes.ALL)
    public void nothingIsNull(Object value) {
        assertNotNull(value);
    }

    // this method is marked with the assumption that none of the
    // assumptions in this class will pass
    @Theory
    @Assumes(not=Assumes.ALL)
    public void allOtherCases(Object value) {
        assertThat(convertToBoolean(value), equalTo(false));
    }

    public static boolean convertToBoolean(Object value) {
        if (value instanceof Boolean) {
            return ((Boolean) value).booleanValue();
        } else if (value instanceof String) {
            if ("true".equalsIgnoreCase((String) value)) {
                return true;
            } else if ("1".equalsIgnoreCase((String) value)) {
                return true;
            }
        } else if (value instanceof Integer) {
            if (((Integer)value) != 0) {
                return true;
            }
        }

        return false;
    }
}

The good thing about the above is that it makes assumptions something you could make reusable. It also gets away from the question of a drop-through theory, giving each theory a positive or negative selection of assumptions it can use. Not sure if it would be easy to implement. What do others think?

[EarthCitizen]

I like your re-usable assumptions idea much more than what I was originally proposing. This is a nice addition even outside of having a default catch-all.

It does though have one flaw that even my proposal had. And that is if you want to test more than one method of the class in the same test class. In this scenario there would need to be some way of grouping the theories/tests with an identifier.

@theory(groupName="methodOrScenario")
@test(groupName="methodOrScenario")

It would be nice to be able to do something like this:

@Test
public void testCatchAll() {
        assumeAllAssumptionFailuresForGroup("methodOrScenario");
        <perform catch-all testing>
}

Going with re-usable assumptions would not only affect the theories runner, but I believe even the default and parameters runners.

As a side note, I believe the assumption methods would also need to have the same parameter signature as the test/theory method?

[ashleyfrieze]

@EarthCitizen - good catch on the method signature of the @assumption method. I had intended it to take the same parameter as a @theory would for that class.

The idea of @assumes was that you'd specify the assumptions in that, which related to the test method.

E.g.

@Theory
@Assumes({"stringType", "stringValueOne")})
public void stringOne(Object value) {
    assertThat(convertToBoolean(value), equalTo(true));
}

And I also made up the "not" syntax so you could negate the assumptions

E.g.

@Theory
@Assumes(not={"booleanType")})
public void stringOne(Object value) {
    assertThat(convertToBoolean(value), equalTo(true));
}

You wouldn't need a special syntax for a group. Just create an assumption method which calls other assumptions. E.g.

@Assumption
public void anythingStringLike(Object object) {
    stringOne();
    stringType();
}

@Theory
@Assumes("anythingStringLike")
public void someTheory(Object o) {
}

I also don't think this could apply to the main runner as there are not usually parameters for test methods in that runner. I don't know if the technique is valid for runners outside of Theories.

I don't really know if this is the right thing to do anyway. It's been fun proposing a possible solution. What do others think? I wouldn't personally find a use for this in anything I'm doing.

[EarthCitizen]

@ashleyfrieze What I am trying to get at is that the "catch-all" would not need to be aware of what it takes to negate all the other conditions. The "otherwise" would fire when all of the other related methods threw an assumption exception and it would not need to even be aware of the necessary conditions that were needed to cause it's execution. With the @assumes(not=) you still need to be aware of what it takes to negate everything else and still leaves the possibility open of missing a scenario.

[ashleyfrieze]

@EarthCitizen I tried to cover that with a special string - Assumes.ALL - which means "all the assumptions in this class". so @assumes(not=Assumes.ALL) would mean "catch all cases where all assumptions fail". If you wanted a subset of those, then you could put them in an @assumption annotationed method.

Did I miss a use case?

[EarthCitizen]

In the case of @assumes(not=Assumes.ALL), which as you say is for the entire class, the person writing the test method for the "otherwise" case does not need to indicate any kind of logic of when to execute it. It just happens because all of the other methods had failed an assumption. There is absolutely nothing indicating any kind of condition around when to run it. And that is exactly matching what I am getting at with the original proposal.

But the other thing I am pointing out is that exactly the same thing should be possible, but only for a subset of methods. And by that I mean that the "otherwise" in that case should not require the test writer to specify any logic for its execution. Only that it executes when the other methods in that set all have an assumption failure. Let me make an example of what I mean:

class Foo {
    public void methodA() {}
    public void methodB() {}
}

class FooTest {
    @Theory(group="methodA")
    @Assumption("methodA_scenario1")
    public void methodA_scenario1() {}

    @Theory(group="methodA")
    @Assumption("methodA_scenario2")
    public void methodA_scenario2() {}

    @Otherwise(group="methodA")
    public void methodA_otherwise() {}

    @Theory(group="methodB")
    @Assumption("methodB_scenario1")
    public void methodB_scenario1() {}

    @Theory(group="methodB")
    @Assumption("methodB_scenario2")
    public void methodB_scenario2() {}

    @Otherwise(group="methodB")
    public void methodB_otherwise() {}
}

I use the @otherwise annotation here only for demonstration.

In the above example, note how each group has its own "otherwise". Also note how each otherwise is not told what conditions to check for. The otherwise is executed only if the other theories in the group all failed due to assumption failures. The otherwise is not even aware of what the assumptions of the theories in the group are. And this is intentional.

Hopefully that clears up what I mean.

[ashleyfrieze]

@EarthCitizen I think you may be looking for a gold plated solution.

The suggestion I made was based on lists of assumptions within the class being either required or not for a theory to run. That can do 80% of what you have described. You could create special composite assumptions for some more flexibility and you could have multiple test classes to deal with more complex groups, putting them in an Enclosed runner if you want them in the same file.

I don't know the value of this feature set to others. I can see how to implement what I have suggested. It is just you and me talking about this right now. Shall we see what others think?

[EarthCitizen]

Regardless of the otherwise feature I am talking about, I think your idea about reusable assumptions is a good one. You should see if it is feasible to implement.

[ashleyfrieze]

Thanks. I like the way it came out from our discussion. I may make a pull request to demo how if could be done.

[kcooney]

I actually don't use Theories, so I was hoping to be of the other maintainers would respond