CombinatorialMemberData cannot be of type IEnumerable<object[]>

[siewers]

The CombinatorialMemberDataAttribute requires the member to return an enumerable which is not an IEnumerable<object[]>.
This breaks compatibility with the Xunit MemberData attribute, which requires the member to return exactly an IEnumerable<object[]>.

We should figure out how to support the same member data type as Xunit support, so we can use the same member data for both MemberData and CombinatorialMemberData.

I think I might have broken that in the latest prerelease version.

[siewers]

Right now I have to do something like this, which is a bit annoying.

public static IEnumerable<string> GetValues 
{
    get 
    { 
        yield "Foo";
        yield "Bar";
        yield "Baz";
    } 
}

public static IEnumerable<object[]> GetValuesXunit 
{ 
    get { return GetValues.Select(x => new object[] { x }); }
}

[Theory]
[MemberData(nameof(GetValuesXunit))]
public void TestWithMemberData(string value)
{
}

public void TestWithCombinatorialData(
    [CombinatorialMemberData(nameof(GetValues))] string value, 
    [CombinatorialValues(1, 2, 3)] int number)
{
}

[siewers]

When I think about it, I'm not sure this is even possible to solve. It also seems like a rather odd constraint in xUnit. It should be able to verify any IEnumerable<T> to match the method signature.

[siewers]

Perhaps using TheoryData<T> could fix it, but currently CombinatorialMemberData doesn't understand that.

[AArnott]

I don't see how these two would be compatible anyway. CombinatorialMemberDataAttribute takes values for an individual parameter, whereas MemberDataAttribute is applied to the whole method and therefore takes values for all the parameters. They are fundamentally different data and applied to different places.
What's more, the whole point of CombinatorialMemberDataAttribute is to apply to just one parameter so that the set of test cases can be generated by the library, whereas MemberDataAttribute requires you to generate the test cases yourself.

So I'd say this is By Design.

[siewers]

What do you think about supporting TheoryData<T>?
That should be doable and would probably support both scenarios 🤔

[AArnott]

I don't know anything about TheoryData<T>. If it's something you're proposing, I'd need more of a spec to comment on it. A new issue would be appropriate for that.

[siewers]

Yeah, it's also not very well documented, if at all. I haven't found any official documentation about it, but Andrew Lock has a nice blog post about it.

I'm a bit reluctant whether it should be supported, given the feature itself is so poorly documented.
I guess it wouldn't hurt and my best guess is that the xUnit maintainers simply just don't have a lot of documentation at all.

If you think it's worth supporting, I'd be happy to create a PR for it, I just don't want to waste my time if you think the idea is pointless 🙂

[AArnott]

Cool. But I still need to understand how that would be used and apply to this library. Thank you for volunteering to contribute to the library. I'd be interested to read sample code and a brief explanation of its behavior as it would apply to this library (no need to make it actually work yet).

[silkfire]

@AArnott

Supporting 1-dimensional TheoryData (i.e. only TheoryData<T> with one type parameter) would be very convenient.

The conversion from object[][] to IEnumerable<T> goes as follows:

return new MyTheoryData().Select(d => (MyType)d.Single());

[siewers]

@silkfire

As part of #95 I'm working on supporting a part of this. If the idea is approved, I intend to support TheoryData (or basically IEnumerable<object[]>) on the CombinatorialMemberDataAttribute as well, which will also get a generic counterpart.

[siewers]

It's not technically impossible to support IEnumerable<object[]> since that is intended to map to multiple parameters in a test method. TheoryData, however, maps to a single type, which is exactly what will make tests a lot easier to read, write and maintain. With TheoryData the type can be whatever you want it to be, so it should be flexible enough to support most cases.

The idea is to allow this:

public class Tests
{
    public record MyTestCase(int Number, string Text);

    public static readonly TheoryData<MyTestCase> MyTestCases = new(
        new MyTestCase(1, "Foo"),
        new MyTestCase(2, "Bar")
    );

    [Theory, CombinatorialData]
    public void TestMyTestCases([CombinatorialMemberData(nameof(MyTestCases))] MyTestCase testCase, bool flag)
    {
        /*
            This will give you:
            testCase(1, "Foo"), true
            testCase(1, "Foo"), false
            testCase(2, "Bar"), true
            testCase(2, "Bar"), false
        */
    }
}

[siewers]

It can also only support TheoryData<T> (with a single generic parameter, since multiple generic parameters act similarly to IEnumerable<object[]> that maps to multiple method arguments.

[AArnott]

Reactivating due to recent activity. I'll review the proposal and get back to you soon.

[siewers]

@AArnott have you had time to review the proposal yet?