Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic loading of tests #7

Open
CharliePoole opened this issue Oct 19, 2013 · 18 comments

Comments

@CharliePoole
Copy link
Member

commented Oct 19, 2013

Obviously, the current implementation is the following:

  • First, the Properties which are used for TestCaseSource are asked (static or not)
  • Then the methods attributed with TestFixtureSetUp are called.
  • Then test startup is done
    and the test called.

It is possible to change the first two steps? The current behaviour is a problem for us. Because the property for the TestCaseSource should have some data from the database which the Connection is established in FixtureSetUp. The parameter vor the database, username, password and so on are set in the constructor of the class.

More info at https://bugs.launchpad.net/nunit-3.0/+bug/664018

@larsmoa

This comment has been minimized.

Copy link

commented Sep 1, 2014

I'm facing a similar issue. Basically, the workaround for me is to call the setup-method from the test case source. However, I wish that this wasn't necessary.

@CharliePoole CharliePoole added this to the Future milestone Feb 21, 2015

@CharliePoole CharliePoole added pri:low and removed pri:wishlist labels Jul 18, 2015

@CharliePoole CharliePoole modified the milestones: Backlog, Future Dec 4, 2015

@dicko2

This comment has been minimized.

Copy link
Contributor

commented Jan 28, 2016

I'm pretty sure this is similar to what i am having issues with so decided not to create a new issue. Here is a good stackoverlfow with info and examples of where people are having issues

http://stackoverflow.com/questions/28603507/c-sharp-nunit-testcasesource-passing-parameter

I think this would be pretty easy to fix though, by adding the ability to specify an object array parameter for the attribute, which is then passed to the constructor

The Reflect.Construct method, which is used in the TestCaseSourceAttribute, has an option for an object parameter already.

public static object Construct(Type type, object[] arguments)

This accepts null argument as well, which makes it even easier.

I understand this has been marked as low priority but looking at the stack overflow post people are having issues with it. And quite frankly myself coming from MSTest, their Data Source Attribute is awesome, and the major thing stopping me from moving a number of projects over to nunit.

@rprouse

This comment has been minimized.

Copy link
Member

commented Jan 28, 2016

@dicko2 thanks for posting and linking to the SO question. People commenting on issues like this strongly influences the priority in which features are added to NUnit.

We also accept pull requests 😉

@dicko2

This comment has been minimized.

Copy link
Contributor

commented Jan 29, 2016

Yea, I tried it out and spent a few hours on it last night after saying "this will be easy"... unfortunately adding parameters to the constructors gives me no love, because the static methods seem to be called before hand by something in the internal library to get a list of the test cases available from the TestDataSource Object/Method, it shows up funny results in Resharper's UI to boot.

I think it would be best not to use static methods for the data source and instantiate the object instead, but changing this would be a "little bit" of a breaking change. I'll have another look on the weekend if i get time, I've resigned myself to the work around of creating a new class for each of the parameters for the time being, as I've only got one project that needs moving urgently.

I might be able to get it to work though and send through a PR, I've got a few ideas.

Is there anyone i can talk to, to get advice on how nunit is put together and the best way to debug changes @rprouse ? do you guys have a gitter chat room? facebook page? gopher server or something?

@CharliePoole

This comment has been minimized.

Copy link
Member Author

commented Jan 29, 2016

Although it happens at runtime, we refer to the current implementation of test cases as "static." What we mean by that is that they are created once, when the tests are loaded, and don't change when the tests are executed.

What you are seeing wtih the calls to your source methods is NUnit actually loading (discovering) the tests, which it will subsequently run. The distinction between load-time and (test) run-time may seem academic if you are thinking of batch execution where the second happens immediately after the first. But if you imagine a gui, the tests are loaded and displayed first, then are run one or more times. That gui display before running the tests would not be possible without "static" loading of tests.

That's the reason OneTimeSetUp cannot affect the loading of tests. The setup is part of the execution phase, which happens only after the tests are loaded. You can think of it like this: we can't run the tests till they exist and they don't exist until we go through the load process.

The reason we call this "static" loading is that we also have a notion of "dynamic" loading, which would take place at the time the tests are executed. That would mean that a parameterized test method could end up getting different data each time it is run in a gui. We don't have such a feature yet - it's just a plan for the future. Since we don't seem to have a main issue for this feature, I'm re-titling this one to reflect it.

Note that it would be possible to allow an argument to be passed to a TestCaseSource method without dynamic test cases. You just could not influence those arguments in your setup. If somebody wants to have the ability to specify some sort of constant argument on the TestCaseSource attribute, that should be entered as a separate issue.

@CharliePoole CharliePoole changed the title TestCaseSource and TestFixtureSetUp TestCaseSource and TestFixtureSetUp (Dynamic Test Cases) Jan 29, 2016

@dicko2

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2016

Thanks @CharliePoole that's some good info for me to work off.

I now understand that i was approaching the issue wrong. Instead of looking at the constructor i should have been looking at the static method to pass parameters too. I've almost got a PR ready to go but one of the unit test is failing, it does look like its meant to fail though as there is an exception throw called "my message" hardcoded into the method.

This is the output i get in resharper

System.Exception : my message at NUnit.Framework.Attributes.TestCaseSourceTests.<get_exception_source>d__39.MoveNext() in D:\SOURCE\Repos\nunit\src\NUnitFramework\tests\Attributes\TestCaseSourceTests.cs:line 325 at NUnit.Framework.TestCaseSourceAttribute.GetTestCasesFor(IMethodInfo method) in D:\SOURCE\Repos\nunit\src\NUnitFramework\framework\Attributes\TestCaseSourceAttribute.cs:line 156

It looks like its meant to fail though so i'll send in the PR. I've just added the ability for parameters to get passed if the target is a method, if its a property or field the additional parameter will be ignored.

So it can be used like this

[Test, TestCaseSource(typeof(DivideDataProvider), "HereIsTheDataWithParameters", new object[] { 100, 4, 25 })]

I guess this should be a separate issue though as you mentioned.

@CharliePoole CharliePoole removed this from the Backlog milestone Jul 25, 2016

@CharliePoole CharliePoole changed the title TestCaseSource and TestFixtureSetUp (Dynamic Test Cases) Dynamic loading of tests Aug 19, 2016

@CharliePoole

This comment has been minimized.

Copy link
Member Author

commented Aug 19, 2016

I renamed this issue to reflect the fact that it has always been the key issue related to what we call dynamic tests, but was hard to find. I'm making it an epic and will add to it any issues that are prerequisites or dependencies of the dynamic test feature.

@rprouse

This comment has been minimized.

Copy link
Member

commented Aug 29, 2017

As a first pass at this, if we have filters, can we skip test loading when we know those tests won't match the filter? For example, if we are filtering on a namespace, skip loading all tests not in that namespace? We won't catch every situation with every filter, but we could speed up test runs for common scenarios like running individual tests from Resharper or the test adapter where we are passed in the full test name.

@CharliePoole

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2017

<meta>
@nunit/core-team I'm looking at this issue in a browser with the ZenHub extension disabled. I suggest everyone try it once, because what the user sees is pretty useless. There's a comment that says I'm making this issue an epic and adding other things to it, but there is no visible indication as to what those other issues are.

I've been questioning the usefulness of ZenHub - or at least the Epic feature - for a while. In the NUnit gui, I removed the webhooks from the repo, which is why I happen to be browsing with ZenHub disabled. What I was hoping to acheive is that all users would see the same thing we do.

Maybe we should think about this for the NUnit Project as well. Meanwhile, I'll switch it on and off in my browser as need be.
</meta>

@CharliePoole

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2017

Currently, no filters are passed until the test is run, because it's possible that a GUI is calling us and we want to be able to see all the tests. In the case of the console, however, we load the tests, run the tests and exit. So it's theoretically possible to take filters into account.

The framework accepts a package setting: FrameworkPackageSettings:LOAD = "LOAD", whose value should be an IList of fixtures to load. Actually, each entry can be the full name of a fixture or a namespace. Only those fixtures will be loaded. Currently, the setting is not exposed to users although we do use it in a few of our tests for efficiency. The copy of FrameworkPackageSettings.cs in the console project has a comment on that entry saying "Remove?".

In older versions of NUnit V2, the equivalent setting was exposed as /fixture. You could specify a single test fixture or namespace and only that was loaded and run. In later versions, it was replaced by /run and subsequently by /test.

It has always been my idea that NUnit is free to decide whether to apply a simple name filter at the stage of loading or running, at least in the console runner. I would see no problem if we applied --test in that way by adding the setting to the package.

Limitations of this approach:

  1. It only impacts the console runner. The GUI and adapter would need to do something else.
  2. If the --where option is used, there isn't much we can do in the console runner. It doesn't understand filters at all any longer, but simply asks the engine to interpret them. If we wanted to extend this to --where filters, we would have to have a way to tell the engine to apply filters at load if possible. That's because a GUI runner probably would not want this to happen, while a batch runner might.

None of this has a lot to do with dynamic tests, at least not as I intended the term.

@CharliePoole

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2017

Concluding, from my earlier comment...

I think we could trivially implement application of --test at load time. AFAICS it would be transparent to the user and would work in combination with any --where clause that contained other filters like categories`.

Application of filters like --where test==SomeTest is a bigger deal, since the engine needs to be modified. More complex filters are an even bigger deal.

@jnm2

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2017

I'm ambivalent about ZenHub. It's good for moving issues, but I always felt our pipelines overlapped the milestones feature.

@CharliePoole

This comment has been minimized.

Copy link
Member Author

commented Aug 30, 2017

Basically, this suggestion was that we stop using the Epics feature and do something else to tie issues together in groups. That could be simple task lists with references to issues or Projects, now available in GitHub. Epics are not viewable by users without ZenHub. We could still use ZenHub to move issues and create boards.

@jnm2

This comment has been minimized.

Copy link
Contributor

commented Aug 30, 2017

I'm in favor of task lists replacing epics.

@RafalChlopczyk

This comment has been minimized.

Copy link

commented Jan 30, 2018

Maybe this tip will help somebody till dynamic loading of tests' data is implemented in NUnit.

I used to write tests like this (it is simplified code for presentation purposes):

[TestFixture]
public class MyTests
{
    [Test, TestCaseSource(typeof(MyDataClass), "TestCases")]
    public void SomeTest(MyTestCase testCase)
    {
        //perform test with use of testCase
    }
}

public class MyDataClass
{
    public static IEnumerable<MyTestCase> TestCases
    {
        get
        {
            yield return new MyTestCase { /* test case data initialization */};
            // ...
        }
    }  
}

public class MyTestCase
{
    //some complex data in here
}

But it turned out that initialization of many complex test data classes takes too much time. So I moved tests' data initialization moment from loading of tests by NUnit to actually every single test execution.
Now I write tests that need complex data like this:

[TestFixture]
public class MyTests
{
    [Test, TestCaseSource(typeof(MyDataClass), "TestCases")]
    public void SomeTest(Func<MyTestCase> testCaseFactory)
    {
        MyTestCase testCase = testCaseFactory();
        //perform test with use of testCase
    }
}

public class MyDataClass
{
    public static IEnumerable<Func<MyTestCase>> TestCases
    {
        get
        {
            yield return () => new MyTestCase { /* test case data initialization */};
            // ...
        }
    }  
}

public class MyTestCase
{
    //some complex data in here
}

And if I need to make use of TestCaseData and eg. its SetName() method then I write it like this:

yield return new TestCaseData(new Func<MyTestCase>(() => new MyTestCase { /* test case data initialization */})).SetName("some test name");

This way of writing test cases drastically shortens time to load tests by NUnit.

@soul4soul

This comment has been minimized.

Copy link

commented Mar 14, 2019

I'm running into this problem now. My tests require information to be passed in from parameters. I then use this information to create a class which generates data that is used by TestCaseSource properties. Initially it made the most sense to read the parameters in the OneTimeSetup of a SetUpFixture and create my class there. Obviously that didn't work at expected. I was able to come up with workarounds similar to those mentioned in this thread.

I came here to post this issue. This is my first time using Nunit and I'v already hit this problem. I'm surprised it's not a higher priority. Even the example in the OP seems like it would be commonly effecting more users.

@ChrisMaddock

This comment has been minimized.

Copy link
Member

commented Mar 17, 2019

@soul4soul - Are you able to solve your problem using TestFixtureSource to generate your classes?

@soul4soul

This comment has been minimized.

Copy link

commented Apr 3, 2019

I suppose that would work. I can create a TestFixtureSource and use it for both the SetUpFixture and test Fixtures. In the TestFixtureSource I can read in the test parameters and set up all of the classes which generate data. Some of the classes I'm setting up are handles for equipment, which contain information that effects TestCaseSourceData attributes. I would have to move the initialization code from the SetUpFixture OneTimeSetup to the TestFixtureSource. Then In the SetUpFixture I would only use the OneTimeTearDown to clean up the handles. I'll code it up to see if I like it any better then my current implementation. I think the code would still be awkward compared to actually fixing the issue but it might be better then what I have now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.