New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Additional Test Events for OneTimeSetUp / OneTimeTearDown #4652
base: master
Are you sure you want to change the base?
Conversation
@z002Holpp please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement ( “Agreement” ) is agreed to by the party signing below ( “You” ), 1. Definitions. “Code” means the computer software code, whether in human-readable or machine-executable form, “Project” means any of the projects owned or managed by .NET Foundation and offered under a license “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any “Submission” means the Code and any other copyrightable material Submitted by You, including any 2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any 3. Originality of Work. You represent that each of Your Submissions is entirely Your 4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else 5. Licenses. a. Copyright License. You grant .NET Foundation, and those who receive the Submission directly b. Patent License. You grant .NET Foundation, and those who receive the Submission directly or c. Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement. 6. Representations and Warranties. You represent that You are legally entitled to grant the above 7. Notice to .NET Foundation. You agree to notify .NET Foundation in writing of any facts or 8. Information about Submissions. You agree that contributions to Projects and information about 9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and 10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and .NET Foundation dedicates this Contribution License Agreement to the public domain according to the Creative Commons CC0 1. |
…or OneTimeSetUp / OneTimeTearDown.
Solution builds now with dotnet build command.
Event for OneTimeTearDown shall only be fired if there was a corresponding OneTimeSetUp. Additionally the events shall only be fired if there are corresponding methods in the SetUpTearDownItem objects.
…Down methods. Reused test fixtures from NUnit.TestData.OneTimeSetUpTearDownData
…e class and one in derived class).
… level (no inheritage). Test is failing since only one event for both events is raised. Tests is valid though.
In order to be non-breaking, changes done to the ITestListener interface have been reverted. Instead, a new interface ITestListenerExt has been intrtoduced. Also many changes done to implementations of ITestListener are reverted. This is the first part of changes done coming from review comments by Terje.
The changes were coming from previously trying out how to add new test cases. However the changes are not needed. Feedback from Terje was that MockAssembly should not be altered anyway. Therefore reverting.
…rDownItem. Events should be raised as close as possible to the method call. Thias was a feedback from Terje.
…w the test events arrive at a custom event listener that implements the ITestEventListener from NUnit.Engine.
@@ -323,6 +323,10 @@ private TestExecutionContext CreateTestExecutionContext(ITest loadedTest, ITestL | |||
// Set the listener - overriding runners may replace this | |||
context.Listener = listener; | |||
|
|||
// Set the listener for extended test events like OneTimeSetUp / OneTimeTearDown | |||
if (listener is ITestListenerExt testListenerExt) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mixing behaviour. Avoid this.
|
||
private static void RaiseStartedEvent(TestExecutionContext context, MethodInfoCache.TestMethodMetadata methodInfo) | ||
{ | ||
if (methodInfo.HasOneTimeSetUpAttribute) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do you check this? You should not be here if you were not in a Setup/Teardown method.
I assume you try to separate between pure OneTimeSetup/TD and the test Setup/TD. That is checked earlier. It might be necessary to state that information inside the SetUpTearDownItem. You find this in the TestContext that is passed along, there the test type is specified. If it is a test fixture it is onetimesetup, if it is a test it is a setup.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. Updated in latest commit.
Changed to distinguish OneTimeSetUp and SetUp with the TestContext.
else if (methodInfo.HasOneTimeTearDownAttribute) | ||
context.ListenerExt.OneTimeTearDownStarted(context.CurrentTest); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you need this if inside here, you know when you call what you have, you can just pass that along, or even better drop this method all together and just call in in-place.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. Updated in latest commit.
private static void RaiseFinishedEvent(TestExecutionContext context, MethodInfoCache.TestMethodMetadata methodInfo) | ||
{ | ||
if (methodInfo.HasOneTimeSetUpAttribute) | ||
context.ListenerExt.OneTimeSetUpFinished(context.CurrentTest); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The ListenerExt may not exist, thus be null, and if it is null, it should not be called.
context.ListenerExt?.OneTimeSetUpFinished(context.CurrentTest);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. Updated in latest commit.
Reverted original QueuingEventListener to be unchanged and added a separate QueuingEventListenerExt for custom events.
Used constrained model for collection asserts.
Differentiation between OneTimeSetUp and SetUp is now done with context.CurrentTest.IsSuite. That enabled the revert of all previous changes in MethodInfoCache. Added also a new test that checks that [SetUp] methods do not trigger OneTimeSetUp events.
We implemented a first draft of a solution that provides additional test events for OneTimeSetUp and OneTimeTearDown.
The main uncertainty during the implementation was to find the right place where the events should be raised.
Please find more information in the corresponding issue.
This PR should serve more as a base for a technical discussion rather than being "production-ready".