RUN_IN_SEQUENCE doesn't work for multiple TestSuites

[doerk77]

Describe the bug
RUN_IN_SEQUENCE executes the tests on the first TestSuite, but the tests for the 2nd TestSuite are not being executed.

When RUN_IN_SEQUENCE calls AreAllTestsFinished(), it returns true (because no test has been registered yet) and therefore the tests never get executed.
But the tests can only be registered in TEST_ORDERED during execution of the test suite.

Note: It all works fine if the test suites are being executed in parallel.

Screenshots
Using TcUnit.RUN() and TEST()

Using TcUnit.RUN_IN_SEQUENCE() and TEST_ORDERED()

Software versions
v1.2.1.0

[sagatowski]

Your output window looks weird, you get double outputs for some reason.

1. Did you define the tests and test-suites correctly?
   You have example projects for this use case here:
   https://github.com/tcunit/ExampleProjects/tree/master/RunTestsInSequenceExampleProjects

Please also check this FAQ: https://tcunit.org/frequently-asked-questions/?Display_FAQ=970

2. Do you have an example project that re-creates this issue?

Note that version 1.2.1.0 of TcUnit doesn't exist. The latest release is 1.2.0.0.

[doerk77]

Just tested with your sample project

As you can see there are clearly more than 4 tests throughout the various test suites. But only 4 tests get executed.

XML output states the same.

PS: I'm using the latest (unreleased) version from GitHub which tells 1.2.1.0.

[sagatowski]

I tested it with the latest release 1.2.0.0 and got the following:

Which means that something in a commit after the release has broken this.

@doerk77 Could you look into it and create a PR?

[sagatowski]

Make sure to read the contributing. Thanks.

[sagatowski]

Here are all the commits since the release of 1.2.0.0.

[sagatowski]

Also, if you could think of any good test to add to the TcUnit-Verifier to cover this case, that would be awesome!

[doerk77]

Happened between Commits 5059342 and c7d7021

First guess is that this is causing the issue:

(Left is 5059342 )

But need to check in detail.
Hadn't been clean before. Therefore the workaround incl. comment ;-)

[sagatowski]

@doerk77 seems reasonable. Good work! TcUnit always needs more contributors!
Bad of me to not discover this during code review :/

Would be really good if the TcUnit-Runner would detect this failure, so we don't end up here again.

[doerk77]

Question: Is there a reason why registering tests is only possible while the testsuite is running?

In general I'd allow it the whole time until the testsuite actually starts running. At this point in time the list should not change any more.
That way it would even allow to register tests during FB_Init or in a method with attribute 'call_after_global_init_slot'.

Not that easy to write a test how it might be possible to detect this kind of issue though... Let me think about it.

[doerk77]

Your output window looks weird, you get double outputs for some reason.

Seems to be an ADSLOGSTR / TwinCAT feature if you have 2 projects in a single solution.

[doerk77]

Your output window looks weird, you get double outputs for some reason.

Seems to be an ADSLOGSTR / TwinCAT feature if you have 2 projects in a single solution.

FYI: Just checked with Beckhoff support. Inside the IDE there's an event listener instantiated per project on the PLC. Therefore the event from ADSLOGSTR is being written to the Error List window twice.
However, as far as I can see this shouldn't cause any trouble. It's just a little annoying to see it pop up multiple times.

[sagatowski]

@doerk77 Yes, you are correct. I just remember that there already was an issue on this particular topic:
#92

[sagatowski]

Solved. Thanks for finding the problem @doerk77.