Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Our test result reporting does not take into account fails after test done #691
Currently, then you run the full coordinated run on desktop, you might see the end report suggesting that everything passed, but in the logs, there might be output like:
The tape test report reports these as failures, but we don't, because we report to coordination server after each test if there were failures, but don't take into account in case a failure in test occurs after the test itself has completed.
This is not super critical and often the issue is just not cleaning up all events handlers at the end of each test. However, this results into tape test report suggest failures we our coordination server thinks all went fine.
@artemjackson This is a relatively easy first bug for you. The issue is that when we run a test in node there are three parts to each and every test. There is setup, the test and then tear down. When we run a 'coordinated' test our coordinated test framework which lives under test/TestServer waits for each device to connect to it and say "I'm ready to run setup". So the test server just waits until everyone says "I'm reading to run setup" and then once it gets o.k. from everyone it's been told to expect then it sends out a command 'run setup'. Once everyone says 'I ran setup' then it says 'run test' and same thing with 'run teardown'. It will keep going until the tests are done. All of this logic btw is defined in test/TestServer/UnitTestFramework.js.
The problem is that we have had failures in tear down that don't seem to be noticed by CI. Looking here it seems like we should notice when a test fails. And when I look here it looks like the client is sending information on setup and teardown success or failure. So what's going on?
Did the bug get fixed and we didn't notice?
So the immediate work item is to add tests to testUnitTestFramework that confirm if we throw an exception or if we fail a test (e.g. call t.fail() for example inside of setup or teardown) then the unit test framework will properly notice this.
Then you need to setup a local test of the coordinator (e.g. the code in test/TestServer) to make sure that if we get an error in setup or teardown that this will be properly reflected in the test results.
According to manually running coordinated tests logs all failures were properly handled by
So, there are some questions.