Custom Frameworks - Report on skipped/ignored tests

[jan-molak]

At the moment Protractor expects the test frameworks to only notify it when a test passes or fails.

Would you be happy to consider extending this functionality to allow the test framework to notify protractor of tests being skipped/ignored too?

This would enable better integration of Protractor-based test frameworks (such as Serenity/JS) with IDEs, such as IntelliJ - please see the conversation here.

What I believe could be done, since Protractor doesn't seem to be using the event emitter for any other purpose, is to:

• add an additional event testDone so that a test framework can notify Protractor when it's done with a test (this would be in line with Protractor's naming convention: testPass, testFail, testsDone)
• extend the Protractor plugin API, adding an optional onTestDone(): Promise | void method, so that it doesn't break backwards compatibility with existing Protractor frameworks and plugins)
• pass the testDone events received by Protractor through runner.emit onto the above plugin interface (probably excluding the testPass and testFail to keep it simple) somewhere around here
• extend the taskRunner to print a comma , (like in Mocha) for any tests that finished without explicit pass or fail status (so with testDone)
• maybe other things I might have missed

With the above functionality, the following scenario could be made possible:

Reporting on ignored tests

• the test framework encounters an ignored test
• the test framework notifies Protractor with runner.emit('testDone' , { name: ..., category: ..., 'additional-meta-data-for-the-ide': { ... } } (as per the spec)
• Protractor prints a , to signify that the test has neither passed nor failed
• IDE notifies the developer that a test was ignored

The reason why I'm suggesting to call the event testDone rather than testIgnored is because the metadata sent to the IDE could be different when a test is "pending" (in cucumber-speak) or "ignored" (xit in Mocha) and the semantic meaning of "ignored", "pending" and "skipped" is different. Keeping it limited to testDone should do the trick.

Looking forward to hearing your thoughts!

Jan Molak