Customizable Failure Modes via Extensions

[james-newell-forge]

Hi Team,

I was looking to make a simple extension for MTP v2 that would emit a failure on a coverage threshold miss, but I kept running into major limitations in the extension architecture and implementation. Specifically, there seems to be no way to have an extension provide this kind of feedback to the TestHost without resorting to throwing an unhandled exception. From what I can tell, public extensions are basically limited to generating file artifacts and logging to the output device.

In my opinion, extensions would be much more useful if they had more power to customize the test flow. As already mentioned, being able to create new error rules would be really helpful, but also to be able to customize the output summary. As an example, in my use case, I would add a new summary line that includes a coverage line. I am sure there are many other use cases the community could come up with.

Example Summary

Test run summary: Failed!
  total: 22
  failed: 0
  succeeded: 22
  skipped: 0
  coverage: 68.4%
  duration: 9s 593ms

I am able to get around this limitation using a Harmony patch to insert my own error code, but it seems like a hack for something that should be supported as a first-class citizen.

[Evangelink]

Hi @james-newell-forge! Thanks for the feedback.

Just to be sure about your needs, you want to:

• Add some information to the summary
• Ensure the result is failed - do you need a specific exit code?