feat: add QA Wolf plugin

[flaurida]

Thank you for your consideration 🐺

Are you adding a plugin or updating one?

• Adding a plugin
• Updating a plugin

Have you completed the following?

• Read and followed the plugin author guidelines.
• Included all required fields in your entry.
• Tested the plugin locally and on Netlify, using the plugin version stated in your entry.

Test plan
This plugin is pretty lightweight - it calls the QA Wolf API to run end-to-end tests on Netlify deployments.

I have included some basic smoke tests in the plugin itself and more comprehensive tests in the QA Wolf repo, where the API lives.

Example project with this plugin
Successful public deploy log
Follow up PR to update README with UI install instructions

Please let me know if I can do anything else, thanks in advance 🙏

[netlify-bot]

Added Packages

• https://registry.npmjs.org/netlify-plugin-qawolf/-/netlify-plugin-qawolf-1.0.3.tgz

[JGAntunes]

Hey @flaurida 👋 thanks a lot for your awesome contribution!

I'm still running through the plugin and I'll be opening a couple of issues to track specific suggestions if that's ok with you. However, I do have some broader questions that are maybe easier to follow up in here, hope you don't mind.

I noticed the plugin is executing on onPostBuild and onSuccess - https://github.com/qawolf/netlify-plugin-qawolf/blob/main/src/index.js#L3-L12 - if I understand QA Wolf correctly, the E2E test suites will execute based on the deploy URL generated for that specific build correct? If that is the case I'm affraid onPostBuild cannot be used given that, at the time of execution, the deployment still hasn't been performed 🤔 meaning whatever execution you're running will be on top of a previous deployment for that same URL. I undertsand the logic behind wanting to fail the build, however we're unable to fail a build that has already been deployed (which I believe is a requirement for QA Wolf?).

I was also trying to test it out myself - https://app.netlify.com/sites/ntfy-init-test/deploys/606471c1c93a6957f1f31cff - however seems like I'm unable to trigger the test suite (I've went through the setup process of setting up a test and trigger, but maybe I'm missing something).

[flaurida]

Hi @JGAntunes thank you for the quick review 🐎

I saw your comments in the plugin and am happy to address those there. In terms of your other questions:

It sounds like I misunderstood the onPostBuild event - good catch 😅 . I wanted to solve for the use case that we could block / rollback production deployments if tests failed, but it seems like that is not possible if we need a public URL before running the tests. I'll update the plugin to only run onSuccess then.

Sorry the test didn't work for you - I haven't deployed our changes to support this plugin to production yet since I wanted to see if we could get the plugin in the Netlify UI first. If you'd like to test it out, you can use our staging URL (https://spirit.qawolf.com), and set the QAWOLF_URL variable to that URL as well in your Netlify project.

Thanks again :)

[flaurida]

I've updated the plugin to only run onSuccess, and used failPlugin rather than failBuild where you suggested. I also updated the links to public deploy logs in the README with a new example.

Let me know if I can make any additional changes 🙏

[JGAntunes]

Sorry the test didn't work for you - I haven't deployed our changes to support this plugin to production yet

No worries at all, was just in the process of reviewing and thought that maybe some of the questions I had could be addressed if I tried it out myself 😄

I've updated the plugin to only run onSuccess, and used failPlugin rather than failBuild where you suggested. I also updated the links to public deploy logs in the README with a new example.

Just saw it, I think we're good to go! 👍

[flaurida]

Thank you! 😊

[JGAntunes]

And it's out! - https://app.netlify.com/plugins/ - You can now find your plugin in our official list 😄 @flaurida

Thank you! 😊

Thank you! 🙌