Create integration test script

[antross]

Currently CI runs webhint against the site, but doesn't verify a scan can actually be performed. We've had this result in bad builds being promoted to production in the past. Using something like puppeteer to actually drive the website in a more comprehensive integration test before the swap to production should help avoid repeating this problem in the future.

[molant]

We can probably create a script that uses puppeteer to open a browser, go to the staging website, navigate to different pages and verify we receive a 200 for all of them and finally performs a scan and a search.

We will be calling that script from the pipelines once it is done.

[LeoSL]

Hi guys, I was planning to dive into this issue since I like CI/CD integrations and it says that the difficulty is easy and also has the good first issue.

However, after a few minutes investigating the repo and executing the setup process I started to have some questions:

• Which CI @antross is making reference to?
• I'm also lost on which staging website @molant is making reference to - to run the integration test. Besides that, where exactly the puppeteer script will perform before the CI runs
• gulp issues on npm run build (detailed below).
• the issue requires at least some basic puppeteer knowledge and from what I saw on the repo there's no puppeteer configured.

That said, I'd argue that this is not a good first issue and in my opinion, not so easy. Making the change could be straightforward to someone familiar with wehint.io current architecture and test suite. However, for someone that is just coming here from scratch, it's kind of tricky.

• gulp issue

[18:59:00] Error in plugin "gulp-imagemin"
Message:
    spawn /Users/leosl/src/webhint.io/node_modules/optipng-bin/vendor/optipng ENOENT
Details:
    errno: ENOENT
    code: ENOENT
    syscall: spawn /Users/leosl/src/webhint.io/node_modules/optipng-bin/vendor/optipng
    path: /Users/leosl/src/webhint.io/node_modules/optipng-bin/vendor/optipng
    spawnargs: -strip,all,-clobber,-o,3,-out,/private/var/folders/z_/rcyh1f793y7d3k6w5wvrzg0c0000gn/T/a369bbd5-57f6-4cb1-9292-4910ec8241d7,-fix,/private/var/folders/z_/rcyh1f793y7d3k6w5wvrzg0c0000gn/T/88702354-471a-49f5-b5a9-f5a3c075dbc8
    killed: false
    stdout:
    stderr:
    failed: true
    signal: null
    cmd: /Users/leosl/src/webhint.io/node_modules/optipng-bin/vendor/optipng -strip all -clobber -o 3 -out /private/var/folders/z_/rcyh1f793y7d3k6w5wvrzg0c0000gn/T/a369bbd5-57f6-4cb1-9292-4910ec8241d7 -fix /private/var/folders/z_/rcyh1f793y7d3k6w5wvrzg0c0000gn/T/88702354-471a-49f5-b5a9-f5a3c075dbc8
    timedOut: false
    fileName: /Users/leosl/src/webhint.io/src/webhint-theme-optimized/formatter/images/json-output.png
    domainEmitter: [object Object]
    domain: [object Object]
    domainThrown: false

[18:59:00] 'optimize:images' errored after 421 ms

[molant]

Hi @LeoSL

I'm sorry you are having issues with this one. We assumed some experience with puppeteer when we tagged it but you are right that could be a bit misleading.

The CI server we are using is Azure Pipelines. You can see the builds for all the webhint projects in here.

The file that decides what is run in Azure DevOps is azure-pipelines.yml

The staging environment is https://sonarwhal-staging.azurewebsites.net. It should look almost the same than the official website.

So this is what I'm thinking as a breakdown:

1. Add puppeteer to the project
2. Create /helpers/integration-tests.js. This is the script with the integration tests. This script should:
   • Start a puppeteer instance
   • Navigate to the following URLs and verify the status code is 200 for each one of them:
     • https://sonarwhal-staging.azurewebsites.net
     • https://sonarwhal-staging.azurewebsites.net/search/?q=bla
     • https://sonarwhal-staging.azurewebsites.net/scanner/
     • https://sonarwhal-staging.azurewebsites.net/about/changelog/1
     • https://sonarwhal-staging.azurewebsites.net/docs/user-guide/hints/

I've split this issue into 2 so the integration with Azure Pipelines is done at a different moment (#842).

Thanks for looking into this @LeoSL !

[molant]

Also re the gulp issues you are seeing I have no idea why. What version of node and in what platform are you working? Also, this should not be a blocker as you don't need to build the website to test https://sonarwhal-staging.azurewebsites.net

[LeoSL]

Thank you for the detailed explanation and quick reply @molant ! I really appreciate it!

I will work on that later today.