WIP: Add custom jest reporter. Refs #10

[badsyntax]

Original PR: #32

This change introduces a custom jest reporter that generates a report of the differencify tests once all tests have completed.

Requirements:

1. Indexes all tests, saves to custom location in HTML format
2. Includes snapshot and diff images from the test directories
3. Includes test statuses
4. Is unit tested
5. Includes debug/log messages (eg, mismatch info)
6. Includes "after" image (not currently saved to disk)

To be fixed:

1. Current implementation does not find the diff image for whatever reason

Issues:

1. Custom reporter config viapackage.json is not supported by CRA
   • You can specify a custom reporter via cli options but you can't specify reporter config.
   • You can supply a custom jest.config.js via cli options but that overrides CRA's jest config.
   • I have set sensible config defaults until this is resolved in CRA. More info here: https://github.com/facebookincubator/create-react-app/blob/master/packages/react-scripts/template/README.md#user-content-configuration
2. Only support jest tests

[NimaSoroush]

@badsyntax : should the report call happen in separate call. something like:

...
  afterAll(async () => {
    await differencify.report();
  });
...

[badsyntax]

@NimaSoroush You don't need to explicitly call it like that, you just enable the reporter via the jest reporters config. (Check this config). The reporter is then run when all tests are complete.

The reporter will then check every test for the existence of the snapshot image and the diff image and add those tests to the report. It's probably best to have a custom test:visual npm script that only runs the reporter on the visual tests. In that case we can configure jest to use a custom reporter via the cli options (not documented on the website).

[NimaSoroush]

@badsyntax : Cool, that makes sense. I was more thinking of programmatically activation/deactivation of the report. But this looks good as the first version

[badsyntax]

@NimaSoroush With this new approach we can support multiple test files. If we have to programmatically activate the report then I don't think we can support a report that indexes multiple tests.

[NimaSoroush]

@badsyntax : even with the thing I suggested to write the report somewhere (e.g. in .txt/json file or keep it in memory like differencify.report) while executing the test, and then when executing differencify.report() it will pick up that file/variable and generate the report?

// differencify
import { reporter } from 'reporter';

class Differencify {
  async report() {
    reporter.report();
  }
  ...
}

// page.js
import { reportObj } from 'reporter';
class Page {
  ...
  async goto(url) {
  ...
  if (this.globalConfig.generateReport) {
    reportObj['debug'] = `
      ${reportObj['debug']}
      Opening new tab...`;
  }
  ...
  }
}

// compareImage.js
  import { reportObj } from 'reporter';
  compareImage = async (capturedImage, globalConfig, testConfig) => {
    ...
    reportObj = {
      summary: 'passed',
      referenceImg: './path/reference.png',
      diffImg: './path/diff.png',
    }
    

what do you think?

[badsyntax]

So I actually prefer for it to be explicit like that, and this is similar to my original approach, but I found it doesn't work well with multiple test files. I changed the approach to use a custom jest reporter to support generating a report that indexes all tests.

Let's consider the following scenario:

test1.test.js - running 
test2.test.js - running 

test2.test.js - complete - differencify.report()
test1.test.js - complete - differencify.report()

The problem with this approach is that:

• The report is generated multiple times.
• test1 doesn't know anything about test2 so for each differencify.report() we'd need to do a directory scan to find snapshots, diff and metadata of all other tests.
• potential race condition between test2 and test1 trying to save the report (tests are run in parallel)

With the new approach, using a custom jest reporter, it works like so:

test1.test.js - running 
test2.test.js - running 

test2.test.js - complete
test1.test.js - complete

all tests complete - generate report

In this case, we know which exact tests were executed, so we don't need to a directory scan, and the report is only generated once.

[badsyntax]

@NimaSoroush Hit me up on Gitter or slack if you'd like to chat in more detail about this

[NimaSoroush]

@badsyntax : Hey Richard, have you got any update on this PR? I am happy to help if anything left

[nimishseraphim]

differencify version update not needed ?

[NimaSoroush]

It does. Richard probably going to update this at some point

[badsyntax]

I'm not entirely what you mean @nimishseraphim, but this branch is now up to date with master.

[NimaSoroush]

He means Differencify@1.3.0 -> Differencify@1.4.0

Can you add this @badsyntax ?

[badsyntax]

I've updated this 👍

[badsyntax]

@NimaSoroush I'll update and finish of these changes this afternoon. Very sorry for the delay.

[NimaSoroush]

@badsyntax : No worries Richard. Take your time and thanks for doing this. I can also help out if you want

[badsyntax]

@NimaSoroush Hi Nima. This MR is ready for review again.

[NimaSoroush]

This is fine to be ../dist/reporter but it will break the circleCi pipeline as we don't do npm run build before npm run test:integration

[NimaSoroush]

Can you update that?

[NimaSoroush]

@badsyntax : Hi Richard, Sorry to get back on this with a long delay. I just got some time to check this and put some comments.

I ran this locally and noticed that it only generates reports for failed tests? I'll add my generated files here. 2 tests passed and the rest failed, but in the report, I only see failed ones

Archive.zip

[NimaSoroush]

Can we add a link to differencify reporter example here?

[NimaSoroush]

This PR can be closed in favor of #103