Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce unit test based on image comparison #3988

Merged
merged 1 commit into from Mar 5, 2017

Conversation

3 participants
@simonbrunel
Copy link
Member

simonbrunel commented Mar 5, 2017

Attempt to make easier the creation of unit tests that check the drawing output. Until now, this was done by checking calls on a 'fake' context, which is hard to maintain (need to update pixel values by hands) and also not reliable when optimizing code (i.e. different calls sequence but same result).

As of now, it's possible to define 'auto' tests based on JSON/PNG fixtures: chart is generated from the JSON file and compared to the associated PNG image. The image diff is done using pixelmatch. As an example (and in preparation of the filler plugin), add auto tests for the line element fill options.

Creating image based unit tests

  • Create a JS file or JSON file (similar to this one) that defines chart config1 and generation options.
  • Add this file intest/fixtures/{spec.name}/{feature-name}.json.
  • Make sure this kind of line is at the beginning of specs/{spec.name}.js.
  • Set "debug": true in the JSON file to prevent the canvas destruction when running tests.
  • Run gulp unittest --watch --inputs=test/specs/{spec.name}.js.
  • Click the "Debug" button (top/right): a test should fail with the associated canvas visible.
  • Right click on the chart and "Save image as..." test/fixtures/{spec.name}/{feature-name}.png.
  • Refresh the browser page (CTRL+R): test should now pass2.
  • Verify test relevancy by changing the feature values slightly in the JSON file.
  • Remove debug: true from the JSON file.

1 Important: generated charts should be as minimal as possible and focus only on tested features to prevent failure if another feature breaks (e.g. disable the title and legend when testing scales).

2 Tests should pass in both browsers, if not, tolerance and or threshold need to be adjusted at the beginning of the JSON file and kept as low as possible.

Introduce unit test based on image comparison
Attempt to make easier the creation of unit tests that check the drawing output. Until now, this was done by checking calls on a 'fake' context, which is hard to maintain (need to update pixel values by hands) and also not reliable when optimizing code (i.e. different calls sequence but same result).

As of now, it's possible to define 'auto' tests based on JSON/PNG fixtures: chart is generated from the JSON file and compared to the associated PNG image. The image diff is done using `pixelmatch`. As an example (and in preparation of the `filler` plugin), add auto tests for the line element `fill` options.
@simonbrunel

This comment has been minimized.

Copy link
Member Author

simonbrunel commented Mar 5, 2017

Example of a failing test (spanGaps: false instead of true):

image

@tannerlinsley tannerlinsley merged commit 1ca0ffb into chartjs:master Mar 5, 2017

3 checks passed

codeclimate no new or fixed issues
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.2%) to 91.922%
Details
@tannerlinsley

This comment has been minimized.

Copy link
Member

tannerlinsley commented Mar 5, 2017

Spectacular!!

@simonbrunel simonbrunel deleted the simonbrunel:unittest_fixtures branch Mar 5, 2017

@GabrielMancik

This comment has been minimized.

Copy link
Contributor

GabrielMancik commented Mar 27, 2017

I am trying to create new test. But the canvas seems to be destroyed when running tests so i can not create png file. I assume "debug": true should be put inside test/fixtures/{spec.name}/{feature-name}.json. But I have try putting it everywhere there without success.

json: scale-suggestedMin-suggestedMax-no-values.txt is located in test\fixtures\scale.linear

test\specs\scale.linear.tests: scale.linear.tests.txt

roicos added a commit to roicos/Chart.js that referenced this pull request Aug 21, 2017

Introduce unit test based on image comparison (chartjs#3988)
Attempt to make easier the creation of unit tests that check the drawing output. Until now, this was done by checking calls on a 'fake' context, which is hard to maintain (need to update pixel values by hands) and also not reliable when optimizing code (i.e. different calls sequence but same result).

As of now, it's possible to define 'auto' tests based on JSON/PNG fixtures: chart is generated from the JSON file and compared to the associated PNG image. The image diff is done using `pixelmatch`. As an example (and in preparation of the `filler` plugin), add auto tests for the line element `fill` options.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.