Add unit test cases into Processing.R

[gaocegege]

Now Processing.R imports the basic test framework (jacoco and junit), and the test coverage output will redirect to codacy.

[gaocegege]

We need help from the community:

• How to write unit test cases for Processing.R Post on Processing Forum
• Tools to generate Processing References for Processing.R

[gaocegege]

comments from the post on Processing Forum:

I am not an expert on this kind of testing, however, one idea that occurs to me:

Sketches could be tests -- running them to see if things work is a form of testing, even without automation.

Automating sketch-testing might be done with pixel-diffs: expected sketch saveFrame() output vs. actual saveFrame() output. This would be good for checking reimplementations of existing documented sketches, and good for catching regressions.

1. run a canonical sketch in Processing(Java), e.g. to test rect().
2. the sketch generates a known-good image with saveFrame() and/or known-good text log
3. write a sketch stub, e.g. to test rect() in Processing.R
4. the test will pass when running the sketch generates a matching (or almost-matching) image file to the known-good image.

I once wrote a Processing assignment checker that could also be used as a kind of automated regression tester for sketches. Essentially, the test consists of the sketch and known-good screenshot(s) of the sketch on a certain frame. The test runs the sketch, which generates a new screenshot using saveFrame(). The test then compares the known-good screenshot and the new one. A change in visual output may indicate a regression of the feature that the sketch demonstrates.

(This approach wouldn't work with everything -- it could get really complicated simulating live input to interactive sketches, and reproducibility would suffer with clock-based math because frameCount and millis() are not lockstep.)

[gaocegege]

IMO, we should make the boundary clear -- which belongs to Processing.R and which belongs to Processing.

In the test cases in Processing.R, we could:

1. Mock Processing from the code or assume that Processing is always right.
2. Test the logic in Processing.R, such as Runner or RLangMode.

I will try to write some unit test cases. As for integration test or system test, it is a little complex. And I think it could be another GSoC project 🤔 But it has a universal significance for Processing.

[jeremydouglass]

Sounds good to me! FYI here is a recent thread about a Processing(Java) sketch tester:

https://forum.processing.org/two/discussion/22869/pixel-test-for-simple-sketches-feedback-request

[jeremydouglass]

@gaocegege -- might want to join the discussion there.

[gaocegege]

Yeah, I will take a look tomorrow. 😄

[gaocegege]

I have a glance on the post. It is really an awesome idea. It is more like end-to-end test. And I will join the discussion tomorrow, oh, today, since it is 2:00 am now.

[gaocegege]

FMI: There is a issue when uploading the result to codacy in my local env: codacy/codacy-coverage-reporter#56

[gaocegege]

e2e test have a good coverage for runner and RLangPApplet, other classes related to editor or mode couldn't be tested in e2e mode. We could ensure the integration between Processing and Processing.R works well.

[gaocegege]

Now I have merge the e2e test code into master: #87

We should find a way to maintain examples, which will be used in web reference site and e2e test.

Maybe meta programming, aka code generation will help us: generate test cases and reference pages from examples.

[gaocegege]

There is a new config file needed when import a new reference code:

examples/
  reference/
    <function-name>/
      .property.yml
      <function-name>1/
        .test.yml
        <function-name>1.rpde
      <function-name>2/
      ...

test:
  reference: https://processing.org/reference/images/rect_0.png

[gaocegege]

Now the documentation and test cases could be generated from examples/reference automatically. 🎉 It is really awesome. The thing we need to do now is to add new items in examples/reference and re-generate test cases and documentation by running the script https://github.com/gaocegege/Processing.R/blob/master/hack/generate-e2e-test.py and https://github.com/Processing-R/Processing.R-docs/blob/master/scripts/generate-reference.sh

And I think the idea could be used in Processing, too.

[jeremydouglass]

How exciting!!!

I'm happy to see that this approach will work -- and that it will generate both e2e tests and documentation from the same code base of sketch examples and metadata files.

I agree, I think that Processing might be interested in this approach for the main project and other modes.

[gaocegege]

[gaocegege]

Partially done, I think it is time to close the issue. There are some drawbacks of the current solution:

• No way to test editor or mode itself.
• No way to test animation.

And there are two historical problems: #111 and #103

[jeremydouglass]

This looks fine to me!

• Animation: this seems doable, but not a necessary enhancement, as the vast majority of all Processing reference documentation is not animated.
• Testing the editor: is there any specific editor testing that needs to be done, or should that be left to Processing?
• Testing the mode itself: perhaps open a separate issue (unscheduled, no milestone) for this to come back to it in the future.

[gaocegege]

Animation: this seems doable, but not a necessary enhancement, as the vast majority of all Processing reference documentation is not animated.

It works for me.

Testing the editor: is there any specific editor testing that needs to be done, or should that be left to Processing?

I am not sure which layer we should put the test in. But if Processing does it, it works for Processing.R, too. And I will file a new issue about it and try to do a research.

Testing the mode itself: perhaps open a separate issue (unscheduled, no milestone) for this to come back to it in the future.

Yeah, it is helpful. I am not sure if the mode test and the editor test are the same thing, we could put them in one issue.