Automatic integration testing

[krlmlr]

Subject to cleanup, but otherwise complete (#627)

[yihui]

I guess I may not want this, since the knitr-examples repo requires a big toolchain (yihui/knitr-examples@7d4d10a), and may take a long time to install or compile. A few examples include rgl plots, which may require X to be available (not sure if travis has that). That said, you are welcome to try, and see if it really works on travis within 20 minutes. Thanks!

[krlmlr]

Thanks for your input. There's xpra which can be used as "X for headless machines".

[krlmlr]

Are you concerned about promptly receiving a success/failure notification? The knitr test could push a commit to a specially fabricated knitr-examples testing repository which triggers a (seemingly unrelated) test just for the examples.

[yihui]

I remember travis has a time limit for 20 minutes, and I'm not sure if it can be done within this time.

[krlmlr]

It's been raised to 50 minutes.

[krlmlr]

Any chance we move knitr-examples back as a subdirectory to knitr? The submodule approach works, but it's rather tedious, cf. d136544 .

[yihui]

I tend to add a .travis.yml in the knitr-examples repo instead. I want to keep it as a separate repo, since I have provided links to it in quite a few places on the web (it will be hard to correct all of them), and I probably do not want to check this repo every time I push to the knitr repo. It seems it will still take you quite a while to figure out all the dependencies of the knitr-examples repo :)

[krlmlr]

Integration test now counts number of different files and lines and checks against expected values (cf. this diff).

[krlmlr]

This pull request is in a "works for me" state. Shall we start discussing it?

[yihui]

Yes, I'm still trying to catch up with your huge amount of work. You have been running too fast :)