Create integration tests for some major/bigger projects

[t-paul]

This is meant to track down regressions against more real-world designs to cover more ground than just with the small targeted scripts in the CI test suite.

Target would be to run regularly in a docker container on the build server using scheduler.

Some ideas to cover in comparison between latest release against nightly build:

• STL export compare
• Image compare as in the normal test suite
• Timing regression, e.g. relative runtime between nightly and release

Possible designs:

• dotSCAD - examples folder
• NopSCADlib - ?
• BOSL - ?

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[martinbudden]

This looks like a good step forward. As a heavy user of NopSCADlib I have a strong interest that future releases of OpenSCAD don't break that library.

I imagine users of the other libraries feel the same. Unfortunately the OpenSCAD community is not very vocal, so it is difficult to know how much each of these libraries is being used, and what the edge cases are.

[nophead]

To test NopSCADlib and also to test OpenSCAD versions I run tests.py, which runs all the tests that make the pictures for the readme.md and the readme itself and I run make_all.py in the example project folder. I then use the Git GUI to see if anything has changed. The library .gitignore filters out the chaff I am not interested in.

Images are generated to a temp file and compared with ImageMagick, using a lossy compare like the OpenSCAD tests, but it is a lot more sensitive. When the number of changed pixels goes over a threshold it updates the permanent png and leaves an xxx_diff.png file around, so I can inspect what has changed. It will get different results on different hardware. It seems that graphics cards put the stair steps in diagonal edges in different places.

After I added this mitigating commit nophead/NopSCADlib@f0c25d3 the image changes are just the small change to text and the STL changes are the small last digit floating point changes discussed in the forum. However the example is too simple to illustrate some of the problems, so I will update it later to make it more representative.

I also sometimes rebuild all my projects and see if any of the gallery pictures change, but that takes hours.

tests.py and make_all.py and all the scripts it calls measure the time taken by OpenSCAD and print the change since the last build. It is real time rather than CPU time, so it depends on the PC and what it is doing.

[t-paul]

The point of having this separate from the normal CI would be to also allow long runing test. Right now with the current test suite running almost 1500 tests, we did hit CI timeouts so some of the heavier tests are already disabled.

Running scheduled should even allow for a staggered coverage, allowing even multiple-hours runs every month or something like that. That's probably something for step 2, we first need to get this going and see how the small hosted VM handles this, but right now this machine is mostly just idle.

Sounds like starting with the make_all.py would be the best bet. Lets see how that goes.

[martinbudden]

I like the idea of running staggered tests to allow large coverage without hitting host limits.

[nophead]

Note that make_all.py only builds a small project, so doesn't have good coverage of the library but it does expose the current issues better, particularly now I have added overrides for the layer_height and extrusion_width and a build platter. With the last release the STLs are correct but not with master. Also if you open platters/all.scad you will get the deprecated warning. I think that was definitely a bug that I was relying on but I do need some way to make imports in the library import from the project. Perhaps an option on import(), a $cwd built in, or simply remove the deprecation message and look first relative to the file and then relative to the top level file as it does now.

tests.py gives almost full coverage of the library but not the framework for applications.