Redesigning Testing Architecture

[DavePearce]

Currently, the testing architecture is pretty simplistic:

• Valid Tests. Contain a single file (e.g. Assert_Valid_1.whiley) which is correct and verifiable.
• Invalid Tests. Contain a single file (e.g. Assert_Invalid_1.whiley), along with an output file (e.g. Assert_Invalid_1.sysout). The output file identifies the expected errors to be generated.

As we move towards incremental compilation, this is no longer going to be sufficient. Specifically, we will need each test to be a sequence of program iterations, some of which may produce invalid code and error messages. In this model, we no longer need to distinguish valid from invalid as every file will contain examples of both. Indeed, we may have different tests which produce the same file but in a different sequence.

In addition to this, we have two major stages which need to be tested:

• Verification. This is currently testing using both valid and invalid tests. However, a number of tests are ignored because they lead to internal failures, etc.
• QuickCheck. This is currently not tested at all. One issue at this point is that QuickCheck cannot correctly identify all invalid tests as invalid (and potentially will never be able to).

Potentially, other things we might like to test include refactoring, etc.

Solution

The proposal here is to have some kind of file format for encoding tests. Specifically, that encodes a sequence of edits on a file along with expected error messages. Potentially, this could also support multiple compilation units to test NameResolution, etc. And, even some regression metrics to help identify performance regressions, etc. Some issues:

• Maintenance. These tests will be much longer and contain more text, thus making them harder to maintain.
• Generation. An interesting question is how to actually go about generating them. This could be done in a programatic way from the current test suite. However, it would also be good to have some "human" entered tests.

[DavePearce]

The key question here is: what is the proposed file format?

Some thoughts:

• (Configuration) Probably want to be able to set certain global properties at the start (e.g. enable / disable this and that).
• (Frames) The body of the file should break into frames where each frame represents a snapshot. Thus, we can represent programming sequences rather than just static programs
• (Errors) Each frame should know what errors / warnings to expect. This could be based entirely on error code, or it could also support the actual error messages.
• (Multiple Files) Each frame should allow updates to multiple files, including deleting files, moving them, etc.
• (Incremental) Each frame should support incremental updates to files using some format reminiscent of diff.

That's a reasonably challenging feature set! But, that would go a long way to helping the testing setup.

[DavePearce]

Initial draft (to become an RFC):

whiley.verify = false
boogie.timeout = 1000
================
>>> src/main.whiley
method main():
>>> src/other.whiley
import main
---
src/main.whiley:1:2..2 E101
other/main.whiley:2:2..3 E301,E302
================
<<< src/other.whiley
>>> src/main.whiley:1:1
method main()
    skip
---
E202:other/main.whiley:2,2-2

An outstanding question is: how to signal a diff rather than a complete replacement? Likewise, for deletions etc.

[DavePearce]

Details within a frame:

U src/main.whiley
@1
+method main()
+    skip
M src/other.whiley other.whiley

This indicates changes to src/main.whiley starting at line 1, and that src/other.whiley is moved to other.whiley.

[DavePearce]

Done. It remains to enable QuickCheck on the tests. Also, need to work it into a framework which is usable by other repos.