Clarify testing levels, supporting code and execution

[JJ]

Problem or new feature

There are currently 4 levels of testing

1. POD basic testing: correctness, some stuff that can be done fast such as check for whitespace by the end of lines. This is in the t directory.
2. POD generation testing, checking that documentable is able to generate a whole documentation set. This is done in Circle-ci, via this config file. This uses a docker container with documentable. We have a limited number of credits in Circle, and they are sometimes exhausted by the end of the month.
3. POD "author" testing, contained in the xt directory. Those are run occasionally, generally by @coke, and are mainly long-running tests like spell checks and stuff like that.
4. The 4th level are tests for code used in tests. Meta-tests, if I may. Right now these are included in the xt directory, and run... I have no idea. Never, probably. Besides, it's not totally clear which are "3" and which are these.

Suggestions

Here's my wishlist

1. Run systematically all tests, from 1 to 4.
2. Clearly distinguish code needed for 1,3,4 and maybe 2 in some way.

This might be related to #2690; we don't need to test everything in every push, but we definitely would need to test everything before every "release"... If we did one. So working through that issue would really help solving this one.

[coke]

I run the xt tests on changed files pretty regularly. I don’t treat tests in xt differently- they are all tests that should be run regularly but shouldn’t break master/main either because they are slow, require extra installs or because the online editor makes it easy to fail them. Based on your comments above, I think only the one examples compilation fits into 4.

[JJ]

El sáb, 21 ago 2021 a las 16:56, Will Coleda ***@***.***>) escribió:

I run the xt tests on changed files pretty regularly. I don’t treat tests in xt differently- they are all tests that should be run regularly but shouldn’t break master/main either because they are slow, require extra installs or because the online editor makes it easy to fail them. Based on your comments above, I think only the one examples compilation fits into 4.

You do a great job. But still, it's manual job, it can be automated, and it should be automated. We should find a way to run them faster (by acting only on changed files, for instance), and even so try and run systematically all tests when, for instance, a "release" happens (if we decide to go that way). I'm warming up to the concept of release, which would allow us to prioritize issues based on the Rakudo release they would correspond to. To be clear, we have still not cleared all issues with 2019.03, so even if we don't do releases, prioritization would help.

[JJ]

I think that something like dividing the xt/ directory in two, docs/ and helper, would clarify a bit the structure. We could run the "helper" tests when the corresponding code changes, and author when we decide to do so.

[coke]

Why only run some of the tests when files change? I’d rather run all the tests for file A when that file changes. FYI, there’s already a script that does this in util/ if you have a clone you can keep around. (If you’re concerned about cycles)

…

On Sun, Aug 22, 2021 at 14:16 Juan Julián Merelo Guervós < ***@***.***> wrote: I think that something like dividing the xt/ directory in two, docs/ and helper, would clarify a bit the structure. We could run the "helper" tests when the corresponding code changes, and author when we decide to do so. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3935 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMIUMURFNU3XZB3DJF2S3T6E5G5ANCNFSM5CRTEHGQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[JJ]

El dom, 22 ago 2021 a las 20:46, Will Coleda ***@***.***>) escribió:

Why only run some of the tests when files change? I’d rather run all the tests for file A when that file changes. FYI, there’s already a script that does this in util/ if you have a clone you can keep around. (If you’re concerned about cycles)

Definitely concerned about cycles. Anyway, running a full suite of tests from time to time is not a bad idea.

[coke]

There are 3 kinds of tests these days: (Documentable tests are no longer part of the repo)

CORE

Tests that should be run every commit - CI has been disabled on travis, and @dontlaugh is working on getting us an equivalent. Assuming limited cycles, this will be the CI tests.

AUTHOR

The author tests are separated as they require either:

• special tooling (like aspell)
• are computationally intensive (like examples compilation)

But are still run on content

MODULE

• tests for the self-contained modules defined in lib/

Make targets

'test' - CORE
'xtest' - all three types

Frequency

Ideally, all content tests should be run for any changed files in the commit. All tests that can be run on a subset of files respect the TEST_FILES env var, so authors can run:

TEST_FILES="file1 file2 file3" make xtest

And get a very fast test run.

see util/test-modified.sh for a linux/mac script to test only files that are modified in the current system, or util/update-and-test to pull the latest changes and create a retest script and run it; catches any failures from commits since the last time you updated and keeps retest around so you can test changes and rerun.