Support test metadata file for UnitTest outside of mason packages

[ben-albrecht]

Today, UnitTest expresses test metadata through (1) calls to UnitTest methods in the unit test function body and (2) the Mason.toml manifest file for a mason package.

The metadata features of (2) are not available to tests written outside of the context of a mason package. These mason-exclusive features currently include:

• compopts
• execopts

These features could be supported outside of mason packages by adding a test metadata file associated with a Chapel program.

Here is a simple straw proposal:

For a file, Foo.chpl that contains various tests. a user can create a Foo.toml to store test metadata that will be used when doing mason test Foo.chpl.

Contents of Foo.toml could look like:

[Foo]
compopts = "--baseline"
execopts = "--correctness"

More features could be added to this in the future, such as using multiple compopts/execopt per test, setting the default number of locales (#15078), and performance testing metadata (#15680)

[ankushbhardwxj]

@ben-albrecht That's a good solution. We can specify two modes for mason test - one inside a package, and other outside a package. If mason test doesn't find a Mason.toml file at cwd, it switches to other mode and runs tests on a file using its corresponding .toml.

[ben-albrecht]

Possible design Q: Would it ever make sense to distinguish compopts/execopts across test suites or specific tests within a file?

I would think not, but if it were ever needed, we could use the TOML hierarchy to support this:

[Foo]
compopts = "--baseline"

[Foo.TestSuiteA]
execopts = "-N=256"

[Foo.TestSuiteB]
execopts = "-N=1024"

[ben-albrecht]

There could be value in storing metadata per test, if we were to support default numLocales in the metadata, e.g.

[test]
# Defaults for all tests
numLocales = "1"

[test.foo]
# Defaults for all tests within foo.chpl
numLocales = "2"
execopts = "-N=256"

[test.foo.testBar]
# Defaults for specific test: testBar()
numLocales = "4"

This would be used for mason bench as well.

[ben-albrecht]

@krishnadey30 - you had asked for a ping on this issue

[ben-albrecht]

Translating an example from mason bench design prototype:

[test] # Settings for all tests
[test.transposePerf] # Settings for all tests within transposePerf.chpl
[test.transposePerf.transposePerf] # Settings for transposePerf() test within transposePerf.chpl

[test.transposePerf.transposePerf.execopts]
m10 = "--m=10 --iters=1000 --reference=true"
m1000 = "--m=1000 --iters=10 --reference=true"
m10000 = "--m=10000 --iters=1 --reference=false"

The above is demonstrating the meanings of the different nested tables. The user might actually write something like:

[test.transposePerf.execopts]
m10 = "--m=10 --iters=1000 --reference=true"
m1000 = "--m=1000 --iters=10 --reference=true"
m10000 = "--m=10000 --iters=1 --reference=false"

inline table version:

[test.transposePerf]
execopts = { m10 = "--m=10 --iters=1000 --reference=true",
             m1000 = "--m=1000 --iters=10 --reference=true",
             m10000 = "--m=10000 --iters=1 --reference=false" }

# This is nice b/c you might want to list some other metadata within the same table.

[bradcray]

In a discussion about unit testing today, I was pointed to CMake's ctest framework as an example of a testing framework that involves meta-issues outside of the code itself such as environment and the like which might be something to look at as we think about how to express things in Chapel's unit testing framework that can't be expressed directly in the code (like how many locales to use, what launcher or communication layer to use, or the like). This seemed like a reasonably associated issue to drop that thought on before I forget it, but let me know if there's someplace better (@ben-albrecht or others).