The tasty-sugar package extends the tasty testing framework with
the ability to generate tests based on golden answer files found for
specific inputs. Multiple answers may be specified with different
parameterization, and there can be associated files that are
presented to the test as well.
The primary use of tasty-sugar is to generate test cases based on the contents of a directory, where the presence of various files determine which tests are generated.
- Tasty.Sugar.CUBE
- Configuration Using Base Expectations
Describes the configuration for tasty-sugar tests, including where they are located and what syntax the files should have.
- Tasty.Sugar.Sweets
- Specifications With Existing Expected Testing Samples
The tasty-sugar library uses one or more CUBE’s to generate a set of test configurations based on the existing files found, outputting a list of Sweets representing the existing test data.
Full information on tasty-sugar features and capabilities will be provided in a later Detailed Information section, but this is a quick introduction describing various testing use cases and how tasty-sugar can be used in those use cases.
For these motivational use cases, the example scenario basis is that the target to be tested is a tool to parse binary ELF files and generate various output information about those files (e.g. similar to objdump, but in a Haskell library form).
- Scenario
- When running a test, it generates output that should be compared to the expected data maintained in a file. There is a simple, single expected output for each test, and no inputs other than the test name.
For the example scenario, several actual ELF binary files were
collected and placed in the test/samples directory:
$ ls test/samples empty.txt fibonacci.c fibonacci foo.tar hello.c hello ls.c ls tmux.c tmux $
Note that there are a couple of non-ELF files in there as well, to verify the errors generated by our library when given
The actual outputs aren’t known yet, but tasty-sugar can help with that. Setup the tasty-sugar CUBE configuration as follows:
cube = mkCUBE { inputDirs = [ "test/samples" ]
, rootName = "*.c"
, expectedSuffix = "exp"
, associatedNames = [ ("binary", "") ]
}
main = do testSweets <- findSugar cube
tests <- withSugarGroups testSweets testGroup $
\sweets expIdx expectation -> return $
testCase (rootMatchName sweets <> " #" <> show expIdx) $ do
let Just binaryName = lookup "binary" $ associated expectation
r <- runTestOn binaryName
e <- readFile $ expectedFile expectation
r @?= e
defaultMain $ testGroup "elf" tests
runTestOn :: FilePath -> IO String
runTestOn f = ...
The tasty-sugar framework does not provide the actual testing: that
is still provided by the developer. Instead, the tasty-sugar
framework reads the contents of the test/samples directory and
analyses the available files to create a list of tests that should
be run. The tasty-sugar package also provides a function that can
organize the tests and invoke the user’s test function once for
each test configuration.
If the tests are run at this point, the tasty-sugar framework will do nothing.
Why?
The tasty-sugar framework will ignore any files in the target
directory that do not have an associated expected file describing
the expected output. This can be confirmed by running the tests
with the --showsearch argument, which will use an alternate tasty
ingredient that does not actually run the tests but write out the
search process and search results.
To get actual tests to run, simply create an expected file for each of the input candidates. The contents of the file can be empty, or any random data.
Running the tests now will result in a test created for each input
file that has a corresponding *.exp file. Note that tasty-sweet
doesn’t actually read in any of the files, just invokes the test
creation function with the Sweets and Expectation data structures
that let the test do whatever is appropriate.
$ ls test/samples empty.txt empty.txt.exp fibonacci finonacci.c fibonacci.exp foo.tar foo.tar.exp hello hello.c hello.exp ls ls.c ls.exp tmux tmux.c $
Note that empty.txt and foo.tar will be ignored, even though
there is an .exp file for them because they don’t match the
source target of *.c. Similarly, tmux will be ignored because
there is no .exp file for it.
At this point, any changes to the target library that cause output changes will be identified when running the tests.
- Scenario
- Similar to the previous scenario, but there is a file containing the expected input needed by the test to generate the output.
To extend the previous example, let us assume that in addition to the pre-existing binaries that we will be generating a number of “interesting” binaries to run the target library on. These will be kept in a different directory where a different part of the build will compile the sources to generate the binaries for testing:
$ ls test/src_samples foo.c foo foo.expct simple.c simple simple.expct recursive.rs recursive recursive.expct functional.hs functional functional.expct $
Note that there are several different source types (C, Rust, Haskell) involved, but each of them has an associated output binary that the target library should be tested on.
In an initial approach, the source files can be ignored by the
testing code: simply create a FILE.expct file for each of the
binaries and use the same Tasty.Sugar.CUBE configuration for this
directory as for the previous directory.
However however an approach where the actual test written by the
user needs access to the source file itself for some reason. This
can be handled by specifying an “associated” file in the
Tasty.Sugar.CUBE configuration:
cube = mkCUBE { inputDirs = [ "test/samples" ]
, rootName = "*.exe"
, expectedSuffix = "expct"
, associatedNames = [ ("c-source", ".c")
, ("rust-source", ".rs")
, ("haskell", ".hs)
]
}
ingredients = includingOptions sugarOptions :
sugarIngredients [cube] <> defaultIngredients
main = do testSweets <- findSugar cube
tests <- withSugarGroups testSweets testGroup $
\sweets expIdx expectation -> return $
testCase (rootMatchName sweets <> " #" <> show expIdx) $ do
e <- readFile $ expectedFile expectation
let assoc = associated expectation
f = rootFile sweets
r <- case lookup "c-source" assoc of
Just c -> runCTestOn f
Nothing ->
case lookup "rust-source" assoc of
Just r -> runRustTestOn f
Nothing ->
runHaskellTestOn f
r @?= e
defaultMainWithIngredients ingredients $
testGroup "elf" tests
runCTestOn :: FilePath -> IO String
runCTestOn f = ...
runRustTestOn :: FilePath -> IO String
runRustTestOn f = ...
runHaskellTestOn :: FilePath -> IO String
runHaskellTestOn f = ...
Now when tasty-sugar generates the test configurations, each test will have a name, a source file, an expected file, and a single associated file. The test is free to use these files in any way it sees fit. For the configuration above, there would be 4 test configurations provided to the test:
| Test Name | Input File | Expected File | Associated Files |
|---|---|---|---|
| simple | simple | simple.expct | (“c-source”, “simple.c”) |
| foo | foo | foo.expct | (“c-source”, “foo.c”) |
| recursive | recursive | recursive.expct | (“rust-source”, “recursive.rs”) |
| functional | functional | functional.expct | (“haskell”, “functional.hs”) |
Note that if both “simple.c” and “simple.hs” files existed, then the simple test configuration would get both as associated files.
- Scenario
- For each input file, multiple tests should be run, each with different parameters, and the expected output may or may not depend on the parameter.
Using the previous example scenario, let’s now assume that for each of the source sample files, two different executables were built: one with and one without optimization. Additionally, if they were a C source file, then there was a version built with GCC and a version built with Clang. The output executables are now named accordingly:
$ ls test/src_samples foo.c foo.noopt.clang.exe foo.O0.gcc.exe foo.opt.clang.exe foo.O2.gcc.exe foo.O3.gcc.exe simple.c simple.noopt.clang.exe simple.noopt.gcc.exe simple.opt-clang.exe simple-opt.gcc-exe recursive.rs recursive.noopt.exe recursive.opt.exe functional.hs functional.noopt.exe functional.opt.exe $
While the filenames are fairly regular, there are different numbers of executables and different naming conventions for different files.
The opt/noopt/O0/O2/O3 and gcc/clang information is known to tasty-sugar as a “parameter”. Parameters can appear in the filename in a specific order, and each parameter may have one of a set of valid values (e.g. gcc or clang) or it may have any (free-form) value (as with the optimization specification).
The Tasty.Sugar.CUBE confguration for is scenario is updated to:
cube = mkCUBE { inputDirs = [ "test/samples" ]
, rootName = "*"
, separators = "-."
, expectedSuffix = "expct"
, associatedNames = [ ("c-source", ".c")
, ("rust-source", ".rs")
, ("haskell", ".hs")
]
, validParams = [
("optimization", Nothing)
,("c-compiler", Just ["gcc", "clang"])
]
}
ingredients = includingOptions sugarOptions :
sugarIngredients [cube] <> defaultIngredients
main = do testSweets <- findSugar cube
tests <- withSugarGroups testSweets testGroup $
\sweets expIdx expectation -> return $
testCase (rootMatchName sweets <> " #" <> show expIdx) $ do
e <- readFile $ expectedFile expectation
let assoc = associated expectation
f = rootFile sweets
r <- case lookup "c-source" assoc of
Just c -> runCTestOn f
Nothing ->
case lookup "rust-source" assoc of
Just r -> runRustTestOn f
Nothing -> runHaskellTestOn f
r @?= e
defaultMainWithIngredients ingredients $
testGroup "elf" tests
runCTestOn :: FilePath -> String
runCTestOn f = ...
runRustTestOn :: FilePath -> String
runRustTestOn f = ...
runHaskellTestOn :: FilePath -> String
runHaskellTestOn f = ...
Parameters are separated by designated separator characters and must appear in the order declared. The default separators are “.” and “-” (e.g. both of the two optimized executable files for the simple.c source above are accepted).
Filenames may omit later parameter values: the file is assumed to apply to all unspecified parameter values if there is no more specific override. This can be very useful to avoid repetition and copying when specifying test files.
In the above example, a simple.expected file would be used for all
four executables, but if there was also a
simple.noopt-gcc.expected and a simple-opt.expected then the
former would be used only for the simple.noopt.gcc.exe and the
latter would be used for both the gcc and the clang executables,
leaving the sample.expected to be used only for the
simple.opt-clang.exe file.
In some cases, there are a large number of possible parameter values, but the expected files change for only some of the parameter variations. While this could normally be handled by an expected file with no corresponding parameter value in its name to represent the default case, this is somewhat more difficult when there are multiple “defaults”. This is handled by parameter ranges.
As an example parameter range, consider a situation that might test the output of the llvm assembler. The llvm assembler output tends to be unchanged over a couple of versions, but then there will be a change (e.g. adding a new LLVM assembly instruction in llvm version 14) that will change the output, but the new format will become the default output.
Conventionally, having a target.ll assembly file as the original default
expected file means that for every version starting at 14, there will be a
specific expected file (e.g. target-llvm14.ll, target-llvm15.ll,
target-llvm16.ll, …) even though all of those files are identical.
To address this, the parameter values can specify an upper or lower range limit
and use the rangedParamAdjuster to adjust the Expectations generated. Using
this, the expected files could be something like target-llvm_pre12.ll,
target-llvm_pre14.ll, target.ll, along with an indication that the
parameter value of llvm_preN formed an upper limit. When used this way, the
target-llvm_pre12.ll file will be chosen for versions 10 and 11 of llvm, the
target-llvm_pre14.ll file would be chosen for versions 12 and 13 of llvm, and
the target.ll file would be chosen for llvm version 14 or higher.
For more information, see the documentation for the rangedParamAdjuster
helper function, and the corresponding tests.
- The tasty-KAT package reads both the inputs and the outputs from a
single file, instaed of allowing the inputs to be a separate file
that can be processed by the target under test.
- The tasty-sugar package allows inputs and outputs to be in separate files, and additional “associated” files to be provided as inputs to the test.
- The tasty-KAT package inputs and outputs must be specifiable in a
file with other KAT markup; this does not easily handle text
markup conflicts and binary inputs/outputs.
- The tasty-sugar package does not attempt to interpret the contents of the files, but simply passes them to the test itself.
- The tasty-KAT package does not allow auxiliary files, or different
parameterized tests.
- As mentioned above, tasty-sugar allows multiple auxiliary files per tests, and allows test inputs and expected outputs to be filename parameterized (with either constrained or free-form parameter values).
- The tasty-golden package requires a 1:1 association between tests
and corresponding golden expected output files; it does not
support file-provided inputs, or associated files.
- The tasty-sugar package allows multiple associated files in addition to the primary input file.
- The tasty-sugar package supports parameterization of expected results (and associated files) as part of the filenames to allow multiple tests per input.
- The tasty-golden package will write the expected results if the
expected file is missing.
- The tasty-sugar package does not actually read or write any of the files identified, it simply provides the names of the files to the test generator function. The user’s test code is responsible for all file processing.
Similar to tasty-golden in functionality.
- Multiple potential outputs, parameterized by filename elements.
- Multiple associated input files.
- Search analysis mode showing how tests are generated based on the available files.
- Automatic grouping of generated tests by parameter values.
- Huge numbers of directories or files will cause performance slowdowns
- Will throw any exception that the listDirectory function can throw.
- There must be a root (input) file to feed to the test
- There must be one or more “expected” results files for a root file
- There may be associated files for the root file required for the test
- All three groups of files may be parameterized by additional fields.
- All fields are represented by a common basename with optional parameters and required associated suffixes, separated by allowable separators.
All of the above may utilize globbing as provided by System.FilePath.Glob
For example, a test which would verify that the size of a compiled file meets the expectations would specify:
CUBE =
{ inputDirs = [ "tests/samples" ] -- relative to cabal file
, separators = ".-"
, rootName = "*.c"
, associatedNames = [ ("exe", "exe")
, ("object", "o")
]
, expectedSuffix = "expected"
, validParams = [ ("arch" : Just ["ppc", "x86_64"]) ]
}
And given the following directory configuration:
tests/samples/ foo.c bar.c bar.exe bar.ppc.exe bar.expected cow.c cow.ppc.exe cow.x86_64.exe cow.expected cow.ppc-expected cow.x86.expected moo.c moo.exe moo-expected dog.exe dog.expected
The result would be:
sweets =
[ Sweets
{ rootMatchName = "bar"
, rootBaseName = "bar"
, rootFile = "tests/samples/bar.c"
, expected =
[ Expectation
{ expectedFile = "tests/samples/bar.expected"
, associated = [ ("exe", "tests/samples/bar.exe") ]
, expParamsMatch = []
}
, Expectation
{ expectedFile = "tests/samples/bar.expected"
, associated = [ ("exe", "tests/samples/bar.ppc.exe") ]
, expParamsMatch = [ ("arch", "ppc") ]
}
]
},
, Sweets
{ rootMatchName = "cow"
, rootBaseName = "cow"
, rootFile = "tests/samples/cow.c"
, expected =
[ Expectation
{ expectedFile = "tests/samples/cow.ppc-expected"
, associated = [ ("exe", "tests/samples/cow.ppc.exe") ]
, expParamsMatch = [ { "arch", "ppc" } ]
}
, Expectedfile
{ expectedFile = "tests/samples/cow.expected"
, associated = [ ("exe", "tests/samples/cow.x86_64.exe") ]
, expParamsMatch = [ ("arch", "x86_64") ]
}
]
},
, Sweets
{ rootMatchName = "moo"
, rootBaseName = "moo"
, rootFile = "tests/samples/moo.c"
, expected =
[ Expectation
{ expectedFile = "tests/samples/moo-expected"
, associated = [ ("exe", "tests/samples/moo.exe") ]
, expParamsMatch = []
}
]
},
Why do the configurations need to be described by a Tasty.Sugar.CUBE
data object? Why can’t they be passed in on the command-line?
- Answer
- They could be, but there are a couple of issues that
would make that more awkward:
- There would need to be a number of command-line arguments to describe all of the CUBE information.
- The tasty framework provides command-line parsing and argument handling (and expects to do so). Handling some command-line arguments prior to tasty and some within tasty would be difficult and brittle (and also note that the set of all tests must be known prior to invoking the tasty main code; they cannot be added dynamically after that point).
- Testing should be consistent: relying on user-input during testing would return different results for different input.