Skip to content
Branch: master
Find file History
Pull request Compare This branch is 67 commits ahead, 129 commits behind idris-lang:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
base001
basic001
basic002
basic003
basic004
basic005
basic006
basic007
basic008
basic009
basic010
basic011
basic012
basic013
basic014
basic015
basic016
basic017
basic018
basic019
basic020
basic021
basic022
basic023
basic024
basic025
basic026
bignum001
bignum002
bignum003
bounded001
buffer001
buffer002
contrib001
corecords001
corecords002
delab001
directives001
directives002
directives003
disambig002
docs001
docs002
docs003
docs004
docs005
docs006
dsl001
dsl002
dsl003
dsl004
effects001
effects002
effects003
effects004
effects005
error001
error002
error003
error004
error005
error006
error007
error008
error009
ffi001
ffi002
ffi003
ffi004
ffi005
ffi006
ffi007
ffi008
ffi009
ffi010
ffi011
ffi012
folding001
idrisdoc001
idrisdoc002
idrisdoc003
idrisdoc004
idrisdoc005
idrisdoc006
idrisdoc007
idrisdoc008
idrisdoc009
interactive001
interactive002
interactive003
interactive004
interactive005
interactive006
interactive007
interactive008
interactive009
interactive010
interactive011
interactive012
interactive013
interactive014
interactive015
interactive016
interactive017
interactive018
interfaces001
interfaces002
interfaces003
interfaces004
interfaces005
interfaces006
interfaces007
interfaces008
interfaces009
interfaces010
interpret001
interpret002
interpret003
io001
io002
io003
layout001
literate001
meta001
meta002
meta003
meta004
pkg001
pkg002
pkg003
pkg004
pkg005
pkg006
pkg007
pkg008
pkg009
pkg010
prelude001
primitives001
primitives002
primitives003
primitives004-disabled
primitives005
primitives006
proof001
proof002
proof003
proof004
proof005
proof006
proof007
proof008
proof009
proof010
proof011
proofsearch001
proofsearch002
proofsearch003
pruviloj001
quasiquote001
quasiquote002
quasiquote003
quasiquote004
quasiquote005
quasiquote006
records001
records002
records003
records004
records005
reg001
reg002
reg004
reg005
reg007
reg013
reg016
reg017
reg020
reg024
reg025
reg027
reg029
reg031
reg032
reg039
reg040
reg041
reg042
reg045
reg048
reg051-disabled
reg052
reg067
reg075
reg076
reg077
regression001
regression002
regression003
scripts
sourceLocation001
st001
st002
st003
st004
st005
st006
st007
sugar001
sugar002
sugar003
sugar004
sugar005
syntax001
syntax002
tactics001
totality001
totality002
totality003
totality004
totality005
totality006
totality007
totality008
totality009
totality010
totality011
totality012
totality013
totality014
totality015
totality016
totality017
totality018
totality019
totality020
totality021
totality022
totality023
totality024
totality025
totality026
tutorial001
tutorial002
tutorial003
tutorial004
tutorial005
tutorial006
tutorial007
unique001
unique004
universes001
universes002
views001
views002
views003
README.md
TestData.hs
TestRun.hs
mktest.pl

README.md

Idris' Testing Suite

Running the test suite

Do make test_c or make test_js. It will run the tests in parallel and through cabal test. You can also manually call cabal test or stack test.

Passing arguments

To the test program

You can pass arguments to the test program. For example, to pass the --help argument:

# Via make
make TEST-ARGS="--help" test_c
# Via cabal
cabal test --test-options="--help"
# Via stack
stack test --test-arguments="--help"

Try it to learn more about the other arguments you can provide. Two are of particular interest: --node to test against the Node code generator and --pattern <regex> to only run the test that match the provided <regex>.

To idris

You can pass arguments to idris in each of its invocation by the tests. There are two ways to this. You can either modify the idrisFlags term in TestRun, or set the $IDRIS environment variable accordingly

Specifying the number of parallel jobs

With make, the test suite runs in parallel by default. You can specify the number of threads with TEST-JOBS. For stack and cabal, you need to explicitly enable parallelism as you would do with any other GHC-compiled executable. Examples:

# Two test jobs via make
make TEST-JOBS=2 test_c
# Two test jobs via cabal
cabal test --test-options="+RTS -N2 -RTS"
# Two test jobs via stack
stack test --test-arguments="+RTS -N2 -RTS"

Running only previously failed tests

Because of the --rerun-update option, make test_c will create a .tasty-rerun-log file in the root directory of the project. Each time the test suite is run, the file will be written with the result of the tests. The next time you do make test, you can specify the rerun-filter argument to, for example, only run previously failed tests. Valid values are given in the --help.

Extending the test suite

Content of the directory

To each test there shall be a folder that holds at least a bash script run and a golden file expected. When running the test suite, the standard output of the script is compared against the golden file. Any mismatch is reported as a failure.

The name of a test folder is the identifier of the test family followed by a three digit number. A test family gathers related tests.

Add to that four top level files:

  • The program TestRun.hs that runs the test suite
  • A Haskell module TestData.hs with the metadata for each test
  • The Perl script mktest.pl to ease the creation of a new test
  • This README.md

Test families

Tests are categorised with the following test families (and their identifiers):

  • basic: Basic language features, some complete programs
  • bignum: Bignums and GMP
  • bounded: Bounds testing for bits
  • corecords: Tests for corecord
  • delab: De-elaboration tests
  • directives: Tests for directives
  • disambig: Disambiguaton tests
  • docs: Testing documentation functionality
  • dsl: Embedded DSLs and features to support DSL development
  • effects: Effects package
  • error: Error messages and error reflection
  • ffi: FFI calls, including type providers
  • folding Testing some folds
  • idrisdoc: Documentation tool functionality
  • interactive: Interactive editing, proof search
  • interfaces: Testing interfaces.
  • io: IO monad
  • literate: .lidr files; literate programming
  • meta: Meta-programming
  • pkg: Packages
  • primitives: Primitive types
  • proof: Theorem proving, tactics
  • proofsearch: Proof search
  • pruviloj: Elaborator reflection
  • quasiquote: Quasiquotations
  • records: Records
  • reg: Regression tests, covering previous bug fixes
  • regression: Regression tests, covering previous bug fixes
  • sourceLocation: Interaction with files from Idris
  • st: ST, state transition systems
  • sugar: Syntactic sugar, syntax extensions
  • syntax: Syntax extensions
  • tactics: Testing for tactics
  • totality: Totality checking
  • tutorial: Examples from the tutorial
  • unique: Uniqueness types
  • universes: Universes
  • views: Views and covering functions

Adding a test family

  1. Choose an available identifier for your test family. It should be short and somewhat self-explanatory.
  2. Add it in the README.md file, i.e. just above.
  3. Add it in TestData.hs, in the testFamiliesData list. Say the chosen identifier is foo. You should append to the list ("foo", "A proper name for foo", [ ]). The empty list will be replaced with the metadata of each test in the family.
  4. Add your tests.

Adding a test

  1. Choose the family your test shall belong to and remember its identifier.
  2. Pick the next available integer in the test family. It will be the index.
  3. Say the family's identifier is foo and the index is 42. You should call ./mktest.pl foo042 ; this will create the directory and a simple run script.
  4. Modify the run script to your liking. If you want to call the idris executable, write ${IDRIS:-idris} $@.
  5. Add any file you may need in the directory. If they don't end in .idr, you should remember them for the next step.
  6. Add your test in TestData.hs. Each family has a list of (Index, CompatCodegen). See the next section for the available values in CompatCodegen. In most cases, you should write ( 42, ANY).
  7. Generate the expected file by doing:
    # Using cabal
    cabal test --test-options="--pattern foo042 --accept"
    # Using stack
    stack test --test-arguments="--pattern foo042 --accept"
    
  8. Check the content of expected. Maybe the test didn't do what you thought it would. Fix and go back to 7 until it's ok.

Specifying compatible backends

Some tests only make sense with specific code generators. Others don't generate code. You need to supply this information in testFamiliesData. Available values for the compatible backends are:

  • ANY: choose this if your test will work with any code generator
  • C_CG: choose this to only test against the C code generator
  • NODE_CG: choose this to only test against the Node code generator
  • NONE: choose this if you don't want your test to be run with multiple code generators (mainly for tests that only perform type checking)

Currently, NONE has the same effect as ANY, but this will change.

Removing a test

  1. Delete the corresponding directory
  2. Remove the corresponding line in the definition of testFamiliesData in TestData.hs

Updating golden files

To update the expected file for every test, do one of the following:

# Using make
make test_update
# Using cabal
cabal test --test-options="--accept"
# Using stack
stack test --test-arguments="--accept"

"Accepted" tests are the ones that update the golden file. A test can still fail if the run script itself crashes.

You can’t perform that action at this time.