[RFC] CRAM tests

[ghost]

This ticket is a proposal for a CRAM like testing framework. It formalizes the blackbox tests we have in Dune. The idea is to make it as convenient as possible for developpers to write such tests, as well as for users to write reproduction cases that can immediately be added to a project as regression tests.

Overview

Since it is often the case that these kinds of tests require several data files, they will be self-contained in directories ending with the extension .t.

A <dir>.t directory will be considered as a CRAM test by Dune. Dune will not try to interpret the dune files inside this directory, it will be the same as writing a jbuild-ignore file with <dir>.t in it. It will expect to find one or more independent <file>.t files at the toplevel of this directory.

Such files will be composed of:

• the dependencies of the test
• commands to execute
• the expected output from these commands including the exit code when different from 0
• comments

In order to ensure portability, the commands will be written using the action DSL. An example of such file could be:

(deps ((package foo))

This test is testing blah blah

  $ (run foo file.txt)
  Some output.

Testing an error case:

  $ (chdir src)
  $ (setenv X plop)
  $ (run foo bar.txt)
  Error: ...
  [1]

A test involving building some stuff:

  $ (run jbuilder ...)
  ...

Where foo is a binary installed by package foo and file.txt, src/bar.txt are files under the <dir>.t directory. Such files do not need to be declared as dependencies as all .t files will automatically depend on (files_recursively_in .).

Such tests will reuse the same workflow as expectation tests: when the expected output doesn't match, a .t.corrected file will be produced and jbuilder promote will copy over this file to the .t file in
the source tree.

[rgrinberg]

One thing that I'm not entirely sure about is the automatic discovery of tests. We've went through a similar discussion with inline tests and the decision was to have to manually declare them. It seems a bit inconsistent to allow for auto discovery here. Maybe we should consider a cram stanza?

[ghost]

It's not really automatic, the directory name tells you that it's a cram test directory. We can make the exact extension configurable via the dune-project file, this way there is an escape hatch for projects that already use the .t extension for something else.

[rgrinberg]

One thing that's a bit concerning is that we'd basically offering an alternative to craml at this point. This could cause some duplication of effort and if we don't make this generic enough, then craml wouldn't really be first class like what we're proposing here. @samoht do you have any thoughts on this?

[ghost]

Let's forget about this part of the proposal. I wrote this before craml existed and it doesn't cover similar kinds of tests, such as toplevel tests. The proposal is now limited to letting users set a global scheme for the their project to avoid repeating the boilerplate. For instance, by writing something like this in their dune-project file:

(using cram-style-tests 1.0
  (scheme (dir_extension .t) (extension sh) (action (run cram %{<}))
  (scheme (dir_extension .t) (extension sh) (alias runtest-all) (action (run cram -foo %{<}))
  (scheme (dir_extension .t) (extension mlt) (action (run ocaml-topexpect %{<})))

Which would mean:

• for any directory with extension .t do the following:
  • don't recurse inside this directory (same as listing it in ignored_subdirs)
  • for every file.sh file, setup a stanza like this: (alias (name runtest) (deps (source_tree .)) (action (run cram file.sh))
  • for every file.sh file, setup a stanza like this: (alias (name runtest-all) (deps (source_tree .)) (action (run cram -foo file.sh))
  • for every file.mlt file, setup a stanza like this: (alias (name runtest) (deps (source_tree .)) (action (run ocaml-topexpect file.sh))

We could also use another way to mark such directories rather than look at the .t extension, such as a special file inside the directory or all the direct sub-directories of a specified directory in the tree.

[seantalts]

+1 !

Also maybe we should rename the issue since it seems like it's now about something more like make style pattern rules?

[ghost]

Actually, the latest proposal here still doesn't capture enough common use cases. We have a more general one in mind. We are planning to write a RFC for it and then implement it.

[rgrinberg]

@diml So in fact, this proposal is rejected. Shall we close this issue if the use case is intended to be covered by plugins?

[ghost]

Yh, let's close this